16. Learning: Support Vector Machines

What people said

Strong rec Nikolas Rieble · 2018-08 · Cross Validated Stack Exchange · on “Support Vector Machines: a beginner's question about the underlying math”

I have never seen a better explanation that this one: [this video] Any answer that i could write would just be summary of this video.

Strong rec Tom · 2018-10 · Cross Validated Stack Exchange · on “Why are the support vectors in an SVM classifier on hyperplanes that are 1 or -1 away from the wanted hyperplane”

It is arbitrary, the fact that b is there makes the actual value irrelevant. You should watch this video: [this video] It is the best out there in my opinion.

Recommends Nikolas Rieble · 2016-12 · Cross Validated Stack Exchange · on “what is weight vector and bias in svm”

This video helped me once to fully and thoroughly understand SVMs [this video] (Yet no explanation of weight vector and bias, just great explanation of the whole concept)

Recommends Nikolas Rieble · 2017-01 · Cross Validated Stack Exchange · on “How does a Support Vector Machine (SVM) work?”

In addition to this great answer, I want to recommend this video which walks through the mathematics behind SVM and especially clarifies the question @tosik commented [this video]

Recommends Nikolas Rieble · 2017-01 · Data Science Stack Exchange · on “Difference between subgradient SVM and kernel SVM?”

You are gravely misunderstanding SVM. Sub-gradient descent algorithm for SVM is a method to solve the underlying optimization problem of SVM. SVM is always a linear classifier, which can yet by using kernels operate in a higher dimensional space. Therefore in input space, the separating hyperplane (linear!) computed in feature space (kernel!), seems non-linear. Effectively you are thereby solving a non-linear classification task, but you are projecting into a higher dimensional feature space where the classification task is solved by a linear classifier. Please read: Wikipedia on svm [LINK] Please watch: great lecture about SVM [this video]

Recommends Antoni Parellada · 2017-02 · Cross Validated Stack Exchange · on “How does a Support Vector Machine (SVM) work?”

Nothing original... Just my own notes at a more entry level. Basically from this video from MIT [this video] with my own illustrations. For errors, please let me know. For insightful answers, and further details go to the expert level (Franck's post and others).

Recommends Ankit Seth · 2018-03 · Data Science Stack Exchange · on “Why is there a $2$ at the denominator of the mean squared error function?”

This is just for mathematical convenience. When you differentiate $C(w,b)$, you will get an extra $2$. To eliminate that, $2$ is kept beforehand in denominator. You can also watch this video on SVM lecture [this video] by Patrick Winston, where he uses a similar formula and then tells that he is using $2$ in denominator just for mathematical convenience.

Recommends Florian H · 2018-07 · Stack Overflow · on “SVM visualization really random and inaccurate”

You are using a svc with the standard parameters. Especialy your Kernel is an rbf kernel (also known as gaußian). How SVM's work a litle bit too complicated to post here and with a kernel it's even more complex. If you are intrested i can recommend you the lecture from MIT. [this video] But in short your gaußian kernel does the separation still linear but in a higer dimensional vectorspace. It transforms your data for the separation into that space and separates it linearily with a hyperplane. If you visualize your data later with your support vectors in two dimensions the separation is not linar but in your "kernel-vector-space" it is. Btw. you should also think about standardization [LINK] .

Recommends GrowinMan · 2019-02 · Mathematics Stack Exchange · on “Do we maximize the Lagrangian function or minimize it for the SVM derivation? And why?”

These are the resources I've looked into: MIT lecture [this video] Math.stackexchange question [LINK] Slide from MIT [LINK] Stanford lecture [LINK] Caltech lecture [LINK] The professor in (1) at 25:47 says that "I don't care if we find the maximum or minimum (of the Lagrangian function) .... we'll just find the extremum..." Accepted answer in (2) tries to imply that we're trying to minimize the (primal) Lagrangian. In (3), slide 14 explicitly mentions that we're trying to minimize the function In (4) at 24:30 the professor mentions that we'll just set the derivative for the Lagrangian function to zero with respect to the hyperplane parameters ( $\boldsymbol{w}$ & $b$ ) (this is of course, how you

Recommends Nitin · 2019-09 · Cross Validated Stack Exchange · on “Support Vector Machine: identifying support vectors and kernel linear separability”

I went through the MIT Artificial Intelligence lecture on Support Vector Machines by Professor Patrick Winston: [this video] I've got a couple of questions. Would be helpful if someone has gone through this lecture or is familiar with the SVM derivation in general but I think anyone who is familiar with Linear Algebra or SVMs can answer these questions without going through the lecture fully. 1) Around ( 27:00 and after ). We are solving the system as a constrained optimisation problem using Lagrange Multipliers. In the Lagrangian the constraints take the form: $$ \sum_{i} \alpha_i[y_i(\bar w.\bar x_i+b)-1] $$ This constraint comes from all the $ \bar x_i $ vectors that lie on the "gutters" of th