Lecture 15 - Kernel Methods

What people said

Recommends Starz · 2017-04 · Cross Validated Stack Exchange · on “Dimensionality of the Gaussian Kernel”

Yes to your question on whether the dimensionality of feature space is 4. For Gaussian kernel, it's infinite dimension as you have mentioned as well. You can learn in more depth from Professor Yaser Abu-Mostafa's lecture [this video] . You can jump straight to around 26:50 if you are already familiar with the subject.

Recommends eSurfsnake · 2017-11 · Cross Validated Stack Exchange · on “Conceptual Understanding of Kernels”

Sort of, but not quite. Suppose you have a two-dimensional feature space, that is, for each item (that can get classified as '+' or '-') you have a tuple with two values: ($x_1, x_2$). Now suppose you have a kernel like the polynomial kernel, $K(x,y)=(x^Ty+c)^d$. If c is zero, and d is 2, then when we have two input vectors $(x_1,x_2)$ and $(y_1,y_2)$ their representation in the expanded feature space is $$(x_1y_1+x_2y_2)^2=x_1^2y_1^2+2x_1y_1x_2y_2+x_2^2y_2^2$$ As you can see, there are three terms in the new feature space. But imagine if $d$ were a much larger number, like 100. Then the number of terms would be huge in the feature space. In fact, it is possible to have an $infinite-dimensio

Recommends edmz · 2021-11 · Cross Validated Stack Exchange · on “How to intuitively explain what a kernel is?”

In the top answer to this question, there's a link to the lecture of Pr. Yaser Abu-Mostafa from CalTech [this video] and it gives a very nice intuition of it... so I'll try to explain what i understood, without equation : a kernel is a function (relatively simple to compute) taking two vectors (living in the X space) and returning a scalar that scalar happens in fact to be exactly the dot-product of our two vectors taken to a higher dimension space Z so, the kernel tells you how close two vectors are in that Z space, without paying the (possibly enormous) price of computing their coordinates there that's all you need to fit an SVM model! in a regular SVM model, you would have used the dot-product

Recommends Your neighbor Todorovich · 2025-11 · Cross Validated Stack Exchange · on “Is the figure showing margin violation for the support vector machine correct?”

I am listening to a lecture on soft margin SVM [this video] [LINK] The lecturer says that the blue dot represents a violation of the margin. I don't really agree with his point. By definition, the (one-sided) margin is the smallest distance between a data point and the optimal hyperplane, and this quantity is calculated to be, $\min_i \dfrac{y_i(w^Tx_i +b)}{\|w\|_2}$ , where $(w,b)$ is the optimal hyperplane. When we impose the constraint that the minimum $y_i(w^Tx_i +b) = 1$ , then the margin is $1/\|w\|_2$ . Observe the margin changes with the optimal hyperplane, $(w,b)$ . According to this definition all linearly separable dataset has a positive margin. The only issue that will arise is when t

Cites Alexey Grigorev · 2015-05 · Cross Validated Stack Exchange · on “How to intuitively explain what a kernel is?”

Kernel is a way of computing the dot product of two vectors $\mathbf x$ and $\mathbf y$ in some (possibly very high dimensional) feature space, which is why kernel functions are sometimes called "generalized dot product". Suppose we have a mapping $\varphi \, : \, \mathbb R^n \to \mathbb R^m$ that brings our vectors in $\mathbb R^n$ to some feature space $\mathbb R^m$. Then the dot product of $\mathbf x$ and $\mathbf y$ in this space is $\varphi(\mathbf x)^T \varphi(\mathbf y)$. A kernel is a function $k$ that corresponds to this dot product, i.e. $k(\mathbf x, \mathbf y) = \varphi(\mathbf x)^T \varphi(\mathbf y)$. Why is this useful? Kernels give a way to compute dot products in some featur

Cites azizj · 2017-08 · Cross Validated Stack Exchange · on “SVM: How to get predicted output from SVM with Gaussian / RBF Kernel? Andrew Ng's course svmPredict”

Thanks @Chaconne! Making some sense now. :) According to this lecture on Kernel methods [this video] , the $sgn(...)$ you have above is the $g(x)$ in the Lagrangian. You're saying that $g(x)$ is the final predicted output? What if I give you a new input $\vec u$? How would you find its predicted output?

Cites John Smith · 2023-03 · Cross Validated Stack Exchange · on “Is a polynomial kernel ridge regression really equivalent to performing a linear regression on those expanded features?”

Thank you user20160, that is an excellent answer. Do you know how to extract the coefficients in feature space. In sk learn, it is only possible for linear SVM. And it would be interesting to know these coefficients to understand the feature importance. Especially for poly kernel. What do you think? For RBF, we could also do something with the consideration of RBF as infinite dim [this video]