2.2 Machine learning lecture 3 course notes  (Page 9/12)

This is a resonable measure of $x$ and $z$ 's similarity, and is close to 1 when $x$ and $z$ are close, and near 0 when $x$ and $z$ are far apart. Can we use this definition of $K$ as the kernel in an SVM? In this particular example, the answer is yes. (This kernel is called the Gaussian kernel , and corresponds to an infinite dimensional feature mapping $\Phi$ .) But more broadly, given some function $K$ , how can we tell if it's a valid kernel; i.e., can we tell if there is some feature mapping $\Phi$ so that $K(x,z)=\Phi {\left(x\right)}^T\Phi \left(z\right)$ for all $x$ , $z$ ?

Suppose for now that $K$ is indeed a valid kernel corresponding to some feature mapping $\Phi$ . Now, consider some finite set of $m$ points (not necessarily the training set) $\{x^{\left(1\right)},...,x^{\left(m\right)}\}$ , and let a square, $m$ -by- $m$ matrix $K$ be defined so that its $(i,j)$ -entry is given by $K_{ij}=K(x^{\left(i\right)},x^{\left(j\right)})$ . This matrix is called the Kernel matrix . Note that we've overloaded the notation and used $K$ to denote both the kernel function $K(x,z)$ and the kernel matrix $K$ , due to their obvious close relationship.

Now, if $K$ is a valid Kernel, then $K_{ij}=K(x^{\left(i\right)},x^{\left(j\right)})=\Phi {\left(x^{\left(i\right)}\right)}^T\Phi \left(x^{\left(j\right)}\right)=\Phi {\left(x^{\left(j\right)}\right)}^T\Phi \left(x^{\left(i\right)}\right)=K(x^{\left(j\right)},x^{\left(i\right)})=K_{ji}$ , and hence $K$ must be symmetric. Moreover, letting ${\Phi }_k\left(x\right)$ denote the $k$ -th coordinate of the vector $\Phi \left(x\right)$ , we find that for any vector $z$ , we have

The second-to-last step above used the same trick as you saw in Problem set 1 Q1. Since $z$ was arbitrary, this shows that $K$ is positive semi-definite ( $K\ge 0$ ).

Hence, we've shown that if $K$ is a valid kernel (i.e., if it corresponds to some feature mapping $\Phi$ ), then the corresponding Kernel matrix $K\in {\mathbb{R}}^{m\times m}$ is symmetric positive semidefinite. More generally, this turns out to be not only a necessary, butalso a sufficient, condition for $K$ to be a valid kernel (also called a Mercer kernel). The following result is due to Mercer. Many texts present Mercer's theorem in a slightly more complicated form involving $L^2$ functions, but when the input attributes take values in ${\mathbb{R}}^n$ , the version given here is equivalent.

Theorem (Mercer). Let $K:{\mathbb{R}}^n\times {\mathbb{R}}^n\mapsto \mathbb{R}$ be given. Then for $K$ to be a valid (Mercer) kernel, it is necessary and sufficient that for any $\{x^{\left(1\right)},...,x^{\left(m\right)}\}$ , ( $m<\infty$ ), the corresponding kernel matrix is symmetric positive semi-definite.

Given a function $K$ , apart from trying to find a feature mapping $\Phi$ that corresponds to it, this theorem therefore gives another way of testing if it is a valid kernel.You'll also have a chance to play with these ideas more in problem set 2.

In class, we also briefly talked about a couple of other examples of kernels. For instance, consider the digit recognition problem, in which given an image (16x16 pixels) of a handwrittendigit (0-9), we have to figure out which digit it was. Using either a simple polynomial kernel $K(x,z)={\left(x^{T}z\right)}^d$ or the Gaussian kernel, SVMs were able to obtain extremely good performance on this problem.This was particularly surprising since the input attributes $x$ were just a 256-dimensional vector of the image pixel intensity values, and the system had no prior knowledge about vision,or even about which pixels are adjacent to which other ones. Another example that we briefly talked about in lecture was that if the objects $x$ that we are trying to classify are strings (say, $x$ is a list of amino acids, which strung together form a protein), then it seems hard to construct a reasonable, “small”set of features for most learning algorithms, especially if different strings have different lengths. However, consider letting $\Phi \left(x\right)$ be a feature vector that counts the number of occurrences of each length- $k$ substring in $x$ . If we're considering strings of english letters, then there are ${26}^k$ such strings. Hence, $\Phi \left(x\right)$ is a ${26}^k$ dimensional vector; even for moderate values of $k$ , this is probably too big for us to efficiently work with. (e.g., ${26}^4\approx 460000$ .) However, using (dynamic programming-ish) string matching algorithms, it is possible to efficientlycompute $K(x,z)=\Phi {\left(x\right)}^T\Phi \left(z\right)$ , so that we can now implicitly work in this ${26}^k$ -dimensional feature space, but without ever explicitly computing feature vectors in this space.