2.5 Machine learning lecture 6 course notes

The perceptron and large margin classifiers

In this final set of notes on learning theory, we will introduce a different model of machine learning. Specifically, we have sofar been considering batch learning settings in which we are first given a training set to learn with, and our hypothesis $h$ is then evaluated on separate test data. In this set of notes, we will consider the online learning setting in which the algorithm has to make predictions continuously even while it's learning.

In this setting, the learning algorithm is given a sequence of examples $(x^{\left(1\right)},y^{\left(1\right)}),(x^{\left(2\right)},y^{\left(2\right)}),...(x^{\left(m\right)},y^{\left(m\right)})$ in order. Specifically, the algorithm first sees $x^{\left(1\right)}$ and is asked to predict what it thinks $y^{\left(1\right)}$ is. After making its prediction, the true value of $y^{\left(1\right)}$ is revealed to the algorithm (and the algorithm may use this information to performsome learning). The algorithm is then shown $x^{\left(2\right)}$ and again asked to make a prediction, after which $y^{\left(2\right)}$ is revealed, and it may againperform some more learning. This proceeds until we reach $(x^{\left(m\right)},y^{\left(m\right)})$ . In the online learning setting, we are interested in the total number oferrors made by the algorithm during this process. Thus, it models applications in which the algorithm has to make predictions even while it'sstill learning.

We will give a bound on the online learning error of the perceptron algorithm. To make our subsequent derivations easier, we willuse the notational convention of denoting the class labels by $y=\in \{-1,1\}$ .

Recall that the perceptron algorithm has parameters $\theta \in {\mathbb{R}}^{n+1}$ , and makes its predictions according to

where

Also, given a training example $(x,y)$ , the perceptron learning rule updates the parameters as follows. If $h_{\theta }\left(x\right)=y$ , then it makes no change to the parameters.Otherwise, it performs the update This looks slightly different from the update rule we had written down earlier in the quarterbecause here we have changed the labels to be $y\in \{-1,1\}$ . Also, the learning rate parameter $\alpha$ was dropped. The only effect of the learning rate is to scale all the parameters $\theta$ by some fixed constant, which does not affect the behavior of the perceptron.

The following theorem gives a bound on the online learning error of the perceptron algorithm, when it is run as an online algorithm that performs an updateeach time it gets an example wrong. Note that the bound below on the number of errors does not have an explicit dependence on the number ofexamples $m$ in the sequence, or on the dimension $n$ of the inputs (!).

Theorem (Block, 1962, and Novikoff, 1962) . Let a sequence of examples $(x^{\left(1\right)},y^{\left(1\right)}),(x^{\left(2\right)},y^{\left(2\right)}),...(x^{\left(m\right)},y^{\left(m\right)})$ be given.Suppose that $\left|\right|x^{\left(i\right)}\left|\right|\le D$ for all $i$ , and further that there exists a unit-length vector $u$ ( ${\left|\right|u\left|\right|}_2=1$ ) such that $y^{\left(i\right)}·\left(u^T{x}^{\left(i\right)}\right)\ge \gamma$ for all examples in the sequence (i.e., $u^T{x}^{\left(i\right)}\ge \gamma$ if $y^{\left(i\right)}=1$ , and $u^T{x}^{\left(i\right)}\le -\gamma$ if $y^{\left(i\right)}=-1$ , so that $u$ separates the data with a margin of at least $\gamma$ ). Then the total number of mistakes that the perceptron algorithm makes on this sequence is atmost ${(D/\gamma )}^2$ .

Proof. The perceptron updates its weights only on those examples on which it makes a mistake. Let ${\theta }^{\left(k\right)}$ be the weights that were being used when it made its $k$ -th mistake. So, ${\theta }^{\left(1\right)}=\overrightarrow{0}$ (since the weights are initialized to zero), and if the $k$ -th mistake was on the example $(x^{\left(i\right)},y^{\left(i\right)})$ , then $g\left({\left(x^{\left(i\right)}\right)}^T{\theta }^{\left(k\right)}\right)\ne y^{\left(i\right)}$ , which implies that

Also, from the perceptron learning rule, we would have that ${\theta }^{(k+1)}={\theta }^{\left(k\right)}+y^{\left(i\right)}{x}^{\left(i\right)}$ .

We then have

By a straightforward inductive argument, implies that

Also, we have that

The third step above used Equation  [link] . Moreover, again by applying a straightfoward inductiveargument, we see that  [link] implies

Putting together  [link] and  [link] we find that

The second inequality above follows from the fact that $u$ is a unit-length vector (and $z^{T}u=\left|\right|z\left|\right|·\left|\right|u\left|\right|cos\phi \le \left|\right|z\left|\right|·\left|\right|u\left|\right|$ , where $\phi$ is the angle between $z$ and $u$ ). Our result implies that $k\le {(D/\gamma )}^2$ . Hence, if the perceptron made a $k$ -th mistake, then $k\le {(D/\gamma )}^2$ . $\square$