2.2 Machine learning lecture 3 course notes  (Page 12/12)

Thus, if we want to update some subject of the ${\alpha }_i$ 's, we must update at least two of them simultaneously in order to keep satisfying the constraints. This motivatesthe SMO algorithm, which simply does the following:

• Repeat till convergence $\{$
  1. Select some pair ${\alpha }_i$ and ${\alpha }_j$ to update next (using a heuristic that tries to pick the two that will allow us to make the biggest progress towards the global maximum).
  2. Reoptimize $W\left(\alpha \right)$ with respect to ${\alpha }_i$ and ${\alpha }_j$ , while holding all the other ${\alpha }_k$ 's ( $k\ne i,j$ ) fixed.
• $\}$

To test for convergence of this algorithm, we can check whether the KKT conditions ( [link] ) are satisfied to within some $\mathit{t}ol$ . Here, $\mathit{t}ol$ is the convergence tolerance parameter, and is typically set to around 0.01 to 0.001. (See the paper and pseudocode for details.)

The key reason that SMO is an efficient algorithm is that the update to ${\alpha }_i$ , ${\alpha }_j$ can be computed very efficiently. Let's now briefly sketch the main ideas for deriving the efficient update.

Let's say we currently have some setting of the ${\alpha }_i$ 's that satisfy the constraints in [link] , and suppose we've decided to hold ${\alpha }_3,...,{\alpha }_m$ fixed, and want to reoptimize $W({\alpha }_1,{\alpha }_2,...,{\alpha }_m)$ with respect to ${\alpha }_1$ and ${\alpha }_2$ (subject to the constraints). From the final equation in [link] , we require that

Since the right hand side is fixed (as we've fixed ${\alpha }_3,...{\alpha }_m$ ), we can just let it be denoted by some constant $\zeta$ :

We can thus picture the constraints on ${\alpha }_1$ and ${\alpha }_2$ as follows:

From the constraints  [link] , we know that ${\alpha }_1$ and ${\alpha }_2$ must lie within the box $[0,C]\times [0,C]$ shown. Also plotted is the line ${\alpha }_1{y}^{\left(1\right)}+{\alpha }_2{y}^{\left(2\right)}=\zeta$ , on which we know ${\alpha }_1$ and ${\alpha }_2$ must lie. Note also that, from these constraints, we know $L\le {\alpha }_2\le H$ ; otherwise, $({\alpha }_1,{\alpha }_2)$ can't simultaneously satisfy both the box and the straight line constraint. In this example, $L=0$ . But depending on what the line ${\alpha }_1{y}^{\left(1\right)}+{\alpha }_2{y}^{\left(2\right)}=\zeta$ looks like, this won't always necessarily be the case; but more generally, there will besome lower-bound $L$ and some upper-bound $H$ on the permissable values for ${\alpha }_2$ that will ensure that ${\alpha }_1$ , ${\alpha }_2$ lie within the box $[0,C]\times [0,C]$ .

Using Equation  [link] , we can also write ${\alpha }_1$ as a function of ${\alpha }_2$ :

(Check this derivation yourself; we again used the fact that $y^{\left(1\right)}\in \{-1,1\}$ so that ${\left(y^{\left(1\right)}\right)}^2=1$ .) Hence, the objective $W\left(\alpha \right)$ can be written

Treating ${\alpha }_3,...,{\alpha }_m$ as constants, you should be able to verify that this is just some quadratic function in ${\alpha }_2$ . I.e., this can also be expressed in the form $a{\alpha }_2^2+b{\alpha }_2+c$ for some appropriate $a$ , $b$ , and $c$ . If we ignore the “box” constraints  [link] (or, equivalently, that $L\le {\alpha }_2\le H$ ), then we can easily maximize this quadratic function by setting its derivative to zero and solving. We'll let ${\alpha }_2^{\mathit{n}ew,unclipped}$ denote the resulting value of ${\alpha }_2$ . You should also be able to convince yourself that if we had instead wanted to maximize $W$ with respect to ${\alpha }_2$ but subject to the box constraint, then we can find the resulting value optimal simply by taking ${\alpha }_2^{\mathit{n}ew,unclipped}$ and “clipping” it to lie in the $[L,H]$ interval, to get

Finally, having found the ${\alpha }_2^{\mathit{n}ew}$ , we can use Equation  [link] to go back and find the optimal value of ${\alpha }_1^{\mathit{n}ew}$ .

There're a couple more details that are quite easy but that we'll leave you to read about yourself in Platt's paper: One is the choice of the heuristics usedto select the next ${\alpha }_i$ , ${\alpha }_j$ to update; the other is how to update $b$ as the SMO algorithm is run.