2.4 Machine learning lecture 5 course notes

Regularization and model selection

Suppose we are trying to select among several different models for a learning problem. For instance, we might be using a polynomialregression model $h_{\theta }\left(x\right)=g({\theta }_0+{\theta }_{1}x+{\theta }_2{x}^2+\cdots +{\theta }_k{x}^k)$ , and wish to decide if $k$ should be 0, 1, ..., or 10. How can we automatically select amodel that represents a good tradeoff between the twin evils of bias and variance Given that we said in the previous set of notes that bias and variance are two very differentbeasts, some readers may be wondering if we should be calling them “twin” evils here. Perhaps it'd be better to think of them as non-identical twins.The phrase “the fraternal twin evils of bias and variance” doesn't have the same ring to it, though. ? Alternatively, suppose we want toautomatically choose the bandwidth parameter $\tau$ for locally weighted regression, or the parameter $C$ for our ${\ell }_1$ -regularized SVM. How can we do that?

For the sake of concreteness, in these notes we assume we have some finite set of models $\mathcal{M}=\{M_1,...,M_d\}$ that we're trying to select among. For instance, in our first example above, the model $M_i$ would be an $i$ -th order polynomial regression model. (The generalization to infinite $\mathcal{M}$ is not hard. If we are trying to choose from an infinite set of models, say corresponding to the possible values of the bandwidth $\tau \in {\mathbb{R}}^{+}$ , we may discretize $\tau$ and consider only a finite number of possible values for it. More generally, most of the algorithms described herecan all be viewed as performing optimization search in the space of models, and we can perform this search over infinite modelclasses as well. ) Alternatively, if we are trying to decide between using an SVM,a neural network or logistic regression, then $\mathcal{M}$ may contain these models.

Cross validation

Let's suppose we are, as usual, given a training set $S$ . Given what we know about empirical risk minimization,here's what might initially seem like a algorithm, resulting from using empirical risk minimization for model selection:

1. Train each model $M_i$ on $S$ , to get some hypothesis $h_i$ .
2. Pick the hypotheses with the smallest training error.

This algorithm does not work. Consider choosing the order of a polynomial. The higher the order of the polynomial, the betterit will fit the training set $S$ , and thus the lower the training error. Hence, this method will always select a high-variance, high-degree polynomialmodel, which we saw previously is often poor choice.

Here's an algorithm that works better. In hold-out cross validation (also called simple cross validation ), we do the following:

1. Randomly split $S$ into $S_{\mathrm{train}}$ (say, 70% of the data) and $S_{\mathrm{cv}}$ (the remaining 30%). Here, $S_{\mathrm{cv}}$ is called the hold-out cross validation set.
2. Train each model $M_i$ on $S_{\mathrm{train}}$ only, to get some hypothesis $h_i$ .
3. Select and output the hypothesis $h_i$ that had the smallest error ${\widehat{\epsilon }}_{S_{\mathrm{cv}}}\left(h_i\right)$ on the hold out cross validation set. (Recall, ${\widehat{\epsilon }}_{S_{\mathrm{cv}}}\left(h\right)$ denotes the empirical error of $h$ on the set of examples in $S_{\mathrm{cv}}$ .)

By testing on a set of examples $S_{\mathrm{cv}}$ that the models were not trained on, we obtain a better estimate of each hypothesis $h_i$ 's true generalization error, and can then pick the one with the smallest estimated generalization error.Usually, somewhere between $1/4-1/3$ of the data is used in the hold out cross validation set, and 30% is a typical choice.