Machine learning lecture 1 course notes  (Page 6/13)

Note that by the independence assumption on the ${ϵ}^{\left(i\right)}$ 's (and hence also the $y^{\left(i\right)}$ 's given the $x^{\left(i\right)}$ 's), this can also be written

Now, given this probabilistic model relating the $y^{\left(i\right)}$ 's and the $x^{\left(i\right)}$ 's, what is a reasonable way of choosing our best guess of the parameters $\theta$ ? The principal of maximum likelihood says that we should should choose $\theta$ so as to make the data as high probability as possible. I.e., we should choose $\theta$ to maximize $L\left(\theta \right)$ .

Instead of maximizing $L\left(\theta \right)$ , we can also maximize any strictly increasing function of $L\left(\theta \right)$ . In particular, the derivations will be a bit simpler if we instead maximize the log likelihood $\ell \left(\theta \right)$ :

Hence, maximizing $\ell \left(\theta \right)$ gives the same answer as minimizing

which we recognize to be $J\left(\theta \right)$ , our original least-squares cost function.

To summarize: Under the previous probabilistic assumptions on the data, least-squares regression corresponds to finding the maximum likelihoodestimate of $\theta$ . This is thus one set of assumptions under which least-squares regression can be justified as a very natural method that'sjust doing maximum likelihood estimation. (Note however that the probabilistic assumptions are by no means necessary for least-squares to be a perfectly good and rational procedure, and there may—and indeedthere are—other natural assumptions that can also be used to justify it.)

Note also that, in our previous discussion, our final choice of $\theta$ did not depend on what was ${\sigma }^2$ , and indeed we'd have arrived at the same result even if ${\sigma }^2$ were unknown. We will use this fact again later, when we talk about theexponential family and generalized linear models.

Locally weighted linear regression

Consider the problem of predicting $y$ from $x\in \mathbb{R}$ . The leftmost figure below shows the result of fitting a $y={\theta }_0+{\theta }_{1}x$ to a dataset. We see that the data doesn't really lie on straight line, and so the fit isnot very good.

Instead, if we had added an extra feature $x^2$ , and fit $y={\theta }_0+{\theta }_{1}x+{\theta }_2{x}^2$ , then we obtain a slightly better fit to the data. (See middle figure) Naively, it might seem that the morefeatures we add, the better. However, there is also a danger in adding too many features: The rightmost figure is the result of fitting a 5-th order polynomial $y={\sum }_{j=0}^5{\theta }_j{x}^j$ . We see that even though the fitted curve passes through the data perfectly, we would not expect this to be a very goodpredictor of, say, housing prices ( $y$ ) for different living areas ( $x$ ). Without formally defining what these terms mean,we'll say the figure on the left shows an instance of underfitting —in which the data clearly shows structure not captured by the model—and the figure on the right is an example of overfitting . (Later in this class, when we talk about learning theory we'll formalizesome of these notions, and also define more carefully just what it means for a hypothesis to be good or bad.)

As discussed previously, and as shown in the example above, the choice of features is important to ensuring good performance of a learning algorithm.(When we talk about model selection, we'll also see algorithms for automatically choosing a good set of features.) In this section, let us talkbriefly talk about the locally weighted linear regression (LWR) algorithm which, assuming there is sufficient training data, makes the choice of featuresless critical. This treatment will be brief, since you'll get a chance to explore some of the properties of the LWR algorithm yourself in the homework.