0.13 Maximum likelihood estimation

In the last lecture we derived a risk (MSE) bound for regression problems; i.e., select an $f\in \mathcal{F}$ so that $E\left[{(f\left(X\right)-Y)}^2\right]-E\left[{(f^{*}\left(X\right)-Y)}^2\right]$ is small, where $f^{*}\left(x\right)=E\left[Y\right|X=x]$ . The result is summarized below.

Complexity regularization with squared error loss

Let $\mathcal{X}={\mathbb{R}}^d$ , $\mathcal{Y}=[-b/2,b/2]$ , ${\{X_i,Y_i\}}_{i=1}^n$ iid, $P_{XY}$ unknown, $\mathcal{F}$ = {collection of candidate functions},

Let $c\left(f\right)$ , $f\in \mathcal{F}$ , be positive numbers satisfying ${\sum }_{f\in \mathcal{F}}{2}^{-c\left(f\right)}\le 1$ , and select a function from $\mathcal{F}$ according to

with $ϵ\le \frac{3}{5b^2}$ and ${\widehat{R}}_n\left(f\right)=\frac{1}{n}{\sum }_{i=1}^n{(f\left(X_i\right)-Y_i)}^2$ . Then,

where $\alpha =\frac{ϵb^2}{1-2b^2ϵ/3}$ .

Maximum likelihood estimation

The focus of this lecture is to consider another approach to learning based on maximum likelihood estimation. Consider the classical signalplus noise model:

where $W_i$ are iid zero-mean noises. Furthermore, assume that $W_i\sim P\left(w\right)$ for some known density $P\left(w\right)$ . Then

since $Y_i-f\left(\frac{i}{n}\right)=W_i$ .

A very common and useful loss function to consider is

Minimizing ${\widehat{R}}_n$ with respect to $f$ is equivalent to maximizing

or

Thus, using the negative log-likelihood as a loss function leads to maximum likelihood estimation. If the $W_i$ are iid zero-mean Gaussian r.v.s then this is just the squared error loss weconsidered last time. If the $W_i$ are Laplacian distributed e.g. $P\left(w\right)\propto e^{-\left|w\right|}$ , then we obtain the absolute error, or $L_1$ , loss function. We can also handle non-additive models such as thePoisson model

In this case

which is a very different loss function, but quite appropriate for many imaging problems.

Before we investigate maximum likelihood estimation for model selection, let's review some of the basic concepts. Let $\Theta$ denote a parameter space (e.g., $\Theta =R$ ), and assume we have observations

where ${\theta }^{*}\in \Theta$ is a parameter determining the density of the { $Y_i$ }. The ML estimator of ${\theta }^{*}$ is

$\widehat{\theta }$ maximizes the expected log-likelihood. To see this, let's compare the expected log-likelihood of ${\theta }^{*}$ with any other $\theta \in \Theta$ .

Why?

On the other hand, since ${\widehat{\theta }}_n$ maximizes the likelihood over $\theta \in \Theta$ , we have

Therefore,

or re-arranging

Notice that the quantity

is an empirical average whose mean is $K(P_{\theta },P_{{\theta }^{*}})$ . By the law of large numbers, for each $\theta \in \Theta$ ,

If this also holds for the sequence $\left\{{\widehat{\theta }}_n\right\}$ , then we have

which implies that

which often implies that

in some appropriate sense (e.g., point-wise or in norm).

Hellinger distance

The KL divergence is not a distance function.

Therefore, it is often more convenient to work with the Hellinger metric,

The Hellinger metric is symmetric, non-negative and

and therefore it is a distance measure. Furthermore, the squared Hellinger distance lower bounds the KL divergence, so convergence in KL divergence implies convergence of the Hellinger distance.

Proposition 1

Proof:

Note that in the proof we also showed that

and using the fact $logx\le x-1$ again, we have

The quantity inside the log is called the affinity between $P_{{\theta }_1}$ and $P_{{\theta }_2}$ :

This is another measure of closeness between $P_{{\theta }_1}$ and $P_{{\theta }_2}$ .