2.9 Machine learning lecture 9 course notes  (Page 3/3)

Let's work out exactly what distribution our model defines. Our random variables $z$ and $x$ have a joint Gaussian distribution

We will now find ${\mu }_{zx}$ and $\Sigma$ .

We know that $\mathrm{E}\left[z\right]=0$ , from the fact that $z\sim \mathcal{N}(0,I)$ . Also, we have that

Putting these together, we obtain

Next, to find, $\Sigma$ , we need to calculate ${\Sigma }_{zz}=\mathrm{E}\left[(z-\mathrm{E}\left[z\right]){(z-\mathrm{E}\left[z\right])}^T\right]$ (the upper-left block of $\Sigma$ ), ${\Sigma }_{zx}=\mathrm{E}\left[(z-\mathrm{E}\left[z\right]){(x-\mathrm{E}\left[x\right])}^T\right]$ (upper-right block), and ${\Sigma }_{xx}=\mathrm{E}\left[(x-\mathrm{E}\left[x\right]){(x-\mathrm{E}\left[x\right])}^T\right]$ (lower-right block).

Now, since $z\sim \mathcal{N}(0,I)$ , we easily find that ${\Sigma }_{zz}=\mathrm{Cov}\left(z\right)=I$ . Also,

In the last step, we used the fact that $\mathrm{E}\left[z{z}^T\right]=\mathrm{Cov}\left(z\right)$ (since $z$ has zero mean), and $\mathrm{E}\left[z{ϵ}^T\right]=\mathrm{E}\left[z\right]\mathrm{E}\left[{ϵ}^T\right]=0$ (since $z$ and $ϵ$ are independent, and hence the expectation of their product is the product of their expectations).Similarly, we can find ${\Sigma }_{xx}$ as follows:

Putting everything together, we therefore have that

Hence, we also see that the marginal distribution of $x$ is given by $x\sim \mathcal{N}(\mu ,\Lambda {\Lambda }^T+\Psi )$ . Thus, given a training set $\{x^{\left(i\right)};i=1,...,m\}$ , we can write down the log likelihood of the parameters:

To perform maximum likelihood estimation, we would like to maximize this quantity with respect to the parameters. But maximizing this formula explicitly is hard (try it yourself),and we are aware of no algorithm that does so in closed-form. So, we will instead use to the EM algorithm. In the next section, we derive EM forfactor analysis.

Em for factor analysis

The derivation for the E-step is easy. We need to compute $Q_i\left(z^{\left(i\right)}\right)=p\left(z^{\left(i\right)}\right|x^{\left(i\right)};\mu ,\Lambda ,\Psi )$ . By substituting the distribution given in Equation  [link] into the formulas  [link] - [link] used for finding the conditional distribution of a Gaussian, wefind that $z^{\left(i\right)}|x^{\left(i\right)};\mu ,\Lambda ,\Psi \sim \mathcal{N}({\mu }_{z^{\left(i\right)}|x^{\left(i\right)}},{\Sigma }_{z^{\left(i\right)}|x^{\left(i\right)}})$ , where

So, using these definitions for ${\mu }_{z^{\left(i\right)}|x^{\left(i\right)}}$ and ${\Sigma }_{z^{\left(i\right)}|x^{\left(i\right)}}$ , we have

Let's now work out the M-step. Here, we need to maximize

with respect to the parameters $\mu ,\Lambda ,\Psi$ . We will work out only the optimization with respect to $\Lambda$ , and leave the derivations of the updates for $\mu$ and $\Psi$ as an exercise to the reader.

We can simplify Equation  [link] as follows:

Here, the “ $z^{\left(i\right)}\sim Q_i$ ” subscript indicates that the expectation is with respect to $z^{\left(i\right)}$ drawn from $Q_i$ . In the subsequent development, we will omit this subscript when there is no risk of ambiguity.Dropping terms that do not depend on the parameters, we find that we need to maximize:

Let's maximize this with respect to $\Lambda$ . Only the last term above depends on $\Lambda$ . Taking derivatives, and using the facts that $\mathrm{tr}a=a$ (for $a\in \mathbb{R}$ ), $\mathrm{tr}AB=\mathrm{tr}BA$ , and ${\nabla }_A\mathrm{tr}AB{A}^{T}C=CAB+C^{T}AB$ , we get:

Setting this to zero and simplifying, we get:

Hence, solving for $\Lambda$ , we obtain

It is interesting to note the close relationship between this equation and the normal equation that we'd derived for least squares regression,

The analogy is that here, the $x$ 's are a linear function of the $z$ 's (plus noise). Given the “guesses” for $z$ that the E-step has found, we will now try to estimate the unknown linearity $\Lambda$ relating the $x$ 's and $z$ 's. It is therefore no surprise that we obtain something similar to the normal equation. There is, however,one important difference between this and an algorithm that performs least squares using just the “best guesses” of the $z$ 's; we will see this difference shortly.

To complete our M-step update, let's work out the values of the expectations in Equation  [link] . From our definition of $Q_i$ being Gaussian with mean ${\mu }_{z^{\left(i\right)}|x^{\left(i\right)}}$ and covariance ${\Sigma }_{z^{\left(i\right)}|x^{\left(i\right)}}$ , we easily find

The latter comes from the fact that, for a random variable $Y$ , $\mathrm{Cov}\left(Y\right)=\mathrm{E}\left[Y{Y}^T\right]-\mathrm{E}\left[Y\right]\mathrm{E}{\left[Y\right]}^T$ , and hence $\mathrm{E}\left[Y{Y}^T\right]=\mathrm{E}\left[Y\right]\mathrm{E}{\left[Y\right]}^T+\mathrm{Cov}\left(Y\right)$ . Substituting this back into Equation  [link] , we get the M-step update for $\Lambda$ :

It is important to note the presence of the ${\Sigma }_{z^{\left(i\right)}|x^{\left(i\right)}}$ on the right hand side of this equation. This is the covariance in the posterior distribution $p\left(z^{\left(i\right)}\right|x^{\left(i\right)})$ of $z^{\left(i\right)}$ give $x^{\left(i\right)}$ , and the M-step must take into account this uncertainty about $z^{\left(i\right)}$ in the posterior. A common mistake in deriving EM is to assume that in the E-step, we need to calculate only expectation $E\left[z\right]$ of the latent random variable $z$ , and then plug that into the optimization in the M-step everywhere $z$ occurs. While this worked for simple problems such as the mixture of Gaussians, in ourderivation for factor analysis, we needed $E\left[z{z}^T\right]$ as well $\mathrm{E}\left[z\right]$ ; and as we saw, $E\left[z{z}^T\right]$ and $\mathrm{E}\left[z\right]\mathrm{E}{\left[z\right]}^T$ differ by the quantity ${\Sigma }_{z|x}$ . Thus, the M-step update must take into account the covariance of $z$ in the posterior distribution $p\left(z^{\left(i\right)}\right|x^{\left(i\right)})$ .

Lastly, we can also find the M-step optimizations for the parameters $\mu$ and $\Psi$ . It is not hard to show that the first is given by

Since this doesn't change as the parameters are varied (i.e., unlike the update for $\Lambda$ , the right hand sidedoes not depend on $Q_i\left(z^{\left(i\right)}\right)=p\left(z^{\left(i\right)}\right|x^{\left(i\right)};\mu ,\Lambda ,\Psi )$ , which in turn depends on the parameters), this can be calculated just once and needs not be further updated as the algorithm isrun. Similarly, the diagonal $\Psi$ can be found by calculating

and setting ${\Psi }_{ii}={\Phi }_{ii}$ (i.e., letting $\Psi$ be the diagonal matrix containing only the diagonal entries of $\Phi$ ).