1.13 Machine learning lecture 14  (Page 4/13)

So it turns out that in order to implement the E step and the M step, there are just two little things – there are actually sort of three key things that are different from the models that you saw previously, so I wanna talk about them. The first one is that the first of the [inaudible] which is that in the E step, Z is now a continuous value random variable, so you now need a way to represent these continuous value densities, probably density functions to represent QI of Z. So fortunately in this probably it’s not difficult to do, and in particular the conditional distribution of ZI given XI and our parameters which I’m gonna omit from this equation is going to be Gaussian with mean and covariance given by these two things, so I write like that, where this is going to be equal to the vector zero minus –

And the way I got this formula was – if you match this to the formula I had previously for computing conditional distributions of Gaussians, this corresponded to the terms mu one minus sigma one two sigma two two inverse times two minus mu. So those are the terms corresponding to the form I had previously for computing the marginal distributions of a Gaussian. And that is going to be given by that, and again those are the terms corresponding to the formulas I had for the very first thing I did, the formulas for computing conditional distributions of Gaussians. And so this is E step. You need to compute the Q distribution. You need to compute QI of ZI, and to do that what you actually do is you compute this vector mu of ZI given XI and sigma of ZI given XI, and together these represent the mean and covariance of the distribution Q where Q is going to be Gaussian, so that’s the E step. Now here’s the M step then. I’ll just mention there are sort of two ways to derive M steps for especially Gaussian models like these. Here’s the key trick I guess which is that when you compute the M step, you often need to compute integrals that look like these. And then there’ll be some function of ZI. Let me just write ZI there, for instance.

So there are two ways to compute integrals like these. One is if you write out – this is a commonly made mistake in – well, not mistake, commonly made unnecessary complication I guess. One way to try to compute this integral is you can write this out as integral over ZI. And while we know what QI is, right? QI is a Gaussian so two pi, so D over two – the covariance of QI is this sigma given XI which you’ve computed in the E step, and then times E to the minus one half ZI minus mu of ZI given XI transpose sigma inverse – so that’s my Gaussian density. And then times ZI ZZI. And so writing those out is the unnecessarily complicated way to do it because once you’ve written out this integral, if you want to actually integrate – that’s the times, multiplication – if you actually want to evaluate this integral, it just looks horribly complicated. I’m not quite sure how to evaluate this. By far, the simpler to evaluate this integral is to recognize that this is just the expectation with respect to ZI drawn from the distribution QI of the random variable ZI. And once you’ve actually got this way, you notice that this is just the expected value of ZI under the distribution QI, but QI is a Gaussian distribution with mean given by that mu vector and covariance given by that sigma vector, and so the expected value of ZI under this distribution is just mu of ZI given XI. Does that make sense? So by making those observations that this is just an expected value, there’s a much easier way to compute that integral. [Inaudible]. So we’ll apply the same idea to the M step. So the M step we want to maximize this. There’s also sum over I – there’s a summation over I outside, but this is essentially the term inside the arg max of the M step where taking the integral of a ZI and just observe that that’s actually – I guess just form some expectation respect to the random variable ZI of this thing inside. And so this simplifies to the following.