In a previous video,
we derived the formulas for updating the Q of theta.
In this video, we'll derive the formulas for updating Q of Z.
So, I'll remind you that we're doing
the variational exploitation organization algorithm. And then E-step.
When you try to minimize the KL divergence
between the posterior distribution over the latent variables given
the data and all our ration distribution that we are searching in the fit for S for.
And we try to apply the formula that we derived for the new field approximation.
All right. Q of Z.
So the formula would be again the logarithm of Q of
Z equals to the expectation over all variables except for the Z and then there is theta.
So, Q of theta.
Logarithm of the journal distribution.
Probability of theta, Z and W plus some constant.
All right.
Now we have to plug in the formula from above into this function.
Well, before we do this let's see which terms are constants.
We are trying to estimate the distribution over Z.
So, let's see which terms do not depend on Z.
So, the first term does not depend on Z so we can say that it is constant.
What about this term?
So this term actually it depends on Z since we have it here.
And those two, some of those would also depend on Z. All right.
So let's rewrite this formula here and see what we get.
So, it would be that expectation and with respect to Q of theta.
Sum over all documents.
Now we have also sum over all words.
And from one to D-N. And also the sum over all topics.
Sum over T from one to capital-T. All right,
the indicator that ZDN equals to
T. And now we have the logarithm of theta DT,
plus the logarithm of
FI TWDN, plus non-constant.
Okay. Now let's take the expectation and put it under the summation.
So expectation is taken with respect to theta.
This term doesn't depend on theta,
this doesn't depend either.
So we can take the expectation here.
All right. So, now I write sum over three variables,
sum over D, over documents, sum over words.
And some over topics.
The indicator, that does not depend on theta so we put the expectation further.
The expectation of the logarithm
of theta DT.
Plus the logarithm of FI TWDN.
And again plus non-constant.
All right. This is the logarithm of the distribution over Z.
Let us take the exponent of the left hand side and
the right hand side and we'll have the Q of Z.
Actually equals to the products since we take
the exponent the summations become products.
D from 1 to D. Product of N from one to ND.
Now, here is the trick. We know that Z,
sum of 21 over T. Since we assigned only one topic for a word.
And so since here we have summation and here we have the extra distribution over ZDN.
We can see that we can write down
the distribution Q of Z as a product of independent distributions.
So it would be Q of ZDN.
And Q of ZDN can be derived using this term. Let's do it.
The probability is that ZDN equals to
T would be proportional to the exponent of this term.
So we can write down it as follows.
So, it is FI TWDN,
times the exponent of the expectation.
Here we can write
down the expectation only with respect to theta DT since the thing
that we're going expectation of depends only on theta DT.
So, Q of theta DT of the logarithm of theta DT.
What would be the memorization constant?
So actually this thing should sum up to one.
So we can compute the summation over the numerator with respect to
all possible values of T. And they're only capital-T possible ways,
possible values of T. So,
this would be t from 1 to capital-T. Of the same thing.
So that down is prime here.
FI T prime WDN.
Exponent of expectation of Qdd.
Logarithm of Qdt.
Also know that this thing equals to gamma DN at position t.
You can see here the definition.
All right so now we know that this formula's for Q of Z,
we also know the values of gamma DN and so we can iterate the updates of theta and Z.
So, we start up for example with some rather than citation.
We try to update Q of theta.
We update it with this formula.
Where gamma are computed on the previous step.
So for this case those are the initial values of gamma.
Then we go through this step where we update Q of Z and we update it with this formula.
Here we need to compute the expected values logarithm.
You can see the value of this function on the Wikipedia.
So, this would be the expected failure of the stated distribution.
So, we completed the formulas for the E-step and in
the next video we'll derive the updates formulas for the M-step and the prediction.