[MUSIC] In this video,
we will derive the formulas for the model. That is we want to try to train the model
by finding the optimal values of phi. We'll do this by
maximizing the likelihood. So we would try to have the probability
of our data, given the matrix phi. And to maximize it with respect to phi. We'll try to apply the variational
Algorithm for this purpose. So in variatiional EL we have two steps. We'll try to minimize
the counter versions, between the variation distribution, and this case we try to find
the solution in the following form. It would be the distribution over
theta times the distribution over Z. Between these and
the probability of theta and Z, the posterior probability
on them given the data. And minimize it. We'll just get back to theta,
with Q of theta and Q of Z. All right, this is an E-step and
an M-step. We maximize the expected value Of the logarithm, Of the joint probability of theta z and w, Maximize it with respect to theta,
with respect to phi. So our plan for now is to draw
the formulas for the E step, that is we want to find the q of theta and
q of z. Let's start with q of theta. All right, so let's start with theta. The formulas for theta are as follows. So the log of q of theta. Those are formulas from
the means field approximation. Equals to the expected value with
respect to, all variables except for theta and the only variable
that we have left is z, so with respect to q of z,
the logarithm of this distribution Log of p of theta z
given w + some constant. So we actually don't know this term. However, we can rewrite
it using Bay's formula. So it would be equal to the ratio
between the joint probability, actually this isn't base formula, it is just the definition of
the conditional probability over the probabiilty of W,
from this we have W. So this term the denominator
does not depend on Q of theta, so it's just a constant. And so we can rewrite it as expectation. Of q of c, the logarithm of the joined distribution over the theta c and
w plus a constant. All right, so we are estimating
a distribution over theta so before we plug in this value
from this huge formula. Let's see which terms are constant
with respect to theta. So this term depends on theta. We have theta here. We have theta here. However in this one we don't have theta. So this is actually a constant. All right, so,
actually we still have a lot of terms. I'll try and, Solve expectation with respect to q of z. And now we write out these terms. So it is the sum over all documents. Then we have bracket, sum over all topics. Alpha (t-1) logarithm of theta dt + the sum over hours n topics. So, this would be n=1 to Nd, number of words in the document. Sum over all topics that can
be assigned to each word, indicator that the written
variable has the value t. So in this case we assign the topic
t to the word n of the document d. Times a logarithm of theta dt as well,
I already have it here. And we have to close this bracket. All right, we have an expectation and we can take it under the summation. So this term does not depend on z. And here only dependence of z is here. So it will be the sum over all documents. This term will not change. So the sum for t,
from 1 to t over all topics, alpha t minus 1,
logarithm of theta dt plus, then we take the expectation here, so this will be the sum over our hours
[SOUND] the sum over all topics. Expectation with respect to q of zdn of the indicator. And finally,
times the logarithm of theta dt. [SOUND] All right. So, let's note this value, the expectation as gamma dn Now we can group the terms
that have the value of of theta dt, and
get the following expression. We'll have a sum over all documents. Sum over all topics. We'll have alpha t- 1 from this term. We'll have the sum over
all hours of the gamma dn. So summation gamma dn, and finally times the logarithm of theta d t. And actually we should add
a constant here, here and here. All right, so what is this distribution? Do you recognize the form of it. Actually this is a [INAUDIBLE]
distribution again. So if we take the exponent of this
function to get the q of theta we will see that it equals to the product over all documents, [SOUND], product over all topics, theta dt, the power would be in this term alpha t plus sum of gamma d n-1. And times some constant. So we can write the proportional
sign instead of equality. So actually this is
a [INAUDIBLE] distribution. We can write down [INAUDIBLE] the q
of theta equals to the product over all documents,
The distribution over theta d, and since that is a stanard distribution. So you recognise,
you should recognise the form here. Q of theta d would be a distribution. With the following parameters. So we should add one to the power and we will have the following form. So this would be vector alpha. Also vector of summations of gamma dn. So as a parameter of this distribution, we sum up the vector alpha and
the vector sum of gamma d n. So gamma d n itself depends on t
since here we had a value of t. Alright so we derive the update
formula for q of theta. Also know that here we use
the value of gamma dn. And the gamma dn can be approximated only using the distribution q of z. So before we have the training algorithm, we have to derive the value,
the logarithm of q of z, and also we'll have to compute the expectation
of this distribution of zdn equals t. [MUSIC].