[MUSIC] All right,
let's see the mean field approximation. Here is how it works. We select a family of distributions
Q as variational inference. So we select a family of
distributions in the following way. It is a set of all distributions
that are factorized over the dimension itself,
the latent variables. So, this will be a product of qi, the
distribution of the ith latent variable. And then we do an optimization by
minimizing the KL diversions between the variational distribution and
the full posterior. So here is an example, here we have
a distribution over two random variables. The true posterior is the star and the factorized one would
be the q1(z1) q2(z2). I mentioned that the true
posterior is a normal distribution with some covariance
matox sigma and the mean 0. Then the factorized distribution
would be also normal, but it would have a diagonal
convergence matrix. If this was our true posterior,
then the approximation would be like this. So not here that we don't have
the diagonal of the diagonal elements of the current matrix in the red Gaussian. So optimization works as follows. We'll start with some distribution q,
and we'll perform a coordinate descend with respect
to different elements of q. And at first side we'll optimize
with respect to q1, we'll get a new distribution, then on the second set we
optimize with respect to q2, and so on. This will be loop and
we'll optimize in two conversions. Let's now drive the formulas for
update and each step. Our plan for
now is to draw the formulas for the updates of the mean
field approximation. So, here's our step of coordinate descend. We are trying to minimize the KL
conversions with respect to one dimension, qk. Let's write down the values
of KL diversions. So this would be an integral of the product of our all dimensions, i from 1 to d, qi times the logarithm of the ratio, so logarithm. And again, the product for all dimensions, q i over p*. And we integrate it over all z. We can take out the products
from the logarithm and it will be the sum of the logarithms. And we can also take the denominator s and
as a separate integral. Now we'll have, sum for i from 1 to d, integral of the product. Again, let's write it down as j, so that we'll have separate indices. So j for 1 to d, q j logarithm of q i, d z, minus the integral of the denominator. So again, it would be integral, product over j from 1 to d, qj log p*. [SOUND] Easy. All right,
we're interested only in the component qk. So let's separate all this
summation into two terms. One with i = k and all others. So this would be equal to Integral j from 1 to d qj log qk. This is the term that we're
particularly interested in. d z and
plus sum over all other components, so it would be sum of i not equal to k,
and the same integral. So it would be the project of j from 1 to d, qj log qi dz and minus this term, product for j from 1 to d, qj log of p*dz. So let's find out which terms
are constants with respect to qk. Let's start by rewriting the first term. So it actually equals to the integral of qk times the logarithm of qk and also we can eventually
integrate all other dimensions. So here is an integral of the product over j not equal to k, qj dz, let me write it down like, not equal to j. So those are all variables except for
the k, short k here. And finally, we integrate over zk. So this term actually equals to 1,
since q is a distribution and the integral of the distribution
actually equals to 1. So this equals to 1, and finally we get the integral of qk logarithm of qk dzk. All right, so term surely depends on qk. However, these terms will
have a similar form, so those would be equal to the integral
of qj times the logarithm of qj dzj. And since these don't depend on qk, we can say that these are just constants. Those are just constant, so constant. All right, so here's our distribution
again, let's continue deriving the formula This would be equal to an integral of qk times the logarithm of qk dzk minus this integral. Let me separate the dimension
number k out from the integral. So this would be integral of qk,
here I will derive the integral over
all other dimensions. So product of over j not equal to k, qj times logarithm of p* d, so here we integrate over
oj is not equal to k, so we come right down
again is z not equal to k, and we have to close the brackets. So here d z and I guess that's it. All right,
let's now put into one integral. So we can group these as
an integral of qk times the difference between this term and
this term. So, we'll have an integral, qk times the following difference, logarithm of qk minus integral, The product for jl equal to k, qj log p*, and dz not equal to k, and finally, we close this bracket and integrate over the last dimension, zk. All right, what is this term? This term actually equals
to the expectation of logarithmic p* over all
dimensions except for the k. So we have this term to be equal to the expected value for q except for the k, so let me write it down like this. The logarithm of p*. So let's write it down as
some function of the fk. For example, h(zk). So actually, we can turn it out, we can modify it a little bit and
get a distribution. For this, we'll just have to
renormalize the exponent of. So, let's have some new
distribution that equals to exponent of h(zk), and
had to renormalized it. So how to integrate over zk, e to the h(zk) prime, dzk. This will be our new distribution,
let's call it t. Actually, I should write here + const. And here also has some constant. Now, we can notice that
it actually equals to the KL diversions,
between distributions qk and t. So this will be an integral of qk, the lower end of the ratio between qk and t, dzk, plus some new constant since we don't have here the denominator. So, plus some new constant. And we try to minimize it. So, again this is a KL divergence between qk and t + const. So we just have to
minimize the KL diversion. And so we already know what the answer is, we have to take the qk equal to
the distribution t as written here. So, qk goes through t, the better way to write
it down is as follows. So actually while we say is
that those two terms are equal, we say that this term equals to this term. And so we have the log qk = h(zk), there is the expectation over all variable except for the k, so expectation over q without k log p* + const. And so this is our final formula. We'll see how to use it in the next video. [MUSIC]