[MUSIC] Continuing our Story so
far we've shown that if we know the cluster parameters then the soft
assignments are very easy to compute. Well, now lets consider the alternative. Let's assume that we know
the cluster assignments and we want to infer our cluster parameters. And for this part, we're going to assume that we have hard
assignments of observations to clusters. And in pictures, these hard assignments
are showed by the colors of each data point and
when we have these hard assignments. We just have three different colors in
this plot rather than the spectrum of colors that we showed earlier when we were
talking about having soft assignments. But once we've fixed these cluster
assignments, a question is. Does this data point here, which is
assigned to this green cluster influence our parameter estimation problem for estimating the parameters of
the fuchsia cluster or blue cluster? And the answer is no. Only the observations that
are assigned to a given cluster inform the parameters of that cluster. We saw this in k-means when we were
talking about updating the means just using the observations that
were assigned to a given cluster. But now were going to have a more general
form of an update where were not just updating the centers of these clusters,
but also the shapes. But just to emphasize,
fixing the cluster assignments means that our estimation problem decouples
over our different clusters. Let's go back to our image clustering
task and assume that me store our data, the RGB values associated with each image. As well as these hard cluster
assignments in a table as shown here. Well, the first thing that we're going to
do is we're going to split up this table based on these hard cluster assignments. So we're going to have one table of data
points that are assigned to cluster three. One table of data points
assigned to cluster two, and another table of data points
assigned to cluster one. And then we're going to consider
each one of these data tables completely independently when we're
forming our parameter estimates for each one of these clusters. So let's look for example, at the data
points associated with cluster three. When we're going to form
our parameter estimates, what we're going to do is something
called maximum likelihood estimation, which we saw in the classification course. Where we're going to search over
all possible parameter settings, and find the settings that maximize the
likelihood of our observed data under our specified model. So, remember that each one of our clusters
is specified with a Gaussian distribution that has two parameters. A mean and a covariance. Let's spend a little bit
of time talking about the form of the maximum likelihood
estimate of these two parameters. Well for the mean, the maximum likelihood estimate is
exactly equal to just the sample mean. So what we're going to do is simply
average data points that are in cluster k. So here Nk represents the number of observations in cluster k. And here we're just summing over the data
indices of points in this table that had this hard assignment to cluster 3,
for example. And we're summing up all these values. So it would be this vector. So we would just literally
sum these vectors, and then we would divide by three, The total number of observations. Not to be confused with the fact that it's also the label of this cluster
three that's not what I mean. I mean the fact that there
are three rows in this table. Okay, so that would provide
our estimate of the mean, and we denote our estimate
with this little hat here. And if you look at this equation, you
see that it's exactly the same equation that we used in k means when we
went to update the cluster centers. We looked at all data points
assigned to a given cluster and just computed the average
of those data points. Okay. But now, we also have this covariance
term, which determines the spread and orientation of the ellipses
in the Gaussian. And for this, the maximum likelihood
estimate is also given by what's called the sample covariance estimate. Where at first we're going to subtract off
from each of our data points the estimated mean, and then we're going to compute
the outer product of these data points. Only summing over again the data
points in cluster k, and dividing by the total number
of observations in cluster k. So in the Scalar case,
if we just have our estimate. This would be our estimate of
the variance in the kth cluster. This would be equal to 1 over number
of observations in that cluster, summing over observations
in cluster little k. And here instead of this transpose,
because this is a vector. We would just have scalars. So we'd have xi- mu hat k, which would
just be a single number, and then squared. And finally, for our cluster proportions,
we simply count the number of observations in cluster k and divide
by the total number of observations. And that forms our estimate of
the weight on the kth cluster. So remember, pi k was the weight
on the kth cluster, and the hat represents the fact that
this is our estimate of this. In particular,
our maximum likelihood estimate. And I want to emphasize here
that the form of pi hat k. Its not specific to mixtures of Gaussians, which is what we've been
focusing on in this module. But it would hold if you had for
example, mixtures of multinomials, or mixtures of other distributions. But of course,
when we talked about the mean and covariance estimates
on the previous slide, that was specific to having Gaussians
defining each one of our clusters. So in summary,
if we knew the cluster assignments, so the assignments of each data
point to a given cluster. Then computing the estimates of
the cluster parameters is very, very straightforward. But again,
we don't know these hard assignments. So what are we going to do? [MUSIC]