[MUSIC]
Well, to move towards something that we might be able to do, let's talk about
what would change if we only knew the soft assignments instead of hard assignments?
That is, how would we estimate our cluster parameters
from a set of soft assignments of observations to our set of clusters?
Well now, instead of having an observation in a single cluster and
no other cluster, each observation is really going to be in every cluster.
It's going to have some split allocation across our different clusters
based on the responsibility vector, the soft assignment of that observation.
And so we're going to think about how to update our cluster parameters
based on allocating a single observation to our entire set of clusters.
So when we go to do maximum likelihood estimation from soft assignments,
what we're going to do is we're going to form a data table where for
every observation, every row of this table, we're going to introduce
a set of weights that correspond to the responsibility vector.
So this looks very similar to what we did in boosting with weighted observations.
But now instead of having a single weight associated with each row, each different
data point, we have a set of weights and these set of weights sum to one.
But just like in boosting with weighted observations,
these weights are going to modify our row
operations that we're performing in doing our maximum likelihood estimation.
But when we have these split allocations of data points across our
multiple clusters, it's hard to think about counting observations in a cluster.
So in this case, what we can think about doing is computing the total weight in
each one of these clusters just some in over the responsibilities in the cluster.
And think of this as the effective number of observations
in each one of our clusters.
And then what we're going to do is pretty similar to what we did when we had hard
assignments where we're going to form cluster specific data tables.
But now, instead of dividing up our data table into here are the ones in cluster
one, here are the ones in cluster two, cluster three,
every observation is going to appear in each of these data tables because there is
some responsibility taken by each cluster for that observation.
Of course it could be zero, but generically,
let's think of some responsibility for cluster to a given observation.
And so we're going to form a table where we specify the cluster weights
associated with each one of our observations.
So here would be the table associated with cluster one and
here's the table for cluster two and cluster three.
Then for each one of our clusters, we're going to compute a maximum likelihood
estimate of the cluster parameter but
with these weights modifying the row operations that we're doing.
So here are updated maximum likelihood estimates where we're accounting for
weights on our observation which we see here.
So in particular we see that every row operations.
So every time we're touching a given
data element xi it's going to be multiplied by rik.
And the other thing I want to know is that when we're computing the total
number of observations in the cluster, now we're going to call this Nk
soft, based on our set of soft assignment.
And this is going to be the effective number of observations in that cluster
which is just the sum of the responsibilities in that cluster.
We do a similar modification when we go to estimate our cluster proportions.
Where now instead of calculating the total number of observations in a cluster,
we compute the effective number of observations in that cluster.
And then we simply divide by the total number of observations to the data set
which is equivalent to the total effective number of observation,
just the sum of this vector here.
So an equations are updated estimate of our cluster
proportions is written as follows where we simply replace Nk with this Nksoft.
Note that if our response for these just took values zero or one,
that is if we were making hard assignments of observations to a cluster.
Then things would just default back to what we showed in the previous section.
So in particular we can think of this responsibility vector which
now just has a single one in one of these clusters
as representing a one-hot encoding of the cluster assignment.
So to see the equivalence between what we have here and
what we showed in the previous section we can look at our set of equations for
our maximum likely or the estimates based on soft assignments.
And we can show that if we actually plug in hard assignments
then we get out the set of equations we presented in section 2A.
So to begin with we can look at our estimate of the cluster proportions and
note that if rik, just takes values in the set zero or one.
Then what this sum is going to be is we're just going to count.
Observation i
In cluster k
if rik equals one.
So this sum here is just going to default to counting the number
of observations in cluster k.
And that's exactly what we had before, and now if we go to our estimate of the mean.
And when we think about multiplying by rik,
we're only going to be adding xi to this sum,
we remember here our sum is over all the data points if xi is in cluster k.
So only add xi if i is in k,
that is if rik equals one.
And in this case this equation would reduce to one over
Nk sum i in cluster k xi which again is exactly what we had before.
And finally for estimate of the covariance it's going to
be the same as above where we're only going to be adding
these outer products if observation i is in cluster k.
And things are going to default to the same equation
as in the hard assignment case.
And just to make this explicit,
it's 1 over NK sum i in k, xi minus mu hat k,
xi minus mu hat k transpose.
So this at least serves as a little sanity check that the equations that we presented
in this section for our maximum likelihood estimates based on soft assignments
might at least not be wrong because we showed that if we plug in a hard
assignment we got out the maximum likely or
the estimates that we knew to be true from before.
What we've seen in this is that based just on soft assignments it's still
straight forward to compute our estimates of the cluster parameters.
[MUSIC]