In this video, I'm going to talk about the mixture of experts model that was
developed in the early 1990s. The idea of this model is to train a
number of neural nets, each of which specializes in a different part of the
data. That is, we assume we have a data set
which comes from a number of different regimes,
And we train a system in which one neural net will specialize in each regime, and a
managing neural net will look at the input data, and decide which specialist to give
it to. This kind of system, doesn't make very
efficient use of data, because the data is, fractionated over all these different
experts. And so with small data sets, it can't be
expected to do very well. But as data sets get bigger, this kind of
system may well come into its own, because it can make very good use of extremely
large data sets. In boosting, the weights on the models are
not all equal, But after we finish training, each model
has the same weight for every test case. We don't make the weights on the
individual models depend on which particular case we're dealing with.
In mixture of experts, we do. So the idea is that we can look at the
input data for a particular case during both training and testing to help us
decide which model we can rely on. During training this will allow models to
specialize on a subset of the cases. They then will not learn on cases for
which they're not picked. So they can ignore stuff they're not good
at modeling. This will lead to individual models that
are very good at some things and very bad at other things.
The key idea is to make each model, or expert as we call it, focus on predicting
the right answer for cases where it's already doing better than the other
experts. That will cause specialization.
So there's a spectrum of models from very local models to very global models.
Nearest neighbors, for example, is a very local model.
To fit it, you just store the training cases.
So, that's really simple, And then if you have to predict Y from X,
you simply find the stored value of X that's closest to the test value of X,
then you predict the value of Y that's the same as for the stored value.
The result of that is that the curve relating the input to the output consists
of lots of horizontal lines connected by cliffs.
It would clearly make more sense to smooth things out a bit.
At the other extreme, we have fully global models, like fitting one polynomial to all
the data. They're much harder to fit to data, and
they may also be unstable. That is, small changes in the data may
cause big changes in the model you fit. That's because each parameter depends on
all the data. In between these two ends of the spectrum,
we have multiple local models, that are of intermediate complexity.
This is good if the data set contains several different regimes and those
different regimes have different input/output relationships.
In financial data for example the state of the economy has a big effect on
determining the mappings between inputs and outputs, and you might want to have
different models for different states of the economy.
But you might not know in advance how to decide what constitutes different states
of the economy, you're going to have to learn that too.
So we have this problem if we're going to use different models for different
regimes, of how do we partition the data session to these different regimes.
In order to fit different models to different regimes we need to cluster the
training data into subsets, one for each of these regimes.
But we don't want to cluster the data based on the similarity of input vectors.
All we're interested in is the similarity of input-output mappings.
So if you look at the case on the right, there's four data points that are nicely
fitted by the red parabola and another four data points that are nicely fitted by
the green parabola If she partition the data based on the input I put mapping,
that is based on the idea that a parabola will fit the data nicely, then you
partition the data where that brown line is.
If however you partitioned the data by just clustering the inputs, we partition
where the blue line is, and then if you looked to the left of that blue line,
you'll be stuck with a subset of data that can't be modeled nicely by a simple model.
So I'm going to explain an error function that encourages models to cooperate.
And then I'm going to explain an error function that encourages models to
specialize. And I'm going to try to give you a good
intuition for why these two different functions have these very different
effects. So if you want to encourage cooperation,
what you should do is compare the average predictors with the target and train all
the predictors together to reduce the difference between the target and their
average. So using angle back as for expectation
again, the error would be the difference between the target and the average of all
the predictors of what they predict. That will overfit badly.
It will make the model much more powerful in training each predictor separately,
because the models will learn to fix up the error is that other models make.
So, if you're averaging models during training, and training so that the average
works nicely, you have to consider cases like this.
On the right, we have the average of all the models except for model I.
So, that's what everybody else is saying when their votes are averaged together.
On the left, we have the output of model I.
Now if we'd like the overall average to be closer to the target, what do we have to
do to the output of the Ith model? We have to move it away from the target.
That will take the overall average towards the target.
You can see that what's happening is model I is learning to compensate for the errors
made by all the other models. But do we really want to move model I in
the wrong direction? Intuitively it seems better to move model
I towards the target. So here is an arrow function that
encourages specialization, and it's not very different.
To encourage specialization, we compare the output of each model with the target
separately. We also need to use a manager to determine
the weight we put on each of these models, which we can think of as the probability
of picking each model, if we have to pick one.
So now, our error is the expectation over all the different models of the squared
error made by that model times the probability of picking that model,
Where the manager or gating network, is determining that probability by looking at
the input for this particular case. What will happen if you try to minimize
this error is that most of the experts will end up ignoring most of the targets.
Each expert will only deal with the small subset of the training cases and it will
learn to do very well on that small subset.
So here's a picture of the mixture of expert's architecture.
Our cost function is the squared difference between the output of each
expert in the target averaged over all the experts.
But with the weights in that average determined by the manager.
It's actually a better cost function will come to later, based on the mixture model.
But this was a cost function I first thought of, and I think it's easier to
explain the intuition with this cost function.
So we have an input. Our different experts will look at that
input. They all make their predictions based on
that input. In addition we have a manager, a manager
might have multiple layers and the last layer for manager is a soft max layer, so
the manager outputs as many probabilities as there are experts,
And using the outputs of the manger and outputs of the experts, we can then
compute the value of that error fraction. If we look at the derivative of that other
function, The outputs of the manager are determined
by the inputs xi to the soft max group in the final layer of the manager.
And then the error is determined by the outputs of the experts, and also the
probabilities output by the manager. If we differentiate that error with
respect to the outputs of an expert, we get a signal for training that expert and
that gradient that we get with respect to the output of an expert is just the
probability of picking that expert, times the difference between what that expert
says in the target. So if the manager decides that there's a
very low probability of picking that expert for that particular training case,
the expert will get a very small gradient, and the parameters inside that expert
won't get disturbed by that training case. It'll be able to save its parameters for
modeling the training cases where the manager gives it a big probability.
We can differentiate with respect to the outputs of the gating network.
And actually what we're gonna do is differentiate with respect to, the
quantity that goes into the soft max. That's called the low jet, that's xi,
And if we take the derivative with respect to xi, we get the probability that, that
expert was picked times the difference between the squared arrow made by that
expert and the average overall experts when you use the weighting provided by the
manager of the squared arrow. So what that means is, if expert I makes a
lower squared error than the average of the other experts, then we'll try to raise
the probability of expert i. But if expert I makes a higher squared
error than the other experts, we'll try and lower his probability.
That's what causes specialization. Now there's actually a better cost
function. It's just more complicated.
It depends on mixture models, which I haven't explained in this course.
Again, those will be well explained in Andrew Ing's course.
I did explain, however, the interpretation of maximum likelihood, when you're doing
regression, as the idea that the network is actually making a Gaussian prediction.
That is the network outputs a particular value, say Y1 and we think of it as making
bets about what the target value might be that are a Gaussian distribution around Y1
with unit variance. So the red expert makes a Gaussian
distribution of predictions around by Y1 and the green expert makes a prediction
around Y2. The manager then decides probabilities for
the two experts and those probabilities are used to scale down the Gaussians.
Those probabilities have to add to one and they are called mixing proportions.
And so once we scale down the Gaussians we get to distribution that's no longer a
Gaussian, is the sum of the scale down red Gaussian and the scale down green
Gaussian. And that's the predictive distribution
from share experts. What we want to do now is maximize the log
probability of the target value under that black curve and remember the black curve
is just the sum of the red curve and the green curve.
So that leads to the following model for the probability re-target, given a mixture
of experts. The probability, is on the left,
And it's the sum over all the experts, of the mixing proportion assigned to that
expert by the manager or gating network times e squared the squared difference
between the target and the output of that expert,
Scaled by the normalization term for a Gaussian with a variance of one.
And so our cost function is simply going to be the negative log of that probability
on the left. We're going to try and minimize the
negative log of that probability.