In this video, I'm going to talk about the
reason why we want to combine many models
when we're making predictions.
If we have a single model, we have to
choose some capacity for it.
If we choose too little capacity, it would
be able to fit the regularities in the
training data.
And if we choose too much capacity, it
won't be able to fit the sampling error in
the particular training set we have.
By using many models, we can actually get
a better tradeoff between fitting the true
regularities, and overfitting the sampling
error in the data.
At the start of the video,
I'll show you that when you average models
together, you can expect to do better than
any single model.
This effect is largest when the models
make very different predictions from each
other.
And at the end of this video, I'll discuss
various ways in which we can encourage the
different models to make very different
predictions.
As we've seen before, when we have a
limited amount of training data, we tend
to get overfitting.
If we average the predictions of many
different models we can typically reduce
that overfitting.
This helps most when the models make very
different predictions from one another.
For regression, the squared arrow can be
decomposed into a bias term and a variance
term.
And that allows us to analyze what's going
on.
The bias term is big if the model has too
little capacity to fit the data.
It measures how poorly the model
approximates the true function.
The variance term is big if the model has
so much capacity that it's good at
modeling the sampling error in our
particular training set.
So, it's called variance, because if we go
and get another training set of the same
size from the same distribution, our model
will fit differently to that training set,
because it has different sampling error.
And so we'll get variance in the way the
models fit to different training sets.
If we average models together, what we're
doing is we're averaging away the
variance,
And that allows us to use individual
models that have high capacity and
therefore high variance.
These high capacity model typically have
low bias.
So we can get the low bias without
incurring the high variance by using
averaging to get rid of the variance.
So now let's try and analyze how an
individual model compares with an average
of models.
On any one test case some individual
predictors may be better than the combined
predictor.
The different individual predictors will
be better on different cases.
And if the individual predictors disagree
a lot, the combined predictor is typically
better than all of the individual
predictors when we average over test
cases.
So we should aim to make the individual
predictors disagree, without making them
be poor predictors.
The art is to have individual predictors
that make very different errors from one
another, but are each fairly accurate.
So, now let's look at the math and what
happens when we combine networks.
We're going to compare two expected
squared errors.
The first expected squared error is the
one we get if we pick one of the
predictors at random and use that for
making our predictions.
And then what we do is we average overall
predictors, the error we'd expect to get
if we followed that policy.
So Y bar is the average of what all the
predictors say, and YI is what an
individual predictor says.
So Y bar is just the expectation over all
the individual predictors I of YI and I'm
using those angle brackets to represent an
expectation, where the thing that comes
after the angle bracket tells you what
it's an expectation over.
We can write the same thing as one over n
times the sum overall of the n of the yi.
Now, if we look at the expected squared
error we'd get if we chose a predictor at
random,
What we'd have to do is compare that
predictor with the target, take the
squared difference.
And then average that over all predictors.
That's also on the left hand side there.
If I simply add a Y bar and subtract a Y
bar, I don't change the value.
And now it's going to be easier to do some
manipulations.
I can now multiply it that squared and
inside this expectation bracket I have t
minus y bar squared, y I minus y bar
square, and t minus y bar into y I minus y
bar, which has the c will disappear.
So the first term, T minus Y bar squared,
doesn't have an I in it anymore, and so we
can forget about the expectation brackets
for that.
That really is T minus Y bar squared.
And that's the squared arrow you'd get if
you compared the average of the models
with the target.
And our aim is to show the thing on the
left hand side is bigger than that, i.e.,
by using that average, we've reduced the
expected squared error.
So the extra term we have on the right
hand side, is the expectation of y i minus
y bar squared.
And that's just the variance of the y i.
It's the expected squared difference
between y I and y bar.
And then the last tone disappears, it
disappears because the difference of Y I
from Y bar we expect to be uncorrelated
with the difference between the arrow that
the average of the networks makes on the
target.
And so we're multiplying together two
things that are zero mean and uncorrelated
and we expect to get zero on average.
So the result is that the expected squared
error we get by picking a model at random
is greater than the squared error we get
by averaging the models by the variance of
the outputs of the models.
That's how much we win by when we take an
average.
So, I want to show you that in a picture.
So, along the horizontal line, we have the
possible values of the output, and in this
case, all of the different models predict
a value that is too high.
The predictors that are further than
average from T make bigger than average
squared errors, like that bad guy in red,
and the predictors that are less than the
average distance from T make smaller than
average squared arrows.
And the first effect dominates, because
we're using squared error.
So if you look at the math, let's suppose
that the good guy and the bad guy were
equally far from the mean.
So the average squared error they make is
Y bar minus epsilon squared plus Y bar
plus epsilon squared.
And when we work that out, we get the
squared error that the mean of the
predictors makes, plus an epsilon squared.
So we win by averaging predictors before
we compare them with the target.
That's not always true.
It depends very much on using a squared
error.
If, for example, you have a whole bunch of
clocks.
And you try and make them more accurate by
averaging them all,
That'll be a disaster.
And it'll be a disaster because the noise
you expect in clocks isn't Gaussian noise.
What you expect is that, many of them will
be very slightly wrong and a few of them
will have stopped or will be wildly wrong.
And if you average, you make sure they are
all significantly wrong, which is not what
you want.
The same thing applies to the discrete
distribution as we have our class labeled
probabilities.
So suppose that we have two models, and
one gives the correct label of probability
of Pi, and the other gives the correct
label of probability of Pj.
Is it better to pick one model at random,
or it is it better to average those two
probabilities, and predict the average of
Pi and Pj.
What if I had a measure is the log
probability of getting the right answer?
Then, the log of the average of Pi and Pj
is going to be a better bet than the log
of Pi plus the log of Pj averaged.
That's most easily seen in a diagram
because of the shape of the log function.
So that black curve is the log.
On the horizontal access I've drawn Pi and
Pj,
And the gold colored line, joins log Pi to
log Pj.
You can see that if we first start with Pi
and Pj together, to get that average value
at the blue arrow is, and then we compute
the log, we get that blue dot.
Whereas if we first take the log of pi,
and separately take the log of pj, and
then we average those two logs, we get the
mid-point of that gold line,
Which is below the blue dot.
So to make this averaging be a big win, we
want our predictors to differ by a lot.
And there's many different ways to make
them differ.
You could just rely on a learning
algorithm that doesn't work too well, and
get stuck in different local optima each
time.
It's not a very intelligent thing to do,
but it's worth a try.
You could use lots of different kinds of
models, including ones that are not neural
networks.
So, it makes sense to try decision trees,
Gaussian process models, support vector
machines.
I'm not explaining any of those in this
course.
In Andrew Ng's machine on Coursera, you
can learn about all those things.
Well you could try many other different
kinds of model.
If you really want to use a bunch of
different neural-network models, you can
make them different by using a different
number of hidden layers or a different
number of units per layer or different
types of unit.
Like in some nets you could use
rectified-linear units,
And in other nets you could use logistic
units.
You could use different types or strengths
of weight penalty.
So you might use early stopping for some
nets, and an L2 weight penalty for others,
and an L1 weight penalty for others.
You could use different learning
algorithms.
So for example you could use full batch
for some, and mini batch for others, if
your data set is small enough to allow
that.
You can also make the models differ by
training the models on different training
data.
So, there's a method introduced by Leo
Breiman called bagging, where you train
different models on different subsets of
the data.
And you get these subsets by sampling the
training set with replacement.
So we sampled a training set that had
examples A, B, C, D, and E.
And we got five examples, but we'll have
some missing and some duplicated.
And we train one of our models on that
particular training set.
This is done in a method called random
forest that uses bagging with decision
trees, which Leo Breiman was also involved
in inventing.
When you train decision trees with bagging
and then average them together, they work
much better than single decision tree bys
themselves.
In fact, the connect box uses random
forests to convert information about depth
into information about where your body
parts are.
We could use bagging with neural nets, but
it's very expensive.
If you wanted to train say, twenty
different neural nets this way, you'd have
to get your twenty different training
sets.
And then it would take twenty times as
long as training one net.
That doesn't matter with decision tress
cuz they're so fast to train.
Also, at test time, you'd have to run
these twenty different nets.
Again, with decision trees, that doesn't
matter, cuz they're so fast to use at test
time.
Another method for making the training
data different is to train each model on
the whole training set,
But to weight the cases differently So, in
boosting, we typically we use a sequence
of fairly low capacity models.
And we weight the training cases for each
model differently.
What we do is we up weight the cases the
previous model got wrong and we down
weight the case of previous model got
right.
So the next model in the sequence doesn't
waste its time trying to model cases that
are already correct.
It uses its resources to try to deal with
cases the other models are getting wrong.
An early use of boosting, was with neural
nets for MNIST,
And there when computer's are actually
slower.
One of the big advantage is was that it
focused to competitional resources on
modelling the tricky cases,
And didn't waste a lot of time, going over
easy cases again and again.