In this video, I'm going to talk about improving generalization by reducing the
overfitting that occurs when a network has too much capacity for the amount of data
it's given during training. I'll describe various ways of controlling
the capacity of a network. And I'll also describe how we determine
how to set the metric parameters when we use a method for controlling capacity.
I'll then go on to give an example where we control capacity by stopping the
learning early. Just to remind you, the reason we get
over-fitting is because as well as having information about the true regularities in
the mapping from the input or output, any finite set of training data also contains
sampling error. There's accidental regularities in the
training set, just because of the particular training cases that were
chosen. So when we fit the model, it can't tell
which of the regularities are real, and would also exist if we sampled the
training set again, And which are caused by the sampling
error. So the model fits both kinds of
regularity. And if the model's too flexible, it'll fit
the sampling error really well, and then it'll generalize badly.
So we need a way to prevent this over fitting.
The first method I'll describe is by far the best.
And it's simply to get more data. There's no point coming up with fancy
schemes to prevent over fitting if you can get yourself more data.
Data has exactly the right characteristics to prevent over fitting.
The more of it you have the better. Assuming your computer's fast enough to
use it. A second method is to try and judiciously
limit the capacity of the model so that it's got enough capacity to fit the true
regularities but not enough capacity to fit the spurious regularities caused by
the sampling error. This of course is very difficult to do.
And I'll describe in the rest of this lecture, various approaches to trying to
regulate the capacity appropriately. In the next lecture, I'll talk about
averaging together many different models. If we average models that have different
forms and make different mistakes, the average will do better than the individual
models. We could make the models different just by
training them on different subsets of the training data.
This is a technique called bagging. There's also other ways to mess with the
training data to make the models as different as possible.
And the fourth approach, which is the basian approach, is to use a single neural
network architecture, but to find many different sets of weights that do a good
job of predicting the output. And then on test data, you average the
predictions made by all those different weight vectors.
So, there's many ways to control the capacity of a model.
The most obvious is via the architecture. You limit the number of hidden layers, and
the number of units per layer. And this controls the number of
connections in the network, i.e. The number of parameters.
A second method which is often very convenient is to start with small weights
and then stop the learning before it has time to overfit.
Again on the assumption that it finds the true regularities before it finds the
spurious regularities that have just to do with the particular training set we have.
I'll describe that method at the end of this video.
A very common way to control the capacity of a neural network is to give it a number
of hidden lairs or units per lair is a little to large, but then to penalize the
weights using penalties or constraints using squared values of the weights or
absolute values of the weights. And finally, we can control the capacity
of a model by adding noise to the weights, or by adding noise to the activities.
Typically, we use a combination of several of these different capacity control
methods. Now for most of these methods, there's
meta parameters that you have to set. Like the number of hidden units, or the
number of layers, or the size of the weight penalty.
An obvious way to transit those meta parameters is to try lots of different
values of one of the meta parameters like, for example, the number of hidden units,
and see which gives the best performance on the test set.
But there's something deeply wrong with that.
It gives a false impression of how well the method will work if you give it
another test set. So the settings that work best for one
particular test set are unlikely to work as well on a new test set that's drawn
from the same distribution because they've been tuned to that particular test set.
And that means you get a false impression of how well you would do on a new test
set. Let me give you an extreme example of
that. Suppose the test set really is random,
quite a lot of financial data seems to be like that.
So the answers just don't depend on the inputs or can't be predictive from the
inputs. If you choose the model that does best on
your test set, that will obviously do better than chance because you selected it
to do better than chance. But if you take that model and try it on
new data that's also random, you can't expect it to do better than chance.
So by selecting a model, you got a false impression of how well a model will do on
new data and the question is, is there a way around that?
So here's a better way to choose the meta-parameters.
You start by dividing the total data set into three subsets.
You have the training data, which is what you're going to use to train your model.
You hold back some validation data, which isn't going to be used for training.
But is going to be used for deciding how to set the meta parameters.
In other words, you're going to look at how well the model does on the validation
data to decide what's an appropriate number of hidden units or an appropriate
size of weight penalty. But then once you've done that, and
trained your model with what looks like the best number of hidden units and the
best weight penalty, You're then going to see how well it does
on the final set of data that you've held back which is the test data.
And you must only use that once. And that'll give you an unbiased estimate
of how well the network works. And in general that estimate will be a
little worse than on the validation data. Nowadays in competitions, the people
organizing the competitions have learned to hold back that true test data and get
people to send in predictions so they can see whether they really can predict on
true test data, or whether they're just over-fitting to the validation data by
selecting meta-parameters that do particularly well on the validation data
but won't generalize to new test sets. One way we can get a better estimate of
our weight penalties or number of hidden units or anything else we're trying to fix
using the validation data, is to rotate the validation set.
So, we hold back a final test set to get our final unbiased estimate.
But then we divide the other data into N equal sized subsets and we train on all
but one of those N, and use the Nth as a validation set.
Then we can rotate and a hold back a different subset as a validation set, and
so we can get many different estimates of what the best weight penalty is, or the
best number of hidden units is. This is called N-fold cross-validation.
It's important to remember, the N different estimates we get are not
independent of one another. If for example, we were really unlucky and
all the examples of one class fell into one of those subsets,
We'd expect to generalize very badly. And we'd expect to generalize very badly,
whether that subset was the validation subset or whether it was in the training
data. So now I'm going to describe one
particularly easy to use method for printing over fitting.
It's good when you have a big model on a small computer and you don't have the time
to train a model many different times with different numbers of hidden units or
different size weight penalties. What you do is you start with small
weights, and as the model trains, they grow.
And you watch the performance on the validation set.
And as soon as it starts to get worse, you stop training.
Now, the performance civilization on the set may fluctuate particularly if you're
error rate rather than a squared error or presentory error.
And so its hard to decide when to stop and so what you typically do is keep going
until you're sure things are getting worse and then go back to the point at which
things were best. The reason this controls the capacity of
the model, is because models with small weights generally don't have as much
capacity, and the weights haven't had time to grow big.
It's interesting to ask why small weights lower the capacity.
So consider a model with some input units, some hidden units, and some output units.
When the weight's very small, if the hidden unit's a logistic units, their
total inputs will be close to zero, and they'll be in the middle of their linear
range. That is, they'll behave very like linear
units. What that means is, when the weights are
small, the whole network is the same as a linear network that maps the inputs
straight to the outputs. So, if you multiply that weight matrix W1
by that weight matrix W2, you'll get a weight matrix that you can use to connect
the inputs to the outputs and provided the weights are small, a net with a layer of
logistic hidden units will behave pretty much the same as that linear note.
Provided we also divide the weights in the linear note by four, which take into
account the fact that when there's hidden units there, in that linear region, and
they have a slope of a quarter. So it's got no more capacity than the
linear net, so even though in that network I'm showing you there's three six + six
two weights, it's really got no more capacity than a network with three two
weights. That's the way its grow.
We start using the non linear region of the sequence.
And then we start making use of all those parameters.
So if the network has six weights at the beginning of learning and has 30 weights
at the end of learning, Then we could think of the capacity as
changing smoothly from six perimeters to 30 perimeters as the weights get bigger.
And what's happening in early stopping is we're stopping the learning when it has
the right number of parameters to do as well as possible on the validation data.
That is when it's optimized the trade off between fitting the true regularities in
the data and fitting the spurious regularities that are just there because
of the particular training examples we chose.