Figuring out how to get the error
derivatives for all of the weights in a
multilayer network is the key to being
able to learn efficient neural networks.
But there are a number of other issues
that have to be addressed before we
actually get a learning procedure that's
fully specified.
For example, we need to decide how often
to update the weights.
And we need to decide how to prevent the
network from over-fitting very badly if we
use a large network.
The back propagation algorithm is an
efficient way to compute the derivatives
with respect to each weight of the error
for a single training case.
But that's not a learning algorithm.
You have to specify a number of other
things to get a proper learning procedure.
We need to make lots of other decisions.
Some of these decisions are about how
we're going to optimized, that is how
we're going to use the other derivatives
on the individual cases, to discover good
set of weights.
Those will be described in detail in
Lecture six.
Another set of issues is how do we ensure
that the weights that we've learned will
generalize well, that is how do we make
sure they work on cases that we didn't see
during training and Lecture seven will be
devoted to that issue.
What I'm going to do now is give you a
very brief overview of these two sets of
issues.
So, optimization issues are about how you
use the weight derivatives.
The first question is how often should you
update the weights?
We could try out dating the weights after
each training case.
So, you compute the error derivatives on a
training case using back propagation and
then, you make a small change to the
weights.
Obviously, this is going to zigzag around
cuz on each training case, you'll get
different error derivatives.
But on average, if we make the weight
changes small enough, it'll go in the
right direction.
What seems more sensible is to use full
batch training, where you do a full sweep
through all of the training data, you add
together all of the error derivatives you
get on the individual cases, and then you
take a small step in that direction.
A problem with this is that we start off
with a bad set of weights, and we might
have a very big training set.
And we don't want to do all that work of
going through the whole training set in
order to fix up some weights that we know
are pretty bad.
Really, we only need to look at a few
training cases before we get a reasonable
idea of what direction we want to move the
weights in.
And we don't need to look at a large
number of training cases until we get
towards the end of learning.
So, that gives us mini batch learning,
where we take a small random sample of the
training cases and we go in that
direction.
We'll do a little bit of zigzagging, not
nearly as much zigzagging as if we did
online where we use one training case at a
time.
And mini batch learning is what people
typically do when they're training big
neural networks on big data sets.
Then there's the issue of how much we
update the weights.
How big a change we make.
So, we could just by hand try and pick
some fixed learning rate and then learn
the weights by changing each weight by the
derivative that we've computed times that
learning rate.
It seems more sensible to actually adapt
the learning rate.
We could get the computer to adapt it by,
if we're oscillating around, if the error
keeps going up and down, then we'll reduce
the learning rate.
But if we're making steady progress, we
might increase the learning rate.
We might even have a separate learning
rate for each connection in the network,
so that some weights learn rapidly and
other weights learned more slowly, or we
might go even further and say, we don't
really want to go in the direction of
steepest decent at all.
If you look at the figure on the right,
when we had a very elongated ellipse, the
direction of steepest decent is almost at
right angles to the direction to the
minimum that we want to find.
And this is typical particularly towards
the end of learning of most learning
problems.
So, there are much better directions to go
in than the direction of steepest decent.
The problem is, it's quite hard to figure
out what they are.
The second set of issues is to do with how
well the network generalizes the cases it
didn't see during training.
And the problem here is that the training
data contains information about the
regularities in the mapping from input to
output, but it also contains two types of
noise.
The first type of noise is that the target
values may be unreliable.
And from neural network, that's usually
only a minor worry.
The second type of noise is that the
sampling error.
If we take any particular training set,
especially if it's a small one, there will
be accident irregularities that are caused
by the particular cases that we chose.
So, for example, if you show someone some
polygons, if you're a bad teacher, you
might choose to show them a square and a
rectangle.
Those are both polygons, but there's no
way for someone to realize from that, that
polygons might have three sides or seven
sides.
There's no way for them to understand that
the angles don't have to be right angles.
If you're a slightly better teacher, you
might show them a triangle and a hexagon.
But again, from that, they can't tell
whether polygons are always convex, and
they can't tell whether the angles in
polygons are always multiples of 60
degrees.
And, however carefully, you choose
examples.
For any finite set of examples, there'll
be accidental regularities.
Now, when we fit a model, there's no way
it can tell the difference between an
accident regularity that's just there
because of the particular samples we chose
and a real regularity that's, that we'll
generalize properly to new cases.
So, what the model will do is it will fit
both kinds of regularity.
And if you've got a big powerful model,
it'll be very good at fitting the sampling
error, band that will be a real disaster.
That will cause it to generalize really
badly.
This is best understood by looking at a
little example.
So here, we've got six data points shown
in black, and we can fit a straight line
to them, but the model has two degree of
freedom and it's fitting the six y values,
given the six x values, or we can fit a
polynomial that has six degrees of
freedom.
Ad by hand, I've drawn in red, my idea of
a polynomial with six degrees of freedom
fitting this data.
And you'll see the polynomial goes through
the data points exactly and so it's a much
better fit to the data.
But which model do you trust?
The complicated model certainly fits the
data much better.
But it's not economical.
For a model to be convincing, what you
want it to do is be a simple model that
explains a lot of data surprisingly well.
And the polynomial doesn't do that.
It explains these six data points, but
it's got six degrees of freedom.
So, wherever these data points were, it
won't be able to explain them.
We're not surprise that the model as
complicated can fit that data very well
and it doesn't convince us that this is a
good model.
So, if you look at the arrow, which output
value do you predict for this input value?
Well, you'd have to have a lot of faith in
the polynomial model in order to predict a
value that's outside the range of values
in all of the training data you've seen so
far.
And I think almost everybody would prefer
to predict the blue circle that's on the
green line rather than the one on the red
line.
However, if we had ten times as much data,
and all of these data points lay very
close to the red line, then we would
certainly prefer the red line.
There's a number of ways to reduce
over-fitting that have been developed for
neural networks and for many other models,
and I'm going to give just a brief survey
of them here.
There's weight decay where you try and
keep the weights of the network small, or
try and keep many of the weights at zero.
And the idea of this is that it will make
the model simpler.
It's weight sharing, where again, you make
the model simpler, by insisting that many
of the weights have the exactly same value
as each other.
You don't know what the value is and
you're going to learn it.
But it has to be exactly the same for many
of the weights.
We'll see that in the next lecture, how
weight sharing is used.
There's early stopping, where you make
yourself a fake test set.
And as you're training the net, you peek
at what's happening on this fake test set.
And once the performance on the fake test
set starts getting worse, you stop
training.
This model averaging where you train not
so different neural on that, and you
average them together in the hopes that,
that will reduce the errors you're making,
Those Bayesian fitting of your own eyes,
which is just a fancy form of model
averaging, is dropped out but you try and
make your model more robust by randomly
emitting hidden units when you're training
it.
And this generative pretraining which are
somewhat more complicated and I'll
describe towards the end of the course.