In this video, we're going to look at stochastic gradient descent learning for a
neural network, Particularly the mini batch version, which
is probably the most widely used learning algorithm for large neural networks.
We've seen this before, but let's start with a reminder about what the error
surface looks like for a linear neuron. The error surface means a surface that
lies in a space where the horizontal axes correspond to the weights of the neural
net. And the vertical axis corresponds to the
error it makes. For a linear neuron with a squared error,
that surface always forms a quadratic bowl.
The vertical cross sections are parabolas, and the horizontal cross sections are
ellipses. For multilayer non linear nets the error
surface is much more complicated, But as long as the weights aren't to big
it's a smooth error surface, and locally it's well approximated by a fraction of a
quadratic bowl. It might not be the bottom of the bowl but
there's a piece of quadratic bowl that will fit the local error surface very
well. If we look at the conversion speed when we
do full-batch learning, when the error surface is a quadratic bubble,
The obvious thing to do is go downhill, this will reduce the error.
But the problem is, that the direction of steepest descent does not point to the
place we want to go to. As you can see in the ellipse, the
direction of steepest descent is almost at rectangles to the direction we want to go
in. You've got a gradient that's very big
across the ellipse, which is the direction which we only want to travel a small
distance, and the gradient's very small along the ellipse, and that's the
direction which we want to travel a large distance.
It's precisely the wrong way around. Now you might think that studying linear
systems like this, is not a good idea if you want to optimize big non-linear nets.
But even for these non-linear multi-line nets, this kind of a problem arises.
It's a very similar problem that arises even though the error surfaces aren't
globally quadratic bowls. Locally they have all these same kind of
properties. That is they tend to be very curved in
some directions, and very uncurved in other directions.
So the way the learning goes wrong if you use a big learning rate is that you slash
to and fro in the directions in which the area surface is very curved.
So we'll say call that slashing across a ravine.
And with the line rate too big you'll actually diverge.
What we want to achieve, is that we go quickly along the ravine in directions
that have small, but very consistent gradients.
And we move slowly in directions with these big, but very inconsistent
gradients. That is if you go in that direction for a
short distance, the gradient will reverse sign.
Before we go into how we achieve that, I need to talk a little bit about stochastic
gradient descent, and the motivation for using it.
If you have a data set that's highly redundant, then if you compute the
gradient for a weight on the first half of the data set, you'll get almost exactly
the same answer as you get if you compute the gradient on the second half.
So it's a complete waste of time to compute the gradient on the whole data
set. You'd be much better off computing the
gradient on a subset of the data, then updating the weights and on the remaining
data, computing the gradient for the updated weights.
We can take that to extremes and say we're going to compute the gradient on a single
training case, we're going to update the weights and then we're going to compute
the gradient on the next training case using those new weights.
That's called online learning. In general, we don't want to go quite that
far. It's usually better to use small mini
batches, typically ten or a 100 or even 1000 examples. One advantage of a small
mini batch, is that less computation is used for actually updating the weights,
cuz you do that less often, compared with online.
Another advantage is that when you compute the gradient, you can compute the gradient
for a whole bunch of cases in parallel. Most computers are very good at doing
matrix, matrix multiplies, and that will allow you to consider a whole bunch of
training cases and apply the weights to a whole bunch of training cases at the same
time to figure out the activities going into the next layer for all of those
training cases. That gives you a matrix, matrix multiply,
and it's very efficient, especially on a graphics processor unit.
One point about using mini batches is you wouldn't want to have a mini batch in
which the answer is always the same and then on the next mini batch have a
different answer that's always the same. That would cause the weights to slosh
unnecessarily. The ideal, if you have say ten classes,
would be to have a mini batch with say ten examples or 100 examples, that has exactly
the same number from each class in the mini batch.
One way to approximate that is simply to take all your data and just put it in
random order and grab random mini-batches. But you must avoid having mini batches
that are very uncharacteristic of the whole set of data because the mini-batches
are all of one class. So basically there's two types of learning
algorithms for neural nets. There's full gradient algorithms, where
you compute the gradient from all of the training cases.
And once you've done that, there's a lot of clever ways to speed up learning.
There's things like nonlinear versions of a method called conjugate gradient.
The optimization community has been studying the general problem of how you
optimize smooth nonlinear functions for many years.
Now multi-layer neural networks are pretty untypical of the kinds of problems they
study. So applying the methods they developed may
need a lot of modification to make them work for these multi-layer neural
networks. But when you have highly redundant and
large training sets, it's nearly always better to use mini batch learning.
The mini batches may need to be quite big, But that's not so bad because big mini
batches are more computationally efficient.
I'm now going to describe a basic mini-batch grading descent linear
algorithm. This is what most people would use when
they started training a big neural net on a big redundant data set.
Tyou start by guessing an initial learning rate,
And you look to see if the network learned satisfactorily or if the error keeps
getting worse, oscillates wildly. If that happens, you reduce the learning
rate. You also look to see if the error is
falling too slowly. You expect that the error might fluctuate
a bit if you measure it on a validation set, because the great electronic
mini-batch is just a rough estimate of the over all gradient.
So you don't want to reduce the learning rate every time the error arises.
But what you're hoping is that the error will fall fairly consistently.
And if it is falling fairly consistently and very slowly, you can probably increase
the learning rate. Once you've got that working, you can then
write a simple program to automate that way of adjusting the learning rate.
One thing that nearly always helps is, towards the end of learning with
mini-batches. It helps to turn down the learning rate.
That's because you're going to get fluctuations in the weights caused by the
fluctuations in the gradients that come from the mini batches.
And you'd like a final set of weights. As a good compromise.
So, when you turn down the learning rate, you're smoothing away those fluctuations,
and getting a final set of weights that's good for many mini-batches.
So a good time to turn down the learning rate is when the error stops decreasing
consistently. And a good criterion for saying the error
stopped decreasing is to use the error on a separate validation set.
That is, it's a bunch of examples that you are not using for training and also
they're not going to be used for your final test.