In this video we're going to look at the momentum method for improving the learning
speed when doing grading descent into neural network.
The momentum method can be applied to full batch learning, but it also works for mini
batch learning. It's very widely used.
And probably the commonest recipe for learning big neural nets is to use
stochastic grade and descent with mini batches combined with momentum.
I'm going to start with the intuition behind the momentum method.
So, we think of a ball on the area surface, where the location of the ball in
the horizontal plane represents the current weight vector.
The ball starts off stationary and so initially it will follow the direction of
steepest descent. It will follow the gradient.
But as soon as it's got some velocity it'll no longer go in the same direction
as the gradient. Its momentum will make it keep going in
the previous direction. Obviously we wanted eventually to get to a
low point on the surface, so we wanted to lose energy.
So we need to introduce a bit of viscosity.
That is, we make its velocity die off gently on each update.
What the momentum method does, is it damps oscillations in directions of high
curvature. So if you look at the red starting point,
and then look at the green point we get to after two steps, they have gradients that
are pretty much equal and opposite. As a result, the gradient across the
ravine has cancelled out. But the gradient along the ravine has not
cancelled out. Along the ravine, we're going to keep
building up speed, and so, after the momentum method has settled down, it'll
tend to go along the bottom of the ravine, accumulating velocity as it goes, and if
you're lucky, that'll make you go a whole lot faster, than if you just judge
steepest descent. The equations of the momentum method are
fairly simple. We say that the velocity vector at time t,
is just the velocity vector at time t minus one, time here is the updates of the
weights. So it's the velocity vector that we got
after mini batch t minus one, attenuated a bit.
So we multiply by some number like point.9.
Which is really viscosity, or it's related to viscosity.
But unfortunately, I called it momentum. So we now call alpha momentum.
And then we add in the effect of the current gradient,
Which is to make us go downhill by some learning rate times the gradient that we
have at time t And that'll be our new velocity at time t We then make our weight
change at time t equal to velocity. That velocity can actually be expressed in
terms of previous weight changes as it's shown on the slide share.
Then I will leave it to you to follow the math.
The behavior of the momentum method is very intuitive.
On an air surface that's just a plane, the ball will reach some terminal velocity of
which the gaining velocity that comes from the gradient is balanced by the
multiplicative attenuation of velocity due to the momentum term,
Which is really viscosity. If that momentum term is close to one,
then it'll be going down much faster than a simple gradient descent method would.
So the terminal velocity, the velocity you get at time infinity is the gradient times
the learning weight, multiplied by this factor of one over one minus alpha.
So if alpha is 0.99, you'll go 100 times as fast as you would with the learning
rate alone. You have to be careful in setting
momentum. At the very beginning of learning, if you
make the initial random weights quite big, there may be very large gradients.
You have a bunch of weights that's completely no good for the task you're
doing. And it may be very obvious how to change
these weights to make things a lot better. You don't want a big momentum.
Because you're going to quickly change them to make things better.
And then you're going to start on the hard problem of finding out how to get just the
right relative values of different weights.
So you have sensible feature detectors. So it pays at the beginning of learning to
have a small momentum. It is probably better to have 0.5 than
zero, because 0.5 will average out some sloshes and obvious ravines.
Once the large gradients have disappeared, and you've reached the sort of normal
phase of learning, where you're stuck in a ravine.
And you need to go along the bottom of this ravine without sloshing to and fro
sideways. You can smoothly raise the momentum to its
final value. Or you could raise it in one step, but
that might start an oscillation. You might think that, why didn't we just
use a bigger learning rate. But what you'll discover is that, using a
small learning rate and a big momentum allows you to get away with an overall
learning rate that's much bigger than you could have had if you used learning rate
alone with no momentum. If you use a big learning rate by itself,
you'll get big divergent oscillations] across the ravine.
Very recently Ilya Sutskever has discovered that there's a better type of
momentum. The standard momentum method works by
first computing the gradient at the current location.
It combines that with its stored memory of previous gradients, which is in the
velocity of the ball. And then it takes a big jump in the
direction of the current gradient combined with previous gradients.
So that's its accumulated gradient direction.
Ilya Sutskever has found that it works better in many cases to use a form of
momentum suggested by Nesterov who was trying to optimize convex functions, where
we first make a big jump in the direction of the previous accumulating gradient, and
then we measure the gradient where we end up and make a correction.
It's very, very similar, and you need a picture to really understand the
difference. One way of thinking about what's going on
is in the standard momentum method, you add in the current gradient and then you
gamble on this big jump. In the Nesterov method, you use your
previously accumulated gradient, you make the big jump and then you correct yourself
at the place you've got to. So here's the picture, when we first make
the jump and then make a correction. Here is a stamp in the direction of the
accumulated gradient. So this depends on the gradient that we've
accumulated on, in our previous iteration. We take that step.
We then make it the gradient, and go downhill in the direction of the gradient.
Like that. We then combine that little correction
stat with the big jump we made to get our new accumulated gradient.
We then take that accumulated gradient, we attenuate it by some number, like nine.
Or 99. multiply it by that number, and we now take our next big jump in the
direction of that accumulated gradient, like that.
Then again, at the place where we end up, we measure the gradient and we go
downhill. That correct any errors you made, and we
our new accumulated gradient. Now if you compare that with the standard
momentum method, the standard momentum method starts with a accumulating
gradient, like that initial brand vector, but then it measures the gradient where it
is, so it measures the gradient at its current location, and it adds that to the
brown vector, so that it makes a jump like this big blue vector.
That is just the brown vector plus the current gradient.
It turns out, if you're going to gamble, it's much better to gamble and then make a
correction, than to make a correction and then gamble.