In this video, I'm going to give a brief overview of the Hessian-Free optimizer
that can be used to train recurrent neural networks very effectively.
This is a very complicated optimizer and I don't expect you to get all the details of
it from this video. I just want you to have a general feel for
how it works. And then in the next video, we will see
how well it does on an interesting problem.
When we're training the weights of a neural network, we are trying to get as
far down the error surface as possible. So one question is if we choose a given
direction to go in, How much reduction in the arrow can we
achieve by going just the right distance in that direction?
How much does the arrow decrease before it starts rising again?
And here we'll assume that the curvature is constant.
I will assume it really is a quadratic error surface.
We can assume that the magnitude of the gradient decreases as we move down the
gradient. That amounts to assuming that the error
surface is concave upward like a bowl. The maximum reduction that we can get in
the error by going in a particular direction depends on the ratio of the
gradient to the curvature. So we want to move in directions that have
a good ratio. Even if the gradient is quite small, we
want the curvature to be even smaller. So here's an example of a direction we
could move in where the vertical axis corresponds to the error, the horizontal
axis corresponds to the weights in the direction we're moving in, and the blue
arrow corresponds to the reduction we get if we start at that red point.
Here's a surface that has a gentler gradient but because it's got a better
ratio as the gradient to the curvature, we get a bigger reduction in the error by the
time we get to the minimum. The question is, how can we find
directions like that second one? Directions in which even though the
gradient may be small, the curvature is even smaller.
So let's start with Newton's method. Newton's method addresses the basic
problem at its deepest descent, which is that the gradient isn't the direction that
you want to go in. If the error surface has circular cross
sections and is quadrant, the gradient is a good direction to go.
It will point straight at the minimum. So the idea of Newton's method is to apply
a linear transformation that turns ellipses into circles.
If that we apply that transformation to the gradient vector, it will be as if we
were going downhill in a circular error surface.
To do this, we need to multiply the gradient dE by dW by the inverse of the
curvature matrix. So H is the curvature matrix, sometimes
called the Hessian. Its the function of the weights we have
and we need to take its inverse and multiply the gradient by that,
Then we need to go some distance in that direction.
If it's a truly quadratic surface and we choose epsilon correctly, which is quite
easy to do, we'll arrive at the minimum of the surface in a single step.
Of course, that single step involves something complicated, which was inverting
that Hessian matrix. The problem with this is that even if we
only have a million weights in our neural network, the curvature matrix, the
Hessian, will have a trillion terms, is completely infeasible to invert it.
So curvature matrices look like this. For each weight, Wi or Wj,
They tell you how the gradient in one direction changes as you change in another
direction. In other words, as I change weight i, how
does the gradient of the error with respect to weight j change?
That's what a typical off diagonal term tells you.
The terms on the diagonal tell you how the gradient of the arrow changes in the
direction of a weight as you change that weight.
So the off-diagonal terms in a curvature matrix correspond to twists in the error
surface. A twist means, when you travel in one
direction, the gradient in another direction changes.
If we have a nice circular [inaudible], all those off-diagonal terms are zero.
As we travel in one direction, the gradient in other directions doesn't
change. So, what's going wrong with steepest
descent, when you have an elliptical error surface,
Is that, as we travel in one direction, the gradient in another direction changes.
And so if I update one of the weights, at the same time as I'm updating all the
other weights, all those other updates will cause a change in the gradient for
the first weight. And that means, when I update it, I may
actually make things worse. The gradient may have actually reversed
sine due to all the changes in the other weights.
And so. as we get more and more weights, we need to be more and more cautious about
changing each one of them, Because the simultaneous changes in all
the other weights can change the gradient of our range.
The curvature matrix determines the size of those interactions.
So we have to deal with the curvature. We can't just ignore it.
And we'd like to deal with it without actually inverting a huge matrix, because
the matrix has too many terms in a big neural net.
One thing we can do is to just look at the leading diagonal of the curvature matrix
and make our step size depend on that leading diagonal.
That helps a bit. It will get us to make different step
sizes for different weights, But the diagonal turns only a tiny
fraction of the interactions, so we're ignoring most of the turns in the
curvature matrix when we do that. In fact, we're ignoring nearly all of
them. Another thing we could do is, turn
approximate the coverage of matrix with the matrix of much lower rank that
captures the main aspects of the coverage matrix.
That was done in Hessian-Free methods and LBFGS, and many other methods that trying
and do an approximate second order method for minimizing the error.
In the Hessian-Free Method, we make an approximation to the curvature matrix and
then we assume that the approximation is correct.
So we assume we know what the curvature is and that the error surface really is
quadratic. And then, starting from wherever we are
now, we minimize the error using an efficient technique called conjugate
gradient. Once we've done that, once we got close to
a minimum on this approximation to the curvature, we then make another
approximation to the curvature matrix and we use conjugate gradient to minimize
again. It's also important in recurrent neural
networks to add a penalty for changing any of the hidden activities too much.
That will prevent us for example, from changing a weight early on that causes
huge effects later on in the sequence. We don't want to get effects that are too
big, and if we look at the changes in the hidden activities we can prevent that by
penalizing those changes. If we put a quadratic penalty on those
changes, we can combine that with the rest of the Hessian-Free method.
The last thing I need to explain is conjugate gradient and I'm just going to
explain it briefly. Conjugate gradient is a very clever method
that, instead of trying to go straight to the minimum like in Newton's method, tries
to minimize in one direction at a time. So it starts off by taking the direction
of steepest descend and goes to the minimum in that direction.
That might involve re-evaluating the gradient, re-evaluating the error a few
times to find the minimum in that direction.
Once its done that, it now finds another direction and goes to the minimum in that
second direction. The clever thing about the technique is,
it chooses its second direction in such a way that it doesn't mess up the
minimization it already did in the first direction.
That's called a conjugate direction. Conjugate means that as you go in the new
direction, you don't change the gradients in the previous directions.
It's a funny idea. It's like the idea of a twist in an error
surface. A twist means when you go in one
direction, you change the gradient in another direction.
And a conjugate direction is one you can go in that in a sense, doesn't have a
twist. You go in that direction and the gradient
in the first direction doesn't change. So here is a picture of an ellipse and the
red line is the major axis of the ellipse. We start off by doing one step of steepest
descent all the way to the minimum in that direction.
And if you think about it a bit, you can see that the minimum won't actually lie on
the red line. On the red line, the gradient will be
zero, at right angles for that red line, cuz it's the bottom of the ravine.
But the direction we're going in, isn't actually at right angles to that point.
We can make a little bit more progress by making a small step at right angles to the
red line and then a small step along the red line.
Since the red line slopes down towards the middle of the elipse, that's going to make
some progress for us. So when we minimize in the first
direction, we'll go slightly across the bottom of the ellipse. And when we reach
that point that's a minimum, there's an interesting property of all the points
that lie on the green line. On that green line, the gradient in the
direction of that black arrow is zero. So we can go anywhere along that green
line and we won't destroy the fact that we are at a minimum in the direction of the
black arrow. If we can keep doing that from many
directions in a high dimensional error surface, we'll eventually be at a minimum
in many different directions. And if we are at a minimum in as many
different directions as there are dimensions in the space, we'll be at the
global minimum. So, we take this first step of steepest
descent, we then figure out, and I'm not going to explain how we do that.
We figure out the direction of that green line, and then, we do a search along the
green line to find how far we should go in order to minimize the error along the
green line. And we take our second step, like this.
And now, in this 2-dimensional space, we'll be at the minimum.
Because, we're at the minimum in the direction of the first step and we're now
at a minimum in the direction of the second step,
While still being at a minimum in the direction of the first step and so that
must be the global minimum. What conjugate gradient achieves is that
it gets to the global minimum of an N-dimensional quadratic surface in only N
steps. It's very efficient.
It does that because it manages to get the gradient to be zero in N different
directions. They're not orthogonal directions,
But they are independent of one another and so that's efficient to be at the
global minimum. More importantly, in many less than N
steps on a typical quadratic surface, it will have reduced the area very close to
the minimum value, and that's why we use it.
We're not going to do the full N steps, that would be as expensive as inverting
the whole matrix. We're going to do many less than N steps,
and we're going to get quite close to the minimum.
You can apply conjugate gradient directly to a non-quadratic error surface, like the
error surface for a multilayer non-linear neural net and it usually works quite
well. It's essentially a batch method, but you
can apply it to large mini batches. And when you do that, you do many steps of
conjugate gradient on the same large mini batch and then you move on to the next
large mini batch. That's called non-linear conjugate
gradient. The Hessian-Free optimizer uses conjugate
gradient for minimization on a genuinely quadratic surface and that's what
conjugate gradient is best at. It works much better for that than for a
non-linear surface. This genuinely quadratic surface that HF
is using it for is the quadratic approximation to the true surface that was
made by the Hessian-Free method. So it makes that approximation,
It uses conjugant gradient to get close to a minimum, for the first approximation.
And then it makes a new approximation to the curvature, and does it again.