In this video we're going to look at the error surface for a linear neuron.
By understanding the shape of this error surface, we can understand a lot about
what happens as a linear neuron is learning.
We can get a nice geometrical understanding of what's happening when we
learn the weights of a linear neuron. By considering a space that's very like
the weight space that we use to understand perceptrons, but it has one extra
dimension. So we imagine a space in which all the
horizontal dimensions correspond to the weights.
And there's one vertical dimension that corresponds to the error.
So in this space, points on the horizontal plane, correspond to different settings of
the weights. And the height corresponds to the error
that your making with that set of weights, summed over all training cases.
For a linear neuron, the errors you make for each set of weights define error
surface. And this error surface is a quadratic
bowl. That is, if you take a vertical
cross-section, it's always a parabola. And if you take a horizontal
cross-section, it's always an ellipse. This is only true for linear systems with
a squared error. As soon as we go to a multilayer nonlinear
neuron nets, this error surface will get more complicated.
As long as the weights aren't too big, the error surface will still be smooth, but it
may have many local minimum. Using this error surface we can get a
picture of what's happening as we do gradient descent learning using the delta
rule. So what the delta rule does is it computes
the derivative of the error with respect to the weights.
And if you change the weights in proportion to that derivative, that's
equivalent to doing steepest descent on the error surface.
To put it another way, if we look at the error surface from above, we get
elliptical contour lines. And the delta rule is gonna take it at
right angles to those elliptical contour lines, as shown in the picture.
That's what happens with what's called batch learning, where we get the grayed
in, summed overall training cases. But we could also do online learning,
where after each training case, we change the weights in proportion to the gradient
for that single training case. That's much more like what we do in
perceptrons. And, as you can see, the change in the
weights moves us towards one of these constraint planes.
So in the picture on the right, there are two training cases.
To get the first training case correct, we must lie on one of those blue lines.
And to get the second training case correct, the two weights must lie on the
other blue line. So if we start at one of those red points,
and we compute the gradient on the first training case, the delta rule will move us
perpendicularly towards that line. If we then consider the other training
case, we'll move perpendicularly towards the other line.
And if we alternate between the two training cases, we'll zigzag backwards and
forwards, moving towards the solution point which is where those two lines
intersect. That's the set of weights that is correct
for both training cases. Using this picture of the error surface,
we can also understand the conditions it will make learning very slow.
If that ellipse is very elongated, which is gonna happen if the lines that
correspond to training cases is almost parallel, then when we look at the
gradient, it's going to have a nasty property.
If you look at the red arrow in the picture, the gradient is big in the
direction in which we don't want to move very far, and it's small in the direction
in which we want to move a long way. So the gradient will quickly take us
across the bottom of that ravine. Corresponding to the narrow axis of the
ellipse. And will take a long time to take us along
the ravine, corresponding to the long Xs of the ellipse.
It's just the opposite of what we want. We'd like to get a great into a small
across the ravine, and big along the ravine but that's not what we get.
And so, simple steepest descent, in which you change each weight in proportion to a
learning rate times the error derivative, is gonna have great difficulty, with very
elongated surfaces like the one shown in the picture.