In this video, we are going to look at a number of issues that arise when using
stochastic gradient descent with mini patches.
There is a large number of tricks that make things work much better.
These are the kind of black outed neural networks.
And I'm going to go over some of the main tricks in this video.
The first issue I want to talk about, is initializing the way it's in your own
network. If two hidden units have exactly the same
weights, the same bioses, with incoming and I current, then they can never become
different from one another. Because they would always get exactly the
same gradient. So, to allow them to learn diffrent
feature detectors, you need to start them off different from one another.
We do this by using small random weights to initialize the weights.
That breaks the symmetry. Those small random weights umm shouldn't
all necessarily be the same size as each other.
So if you've got a hidden unit that has a very big fan in if you use quite big
weights it'll tend to saturate it so you can afford to use much smaller weights for
a hidden unit that has a big fan in. If you have a hidden unit with a very
small fan, then you want to use bigger weights.
And since the weights are random, it scales with the square root of the number
of the weights. And so a good principle is to make the
size of the initial weights be proportional to the square root of the
fan. We can also scale the learning rates for
the weights the same way. One thing that has a surprisingly big
affect on the speed with which a neural network will learn, is shifting the
inputs. That is adding a constant to each of the
components of the inputs. It seems surprising that, that could make
much difference. But when you're using steepest decent,
shifting an input value by adding a constant can make a very big difference.
It usually helps to shift each component of the input, so that averaged over all of
the training data, it has a value of zero. That is, make sure it's mean is zero.
So suppose we have a little neuron-like likeness, just a linear neuron with two
weights. And suppose we have some training cases.
The first training case is where the inputs are 101 and a 101, you should give
an output of two. And the second one says when there are a
101 and 99 you should output a zero. And I'm using color here to indicate which
training case I'm talking about If you look at the error surface you get for
those two training cases, it looks like this.
The green line is the line along which the weights will satisfy the first training
case, and the red line is the line along which the weights will satisfy the second
training case. And what we notice is that they're almost
parallel, and so when you combine them, you get a very elongated ellipse.
One way to think about what's going on here is that, because we're using a
squared error measure, we get a parabolic trough along the red line.
The red line is the bottom of this parabolic trough that tells us the squared
error we'll be getting on the red case. And there's another parabolic trough with
the green line along its bottom. And it turns out, although this may
surprise your spatial intuition. If you add together two parabolic troughs,
you get a quadratic bowl. And elongated quadratic bowl, in this
case. So that's where that error surface came
from. Now, look what happens, if we subtract a
hundred from each of those two inbook components.
We get a completely different area surface.
It's, in this case, it's a circle, it's ideal.
The green line is the line along which the weights add to two.
We're going to take the first weight, and multiply it by one.
We're going to take the second weight and multiply it by one.
And we need to get two. So the weights better add to two.
The red line is the line along which the two weights are equal.
Because we're going to take the first weight, and multiply it by one.
And we're going to take the second weight, and multiply it by -one.
So if the weights are equal, we'll be able to get that zero that we need.
So the error surface in this case is a nice circle where gradient descent is
really easy, and all we did was subtract 100 from every input.
If you're thinking about what happens not with the input but with the hidden units.
It makes sense to have hidden units that are hyperbolic tangents that go between
-one and one. The hyperbolic tangent is simply twice the
logistic -one. And the reason that makes sense is because
then the activities of the hidden units are roughly mean zero and that should make
the learning faster in the next level. Of course, that's only true if the inputs
to the hyperbolic tangents are distributed sensibly around zero.
But in that respect, a hyperbolic tangent is better than a logistic.
However there is other respects in which a logistic is better.
For example, logistic gives you a rug to sweep things under.
It gives an output of zero, and if you make the input even smaller than it was,
the output is still zero. So fluctuations in big native inputs are
ignored by the logistic. For the hyperbolic tangent you have to go
out to the end of its plateaus before it can ignore anything.
Another thing that makes a big difference is scaling the inputs.
When we use the steepest descent, scaling the input values is a very simple thing to
do. We transform them so that each component
of the input has unit variance over the whole training set.
So it has a typical value of one or -one. So, again if we take this simple net with
two rates and we look at the error surface when the first component is very small and
the second component is much bigger. We get an error surface in which we get an
ellipse that has got a very high curvature, when the input components big
because small changes in the weight make a big difference in the output.
And very low curvature in the direction in which the input component is small because
small changes to the weight hardly make any difference to the error.
The color here is indicating which axis we're using, not which training example
we're using, as it did in the previous slide.
If we simply change the variance of the inputs, just re-scale them.
Make the first component ten times as big and the second component ten times as
small, we now get a nice circular error surface.
Shifting and scaling the inputs is a very simple thing to do, but something that's a
bit more complicated. That actually works even better cause it's
guaranteed to give you a circle, a circular error surface.
At least it is for linear neuron. What we do is we try and decorrelate the
components of the input vectors. In other words, if you take two components
and look at how they're correlated with one another over the whole training set.
Like, if you remember the early example how the number of portions of chips.
And the number of portions of ketchup might be highly correlated.
We want to try and get rid of those correlations.
That will make learning much easier. There's actually many ways to de-correlate
things. For those of you who know about principle
components analysis. A very sensible thing to do is apply
principle components analysis. Remove the components that have the
smallest eigenvalues which already achieves some dimensionality reduction.
And then scale the remaining components by dividing them by the square roots of their
eigenvalues. For a linear system, that will give you a
circular error surface. If you don't know about principle
components, we'll cover it later in the course.
Once you got a circular error surface, the gradient points straight towards the
minimum, so learning is really easy. Now, let's talk about a few of the common
problems that people encounter. One thing that can happen is if you start
with a learning rate that's much too big, you drive the hidden units either to be
firmly on, or firmly off. That is the incoming weights are very big
in positive or very big in negative. And their state no longer depends on the
input and of course that means that error root is coming from output won't affect
them, because they are on the plateaus where the derivative is basically zero.
And so learning will stop. Because people are expecting to see local
minimum, when learning stops they say, oh, I'm at a local minimum and the error's
terrible. So there are these really bad local
minimum, Usually that's not true.
Usually it's because you got stuck out on the end of a plateau.
A second problem that occurs, is, if you are classifying things and you're using
either a squared error or a cross entropy error.
The best guessing strategy is normally to make the output unit equal to the
proportion of the time that it should be one.
The network will fairly quickly find that strategy and so the error will fall
quickly, but particularly if the network has many layers it may take a long time
before it improves much on that. Because to improve over the guessing
statedgy it has to get sensible information from the input through all the
hidden layers to the output and that could take a long time to learn if you start
with small weights. So again, you learn quickly and then the
error stops decreasing, and it looks like a local minimum but actually it's another
platter. I mentioned earlier that towards the end
of learning, you should turn down the learning rate.
You should also be careful about turning down the learning rate too soon.
When you turn down the learning rate you reduce the random fluctuations in the area
do to the different gradings on different mini batches.
But of course you also reduce the rate of learning.
So if you look at the red curve you see that when we turn the learning rate down
we got a great win. The error fell but after that we get
slower learning. And if we do that too soon we're gonna
loose relative to the green curve. So don't turn down the learning rate too
soon, not too much. I'm now gonna talk about four main ways to
speed up mini-batch learning a lot. The previous things I talked about were
kind of a bag of tricks for making things work better.
And these are four methods all explicitly designed to make the learning go much
faster. I'm now gonna talk about a mathical
moment. In this method we don't use the gradient
to change the position of the whites. That is, if you think of the whites as a
ball on the error surface, standard gradient descent uses the gradient to
change the position of that ball. You simply multiply the gradient by a
learning rate and change the position of the ball by that vector.
In the momentum method, we use the gradient to accelerate this ball.
That is the gradient changes it's velocity.
And then the velocity is what changes the position of the ball.
The reason that's different is because the bull can have momentum.
That is, it remembers previous gradients in its philosophy.
A second method for speeding up when you're batch learning is to use a separate
adaptive learning rate for each parameter. And then to slowly adjust that learning
rate based on empirical measurements. And the obvious empirical measurement is
are we keeping making progress by changing the weights in the same direction?
Or does the gradient keep oscillating around so that the sign of the grading
keeps changing. If the sign of the grading keeps changing,
what we're going to do is reduce the learning rate and if it keeps staying the
same, we're going to increase the learning rate.
A third method is what I now call rms prop and what we do in this method is we divide
by a running average of the magnitudes of the recent gradients flat weight.
So that if the gradients are big you divided by a large number and if the
gradients is small and you divide then divide by small number.
That will deal very nicely with a wide range of different gradients.
It's actually a mini batch version of just using the sign of the gradient which is a
method called R prompt, that was designed for full batch learning.
The final way of speeding up learning, which is what optimization people would
naturally recommend, is to use full batch learning.
And to use a fancy method that takes curvature information into account.
To adapt that method to work for neural nets.
And then maybe to try and adapt it some more, so it works with mini batches.
I am not going to talk about that in this lecture.