In this video, we're going to look at a
method that was developed in the late
1980's by Robbie Jacobs and then improved
by a number of other people.
The idea is that each connection in the
neural net should have its own adaptive
learning rate, which we set empirically by
observing what happens to the weight on
that connection when we update it.
So that if the weight keeps reversing its
gradient, we turn down the learning
weight.
And if the gradient stays consistent, we
turn up the learning weight.
So, let's start by thinking why having
separate adaptive learning weights on each
connection is a good idea.
The problem is, they're in a deep
multilayer net.
The learning weights can vary widely
between different weights, especially
between weights in different layers.
So, if for example, we start with small
weights, the gradience starts from much
smaller in the initial layers than in the
later layers.
Another factor that causes one different
learning rate for different weights is the
fan-in of the unit.
The fan-in determines the size of the
overshoot effects that you get when you
simultaneously change many of the
different incoming weights to fix up the
same error.
It maybe that the unit didn't get enough
input, when you change all these weights
at the same time to fix up the error, it
now gets too much input.
Obviously, that effect is going to be
bigger if there's a bigger fan-in.
So, the net in the diagram on the right
has the same fain-in for both layers more
or less the same fain-in for both layers,
but that's very different in some nets.
So, the idea is that we're going to use a
global learning weight which we set by
hand, and then we're going to multiply it
by a local gain that is determined
empirically for each weight.
A simple way to determine what those local
gains should be is to start with a local
gain of one for every weight.
So that, initially we're going to change
the weight, Wij, by the learning rate
times the gain of one, gij times the error
derivative for that weight.
Then, what we're going to do is we're
going to adapt gij.
We're going to increase gij if the
gradient for the weight does not change
side.
And we're going to use small additive
increases, and multiplicative decreases.
So, if the gradient for the weight at time
t has the same sign as the gradient for
the weight at time t minus one, with t
refers to weight updates, then when you
take that product, it'll be positive.
Cuz you already get two negative gradients
or two positive gradients, and then what
we're going to go is increase gij by small
additive amount.
If the gradients have opposite signs,
we're going to decrease gij. And because
we want to damp down gij quickly if it's
already big, we're going to decrease it
multiplicatively.
That ensures that big gains will decay
very rapidly if oscillation start.
It's interesting to ask what would happen
if the grading was totally random.
So, on each update of the weights, pick a
random gradient.
Then, you'll get an equal number of
increases and decreases cuz it will
equally often be the same sign as the
previous gradient or the opposite sign.
And so, you'll get a bunch of additive
0.05 increases, and multiplicative 0.95
decreases, and they have an equilibrium
point which is when the gain is one.
If the gain's bigger than one, the
multiplying by 0.95 will reduce it by more
than adding 0.05. If the gain's smaller
than one, adding 0.05 will increase it
more than multiplying by 0.95 decreases
it.
So, with random gradients, we'll hover
around one.
And if the gradient is consistently in the
same direction we can get much bigger than
one.
If the gradient is consistently in
opposite directions, which means we're
oscillating across a ravine, we can get
much smaller than one.
There's a number of tricks for making the
adaptive learning rates work better.
It's important to limit the size of the
gains.
A reasonable range is 0.1 to ten.
Or 0.1 to 100.
You don't want the gains to get huge
because then you can easily get into an
instability and they won't die down fast
enough, and you'll destroy all the
weights.
The adaptive learning rates was designed
for full batch learning.
You can also apply it with mini batches
but they had better be pretty big mini
batches.
That'll ensure that the sign, changing
signs of gradience aren't due to the
sampling error of mini batches,
They are really due to the other side of
the ravine.
There's nothing to prevent you combining
adaptive learning rates with momentum.
So, Jacob suggests that, instead of using
the agreement in sign between the current
gradient and the previous gradient, you
use the agreement in sign between the
current gradient and the velocity for that
weight, so the accumulated gradient.
And, if you do that, you get a nice
combination of the advantages of momentum,
and the advantages of adaptive learning
rates.
So, adaptive learning rates only deal with
axis of line defects.
Whereas, momentum doesn't care about the
alignment of the axis.
Momentum can deal with these diagonal
ellipses and going in that diagonal
direction quickly which adaptive learning
rates can't do.