In this video, I'm going to talk about the
exploding and vanishing gradients problem,
which is what makes it difficult to train
recurrent neural networks.
For many years, researchers in URL
networks thought they would never be able
to train these networks to model
dependencies over long time periods.
But at the end of this video, I can
describe four different ways in which that
can now be done.
To understand why it's so difficult to
train recurrent neural networks, we have
to understand a very important difference
between the forward and backward passes in
a recurrent neural net.
In the forward pass, we used squashing
functions, like the logistic, to prevent
the activity vectors from exploding.
So, if you look at the picture on the
right, each neuron is using a logistic
unit shown by that blue curve and it can't
output any value greater than one or less
than zero,
So that stops explosions.
The backward pass, however, is completely
linear.
Most people find this very surprising.
If you double the error derivatives is it
the final layers of this net, all the
error derivatives will double when you
back propagate.
So, if you look at the red dots that I put
on the blue curves,
We'll suppose those are the activity
levels of the neurons on the forward pass.
And so, when you back propagate, you're
using the gradients of the blue curves at
those red dots.
So the red lines are meant to throw the
tangents to the blue curves at the red
dots.
And, once you finish the forward pass, the
slope of that tangent is fixed.
You now back propagate and the back
propagation is like going forwards though
a linear system in which the slope of the
non-linearity has been fixed.
Of course, each time you back propagate,
the slopes will be different because they
were determined by the forward pass.
But during the back propagation, it's a
linear system and so it suffers from a
problem of linear systems, which is when
you iterate, they tend to either explode
or die.
So when we backpropagate through many
layers if the weights are small the
gradients will shrink and become
exponentially small. And that means that
when you backpropagate through time
gradients that are many steps earlier than
the targets arrive will be tiny.
Similarly, if the weights are big, the
gradients will explode.
And that means when you back propagate
through time, the gradients will get huge
and wipe out all your knowledge.
In a feed-forward neural net, unless it's
very deep, these problems aren't nearly as
bad because we typically only have a few
hidden layers.
But as soon as we have a recurrent neural
network trained on a long sequence, for
example 100 time steps, then if the
gradients are growing as we back
propagate, we'll get whatever that growth
rate is to the power of 100 and if they're
dying, we'll get whatever that decay is to
the power of 100 and, so, they'll either
explode or vanish.
We might be able to initialize the weights
very carefully to avoid this and more
recent work, shows that indeed careful
initialization of the weights does make
things work much better.
But even with good initial weights, it's
hard to detect the dependency of the
current output or an input from many
time-steps ago.
So it's hard to make the output depend on
things that happened a long time ago.
Rnn's have difficulty dealing with
long-range dependencies.
Here's an example of exploding and dying
gradients for a system that's trying to
learn attractors.
So suppose we try and train a recurrent
neural network,
So that whatever state we started in, it
ends up in one of these two attractor
states.
So we're going a learn blue basin of
attraction and a pink basin of attraction.
And if we start anywhere within the blue
basin of attraction, we will end up at the
same point.
What that means is that, small differences
in our initial state make no difference to
where we end up.
So the derivative of the final state with
respect to changes in the initial state,
is zero.
That's vanishing gradients.
When we back propagate through the
dynamics of this system we will discover
there's no gradience from where you start,
and the same with the pink basin of
attraction.
If however, we start very close to the
boundary between the two attractors.
Then, a tiny difference in where we start,
that's the other side of the watershed,
makes the huge difference to where we end
up, that's the explosion gradient problem.
And so whenever your trying to use a
recurrent neural network to learn
attractors like this, you're bound to get
vanishing or exploding gradients.
It turns out, there's at least four
effective ways to learn a recurrent neural
network.
The first is a method called long short
term memory and I'll talk about that more
in this lecture.
The idea is we actually change the
architecture of the neural network to make
it good at remembering things.
The second method is to use a much better
optimizer that can deal with very small
gradients.
I'll talk about that in the next lecture.
The real problem in optimization is to
detect small gradients that have an even
smaller curvature.
Heissan-free Optimization, tailored to
your own apps is good at doing that.
The third method really kind of evades the
problem.
What we do is we carefully initialize the
input to hidden weights and we very
carefully initialize the hidden to hidden
weights, and also feedback weights from
the outputs to the hidden units.
And the idea of this careful
initialization is to make sure that the
hidden state has a huge reservoir of
weakly coupled oscillators.
So if you hit it with an input sequence,
it will reverberate for a long time and
those reverberations are remembering what
happened in the input sequence You then
try and couple those reverberations to the
output you want and so the only thing that
learns in an Echo State Network is the
connections between the hidden units and
the outputs.
And if the output units are linear, that's
very easy to train.
So this hasn't really learned the
recurrent. It's used a fixed random
recurrent bit, but a carefully chosen one
and then just learned the hidden tripod
connections.
And the final method is to use momentum,
but to use momentum with the kind of
initialization that was being used for
Echo State Networks and that makes them
work even better.
So it was very clever to find out how to
initialize these recurrent networks so
they'll have interesting dynamics, but
they work even better if you now modify
that dynamic slightly in that direction
that will help with the task at hand.