In this video, I am going to describe an
approach to training recurrent neural
networks that's called Long Short Term
Memory.
You can consider the dynamic state of a
neural network to be a short term memory.
And the idea is, you want to make that
short term memory last for a long time.
This is done by creating special modules
that are designed to allow information to
be gated in, and then information to be
gated out when needed.
And in the intermediate period, the gate
is closed, so the stuff that arrives in
the intermediate period doesn't interfere
with the remembered state.
Long short term memory has been very
successful for tasks like recognizing
handwriting, where it's won a number of
competitions.
In 1997, Hochreiter & Schmidhuber
published a paper in neural computation
that solved the problem of getting a
recurring neural network to remember
things for a long time.
There recurrent nets could remember things
for hundreds of time steps.
They did this by designing a memory cell
that used logistic and linear units with
multiplicative interactions.
So information gets into the memory cell
whenever a logistic write gate is turned
on.
The rest of the recurrent network
determines the state of that write gate,
and when the rest of the recurrent network
wants information to be stored, it turns
the write gate on, and whatever the
current input from the rest of the net to
the memory cell is, gets stored in the
memory cell.
The information stays in the memory cell
so long as its keep gate is on.
So again, the rest of the system is
determining the state of a logistic keep
gate, and if it keeps it on, then the
information will stay there.
And finally, the information gets read
from the memory cell so that it then goes
off to the rest of the recurrent neural
network and influences future states and
it's read by turning on a read gate,
Which again is a logistic unit controlled
by the rest of the neural network.
The memory cell actually stores an analog
value, so we can think of it as a linear
neuron that has an analog value and keeps
writing that value to itself at each time
step by a weight of one, so the
information just stays there.
The weight of one is determined by a keep
gate so the rest of the system determines
the state of that logistic keep gate and
if it puts it into a state of one or close
to one the information just cycles around
and that value of 1.73 will stay there.
As soon as the rest of the system wants to
get rid of that value, all it has to do is
set the keep gate to have a value of zero
and the information will disappear.
To store the information in the memory
cell, the rest of the system has to turn
on the write gate.
And then whatever input is being provided
to the memory cell from the rest of the
system will get written into the memory
cell.
Similarly, to read the information from
the memory cell, the rest of the system
turns on the logistic read gate and then,
the value in the memory cell comes out and
affects the rest of the recurring neural
network.
The point of using logistic units is that
we can back propagate through them because
they have nice derivatives, and that means
we can learn to use this kind of circuit
over many time steps.
So I'm going to show you now a picture of
what backpropagation through a memory cell
looks like.
First we're going to do a forward pass.
So at the initial time, let's suppose that
the keep gate was set to zero, so we wiped
out whatever information was in the memory
cell before,
And the write gate is set to one.
So the value of 1.7 that is coming from
the rest of the recurrent neural network
gets written into the memory cell.
And we're not going to read it at this
time, so the read gate is set to zero.
We then set the keep gate to one, or
rather the rest of the, neural network has
to set the keep gate to one, And that
means that the value is written back into
the memory cell.
It's stored.
At the next time step, we're going to set
the right gate to zero and the read gate
to zero, so the information isn't
influenced by what's going on in the rest
of the net, and it doesn't influence
what's going on in the rest of the net.
It's insulated.
Again, at the next time step, the keep
gate is set to one, so the information is
stored for one more time step.
And then, we're going to t set the right
gate to zero, so no information is written
in, but we're now going to retrieve the
information by setting the reed gate to
one.
The value of 1.7 then comes out of the
memory cell and goes off to influence the
rest of the network.
And if we don't need it anymore then the
keep gate can be set to zero and the
information will be removed.
Now, if you look at the 1.7 that comes out
when we do the retrieve and you look at
the path back to the 1.7 that came in,
along that path is these little triangular
symbols and next to each triangular symbol
is a one.
That means that the effective weight on
that connection is a one.
So as we go back along that path whatever
error derivative we have for the 1.7 when
it's retrieved gets backpropagated to 1.7
when it's stored.
So if you'd rather retrieved a bigger
value to make the right things happen now
you can send the information back and tell
it, it should have stored a bigger value.
And notice that as long as the relevant
gates have values of one, there's no
attenuation in this backpropagated signal.
It's got just the properties we want.
Of course if they're logistic gates there
will be some slight attenuation,
But it can be very small and so
information can travel back through
hundreds of time steps.
Now, let's look at a task that a recurrent
neural network with long short term memory
is very good at.
It's a very natural task for recurrent
neural network.
It's reading cursive handwriting.
The input is just a sequence of the x and
y coordinance of the tip of the pen,
Plus some information about whether the
pen is on the paper or not.
The output is going to be a sequence of
recognized characters.
Graves & Schmidhuber in 2009, showed that
recurrent neural networks with long short
term memory are extremely good at this
task.
So far as I know, they're currently the
best systems there are and I believe
Canada Post is starting to use them for
reading handwriting.
Graves & Schmidhuber who, in 2009, didn't
use pen coordinates as input.
They used a sequence of small images.
And that means they can deal with optical
input where the timing of the pen isn't
known.
They can look at images after they've been
written and read them.
So I'm now gonna show you a demonstration
of Alex Graves's system working on pen
coordinates.
And in the movie that follows you're going
to see four streams of information.
The top row shows the characters as
they're recognized.
The system never revises its output.
So if it has to make a difficult decision,
it delays it for a little bit, so that it
can see a little distance into the future
to help it resolve ambiguities.
The second row shows the states in a
subset of the memory cells, and you should
notice how they get reset when it
recognizes a character.
The third row shows the actual writing and
all the net sees is the x and y
coordinates of the tip of the pen.
Just two numbers plus some information
about whether the pen is up or down.
Finally, the fourth row shows something
much more complicated.
It shows the gradient backpropagated all
the way to the xy locations.
So what you get to see is, for the most
active character,
If you backpropagate from that character
and ask what would make that most active
character more active, you get to see
which bits of the input are affecting the
probability that it's that character.
So that let's you see how the decisions,
are depending on things that happened in
the past.
So here's the movie.