In this video I'm going to describe
eco-state networks.
These use a clever trick to make it much
easier to learn a recurrent neural
network.
They initialize the connections in the
recurrent neural network in such a way
that it has a big reservoir of coupled
oscillators.
So if you provide input to it, it converts
that input into the states of these
oscillators, and then you can predict the
output you want, from the states of these
oscillators.
And the only thing you have learn, is how
to couple the output to the oscillators.
This entirely gets rid of the problem.
Of learning hidden to hidden connections
or even input to hidden connections.
However, to get these networks to be good
at complicated tasks, you need a very big
hidden state.
As we'll see at the end of the video,
There's no reason not to use the
initialization that was carefully designed
for echo state networks, And then to use
back propagation through time with
momentum to train the networks to be even
better at the tasks that they're doing.
One interesting and quite recent idea
about training recurrent neural networks,
is to not train the hidden to hidden
connections at all, but to just fix them
randomly, and hope that you can learn
sequences by just training the way they
affect the outputs.
This has strong similarities with old
ideas about perceptions.
So a very simple way to train a
feedforward neural network, is to make the
early layers of feature detectors just be
random.
You put in sensible sized random weights
and then all you learn is the last layer
so that you're learning a linear model
from the activities of the hidden units in
the last layer to the outputs.
And of course it's much faster to learn a
linear model.
This relies on the idea that a big, random
expansion of the input vector, can often
make it easy for a linear model to fit the
data, when it couldn't fit the data well,
just looking at the raw inputs.
Through the little neural network here,
those red weights will be fixed at random.
They would expand the input vector and
then using that expanded representation,
we try and fit a linear model.
This actually has some quite strong
similarities with support vector machines.
Which are really just a really efficient
way of doing this.
So those same ideas, many years later,
were recycled for recurrent neural
networks.
The idea is to make the input to hidden
connections.
And the hidden to hidden connections have
random values that are carefully chosen.
And just learn the final layer of hidden
to output connections.
The learning is then very simple if you
use linear output units.
And it can be done extremely fast.
This approach is only ever going to work
if you set the random connections very
carefully, so that the recurring neural
network doesn't die out with no activity
and doesn't explode.
So, the way they set the random
connections in a echo state network is
they set the hidden to hidden weights so
that the length of the activity vector
stays about the same after each duration.
For those of you used to linear systems
and matrices, you're setting it so the
spectral radius is one.
That is the biggest eigenvalue of the
matrix of hidden to hidden weights is one.
Or it would be one if it was a linear
system.
And you want to achieve the same property
in a non-linear system.
If you set those weights to be about the
right magnitude, then an input can echo
around in the recurrent state for a long
time.
It's also important to use sparse
connectivity.
So instead of having lots of medium size
weights, we have a few quite large
weights.
And nearly all the weights are zero in the
hidden to hidden connections.
What this does is it makes a lot of
loosely coupled oscillators.
So information can hang around in one part
of the net without being propagated to
other parts of the net too quickly.
It's also important to choose the scale of
the input to hidden connections very
carefully.
Those connections need to drive the states
of the loosely coupled oscillators but,
they mustn't wipe out information that
those oscillators contain about the recent
history.
Fortunately the learning is very fast in
echo state networks so we can afford to
experiment with the scales of the
important connections.
You could think of it as a little learning
loop that's just learning the scales of
those connections and it's doing it by
sort of feedback that involves the
experimenter.
It also helps to learn the level of
sparseness that's needed in the hidden to
hidden connections, and again because the
learning is so fast, you can afford to
experiment with that.
That's important because it's often
necessary to do those experiments to get
the system to work well.
So I'm now going to show you a simple
example taken from the web of an eco-state
network.
It has an input sequence which is a real
value that varies with time, and specifies
the frequency of a sine wave for the
output of the eco-state network.
So you'd like this thing to generate sine
waves, and the input is gonna specify the
frequency.
The target output sequence is going to be
the same wave with the frequency specified
by the output.
And it's going to be learned simply by
putting a linear model that takes the
states of the hidden units and from those
tries to predict the correct scalar output
value.
So here's a picture taken from
scholarpedia of an echo state network
doing this program, the input signal is
the desired frequency of the sine wave.
The output signal after it's learned, or
the teacher signal, when it's learning, is
a sine wave with the frequency specified
by the input.
And the stuff in the middle is a big
dynamical reservoir, so that the inputs
coming from the input signal driver those
loosely coupled oscillators, and cause
complicated dynamics that goes on for a
long time.
And those output weights are learning to
map that complicated dynamics to the
particular dynamics you want for the
output.
All the other pictures are showing you the
actual dynamics of individual units inside
the dynamical reservoir.
One thing to notice is that there are also
connections from the output back to the
reservoir.
Those aren't always needed, but they help
to tell the reservoir what have has been
produced so far.
So here's an example of what they system
actually produces after it's learned, and
you can see that at the beginning it's
producing a sign wave, in phase.
At the end, it's producing a sign wave of
the right frequency, but the phase is
wrong.
And that's because we weren't telling what
phase the sign wave should be in.
So it's satisfying the requirements of
producing an appropriate frequency.
There some very good aspects of echo state
networks.
They can be trained very fast because they
just fit in a linear model.
They also demonstrate how important it is
to initialize the hidden-to-hidden weight
sensibly.
And they can do quite impressive modeling
of one dimensional time savers.
That's where they excel.
They can look at a time series for awhile,
and then predict it very well a long time
into the future.
What they're not so good at is modeling
high dimensional data, like frames of
acoustic coefficients, or frames of video.
In order to model data like that, they
need many more hidden units than a
recurrent neural network where you train
the hidden to hidden connections.
Recently, Ilya Sutskever tried something
which is fairly obvious which is to
initialize a recurrent neural network
using all the tricks developed by the
people doing echo state networks.
Once you've done that, you know you could
learn quite well just by learning the
hidden driver connections.
But then, presumably, you could learn even
better if you also learn to make the
hidden to hidden weights better.
So Ilya tried using the echo state network
initializations but then training with
back propagation through time.
He used rmsprop with momentum and he
discovered that, that is actually a very
effective way to train recurrent neural