In this video, I'm going to describe how a recurrent neural network solves a toy
problem. It's a problem that's chosen to
demonstrate what it is you can do with recurrent neural networks that you cannot
do conveniently with feet forward neural networks.
The problem is adding up two binary numbers.
Off to the recurrent neural network, has learned to solve the problem.
It's interesting to look at its hidden states, and see how they relate to the
hidden states in a finite state automaton that's solving the same problem.
So consider the problem of adding up two binary numbers.
We could train a feed-forward neural network to do that.
And the diagram on the right shows a network that gets some inputs and produces
some outputs. But there's problems with using a
feed-forward neural network. We have to decide in advance what the
maximum number of digits is both for both of the input numbers and for the output
number. And more importantly, the processing that
we apply to the different bits of the input numbers, doesn't generalize.
That is, when we learn how to add up the last two digits and deal with the carries,
that knowledges in some weights. And as we go to a different part of a long
binary number, the knowledge will have to be in different weights.
So we won't get automatic generalization. As a result, although you can train a
neuron feedfoward neural network, and it will eventually learn to do binary
addition on fixed-length numbers, it's not an elegant way to solve the problem.
This is a picture of the algorithm for binary addition.
The states shown here are like the states in a hidden Markov model, except they're
not really hidden. The system is in one state at a time.
When it enters a state it performs an action.
So it either prints a one or prints a zero and when it's in a state it gets some
input, which is the two numbers in the next column.
And that input causes it to go into a new state.
So if you look on the top right, It's in the carry station and it's just
printed a one. If it sees a one, one, it goes back in to
the same stage and print another one. If however it sees a one, zero or zero,
one, It goes into the carry state but prints a zero.
If it sees a zero, zero, it goes into the no carry state, and prints a one.
And so on. So a recurring neuro net for binary
edition needs to have two input units and one output unit.
It's given two input digits at each time stamp.
And it also has to produce an output at each time step.
And the output is the output for the column that it took in two time steps ago.
The reason we need a delay of two time steps, is that it takes one time step to
update the hidden units based on the inputs, and another time step to produce
the output from the hidden state. So the net looks like this.
I only gave it three hidden units. That's sufficient to do the job.
It would learn faster with more hidden units, but it can do it with three.
The three hidden units are fully interconnected and they have connections
in both directions that don't necessarily have the same weight.
In fact in general they don't have the same weight.
The connections between hidden units allow the pattern of one time step to insensate
the pattern of the next time step. The input units have feed forward
connections to the hidden units and that's how it sees the two digits in a column.
And similarly, the hidden units have feed forward connections to the output unit and
that's how it produces its output. It's interesting to look at what the
recurring neural network learns. It learns four distinct patterns of
activity in its three hidden units. And these patterns correspond to the nodes
in the finite state automaton for binary addition.
We must confuse the units in a neural network, with the nodes in a final state
automaton. The nodes in the finite state automaton
correspond to the activity vectors of the recurrent neural network.
The automaton is restricted to being exactly one state at each time.
And similarly, the hidden units are restricted to have exactly one activity
vector at each time in the recurrent neural network.
So a recurrent neural network can emulate a finite state automaton but it's
exponentially more powerful in its representation.
With any hidden neurons, it has 2n to the N possible binary activity vectors.
Of course it only has n squared weights so it can't necessarily make full use of all
that representational power. But if the bottleneck is in the
representation a recurrent neural network can do much better than a finite state
automaton. This is important when the input stream
has two separate things going on at once. A finite state automaton needs to square
its number of states in order to deal with the fact that there's two things going on
at once. A recurrent neural network only needs to
double its number of hidden units. By doubling the number of units, it does
of course square the number of binary vector states that it has.