We're now going to apply Hessian-free optimization, to the task of modeling
character strings from Wikipedia. So, the idea is, you read a lot of
Wikipedia and then try to predict the next character.
Before we get to see what the model learns, I want to describe why we need
multiplicative connections and how we can implement those multiplicative connections
efficiently in a recurrent neural network. I need to start by explaining why we chose
to model character strings rather than strings of words, which is what you
normally do when you're trying to model language.
The web is composed of character strings. Any learning method that's powerful enough
to understand what's going on in the world by reading the web, ought to find it
trivial to learn which strings make words. As we'll see, this turns out to be true.
So we're going to be very ambitious here. We want something that will read Wikipedia
and understand the world. If we have to pre-process the text in
Wikipedia into words it's going to be a big hassle.
There's all sorts of problems. The first problem is morphemes.
The smallest units of meaning, according to linguists, are morphemes.
So we're going to have to break up a word into these morphemes if we want to deal
with it sensibly. The problem is, it's not quite clear what
morphines are. There's things that are a bit like
morphemes, but that a linguist wouldn't call a morpheme.
So in English, if you take any word that starts with the letters sn, it has a very
high chance of meaning something to do with the lips or nose, particularly the
upper lip or nose. So words like snarl, and sneeze, and snot,
and snog, and snort. There's too many of these words for it
just to be coincidence. Many people say yes but what about snow?
That's got nothing to do with the apple lips or nose. But, ask yourself something,
why is snow such a good word for cocaine? Then there's words that come in several
pieces. So normally, we'd want to treat New York
as one lexical item. But if we're talking about the New York
minster roof, then we might want to treat New and York as two separate lexical
items. And then there's languages like Finnish.
Finnish is an agglutinative language, so it puts together lots of morphemes to make
great big words. So here's an example of a word in Finnish
that takes about five words in English to say the same thing.
I have no idea what this word means. But despite my lack of understanding, it
makes the point. So here's an obvious kind of recurring net
we might use to try and model character strengths.
It has a hidden state and in this case, we're going to use 1500 hidden units.
And, the hidden state dynamics is that the hidden state at time T provides inputs to
determine the hidden state at time T+11. And the character also provides some
inputs. So we add together the effect of the
current character with the previous hidden state to get the new hidden state.
And then when we arrive at a new hidden state, we try and predict the next
character. So, we have a single Softmax over the 83
characters, and we get the hidden state to try and assign high probability to the
correct next character, and low probability to the others.
And we train the whole system by backpropagating from that Softmax,
The low probability of getting the correct character.
We backpropagate that through the hidden to output connections back through the
hidden to character connections, and then back through the hidden to hidden
connections, and so on and all the way back'til the beginning of the string.
It's a lot easier to predict 86 characters than 100,000 words.
So it's easier to use a Softmax at the output, we don't have the problem of a
great, big Softmax. Now, to explain why we didn't use that
kind of recurrent net, but instead used a different kind of net that worked quite a
lot better. You could arrange all possible character
strings into a tree with a branching ratio of 86, in our case.
And what I'm showing here, is a tiny little subtree, of that great big tree.
In fact, this little subtree will occur many times, but with different things that
are represented by that dot, dot, dot before the fix.
So this represents that we had a whole bunch of characters, then we had f and
then i and then x. And now if we get an i, we're going to go
to the left. If we get an e, we're gonna go to the
right, and so on. So each time we get a character, we move
one step down in this tree to a new note. There's exponentially many nodes in the
tree of all character strings of length n. So this is going to be a very big tree.
We couldn't possibly store it all. If we could store it all, what we'd like
to do is put a probability on each of those arrows.
And that will be the probability of producing that letter, given the context
of the node. In an RNN, we try and deal with the fact
that the full tree is enormous by using a hidden state vector to represent each of
these nodes. So now, what the next character has to do
is take the hidden state vector that's representing the whole string of
characters followed by fix and operate on the hidden state vector to produce the
appropriate new hidden state vector if the next character was an i.
So when you see an i, you want to turn the hidden state vector into a new hidden
state vector. A nice thing about implementing these
nodes in this character tree by using the hidden state of recurrent neural network,
is that we can share a lot of structure. For example, by the time we arrive at that
node, that says f, i, x, we may have decided that it's probably a verb.
And if it's a verb, then i is quite likely because of the ending i, n, g. And that
knowledge that i is quite likely with a verb, can be shared with lots of other
nodes that don't have f I x in. So we can get I to operate on the part of
the state that represents that it's a verb, and that can be shared between all
the verbs. Notice that, it's really the conjunction
of the current state we're at and the character that determines where we want to
go. We don't want i, to give us a state that's
expecting to get an n next if it wasn't a verb.
So, we don't want to say that i tends to make you expect an n next.
We really want to say, if you already think it's a verb, then when you see an,
i, you should expect an n next. It's the conjunction of the fact that we
think it's a verb, and that we saw an i, that gets us into this state labeled f, i,
x, i, that's expecting to see an n. So we're going to try and capture that by
using multiplicative connections. Instead of using the character inputs to
their current net to give extra additive input to the hidden units, we're going to
use those characters to swap in a whole hidden-to-hidden weight matrix.
The character is going to determine the transition matrix.
Now, if we did that in the naive way, we'd have each of the 86 characters to find a
1500x1500 matrix and that would be a lot of parameters.
If we have that many parameters, then that's likely to overfit, unless we run it
on huge amount of text, for which we might not have time.
So the question is, can we achieve this kind of multiplicative interaction, where
the character determines the hidden-to-hidden weight matrix using many
fewer parameters, by making use of the fact that characters have things in
common. For example, all of the digits are all
quite similar to each other in the way in which they make the hidden state evolve.
So, we want to have a different transition matrix for each of those 86 characters,
but we want those 86 character-specific weight matrices to share parameters, and
that's a reasonable thing to do cuz we know that characters eight to nine should
have very similar transition matrices. So here's how we're going o do it.
We're going to have things called factors, and they're going to be denoted by this
little triangle with an f above it. What that factor means is that Group a and
Group b interact multiplicatively to provide input to Group c.
So, what each factor does is it first computes a weighted sum for each of its
two input groups. So, we take the vector state of Group a,
which I just call a, and we multiply that by the weight sum connections coming into
the factor. In other words, we take the scale of
product of the vector a and the weight vector u, and that gives us a number at
the left hand vertex of that triangle. Similarly, we take the vector stage of
Group b and we multiply it by the weight factor w, and we get another number off
the bottom vertex of the triangle. We now multiply those two numbers together
and that gives us a number or scalar. And we use that scalar to scale the
outgoing weights v in order to provide input for Group c.
So the input to Group c is just the product of the two numbers that come into
the two vertices of the triangle, times the outgoing weight factor v.
We can write that as an equation. The input that factor f provides to Group
c, so its vector input to Group c, is a scalar input to f from Group b, that's got
by multiplying the state of Group b by the weights w, f times a scalar input to f
from Group a, that's got by multiplying that state of Group a by the weights u.
We then take the product of those two scalars and multiple the weight vector vf
by that, and that's the input that the factor gives to Group c.
Then, of course, we're going to have a whole bunch of those factors.
There's another way we can think about these factors that gives more insight into
what's going on. Each of the factors actually defines a
very simple kind of transition matrix. It's a transition matrix that has rank
one. So, the equation we had on the previous
slide treats a factor as computing two scalar products, multiplying them
together, and then using that as a weight on the outgoing vector v.
We can rearrange that equation. So that we get one scalar product, and
then we rearrange the last bit so that now, we take the outer product of the
weight vector u and the weight vector v, And that gives us a matrix. And the scalar
product that would be computed by multiplying b by w is just a coefficient
on that matrix. So we get a scalar coefficient, we
multiply a rank one matrix by that scalar coefficient to give us that scalar matrix.
And then, we multiply the current hidden state a by this matrix to determine the
input the factor f gives to the next hidden state.
If we sum that up over all the factors, the total input to Group c is just a sum
over all factors of a scaler times a rank one matrix, and that sum is a great big
matrix, that's the transition matrix, and it gets multiplied by the current hidden
state to produce the new hidden state. So, we can see that we synthesized the
transition matrix, Actually, these rank one matrices provided
by each factor. And what the current character in Group b has done is, is it's
determine the weight on each of these rank one matrices.
So, bw times w determines the scalar weight, the scalar coefficient to put on
each of the matrices, actually which we are going to compose this great big
character specific right matrix. So here's a picture of the whole system,
we have a number of factors, in fact we'll have about 1500 factors.
And the character input is different in that only one of those is active so there
will only be one relevant weight at a time.
And that weight from the current character k, which called w, kf, is the gain that's
used on the rank one matrix got by taking the outer product of u and v.
So the character determines again W, kf, you multiply the rank one matrix uv by
that gain, you add together the scale matrices for all the different factors and
that's your transition matrix.