In this video, we're going to see what
happens when the Hessian free optimizer is
used to optimize the current neural
network containing multiplicative
connections and the network is trained to
predict the next character in Wikipedia.
The network is trained on millions of
characters and it works remarkably well.
It learns a lot about English and it
becomes very good at completing sentences
in interesting ways.
Ilya Sutskever used five million strings
of 100 characters each taken from English
Wikipedia.
For each string, he starts predicting
after eleventh characters.
So the recurrent network starts off in a
default state.
It reads eleven characters, changing its
hidden state each time.
And then its ready to start predicting.
And it gets trained by back propagating
the errors it makes in prediction.
He used the Hessian free optimizer, and it
took about a month on a very fast GPU
board to get a really good model.
His current best recurrent neural network
for character prediction is probably the
best single model there is for predicting
characters.
You can do better than this model by
combining many different models, using a
neural network to decide which one to use,
but for a single model, it is as good as
it gets.
It works in a very different way, from the
best other models.
So Ilya's model can balance quotes and
brackets over long distances.
Any model that relies on matching a
specific previous context can't do that
So, for example, if it has a bracket, and
it wants to close it 35 characters later,
in order to do that properly a model that
relies on matching previous contexts,
would have to match all 35 intervening
characters.
And it's very unlikely that it has that
whole string stored.
Once the model's learned, you can see what
it knows by generating strings from the
model.
Of course you have to be very careful not
to over-interpret what it says.
The way we generate strings is we start in
the default to hidden state.
We then give it a burning sequence.
So we feed it characters and let it update
its hidden state after each character.
And then we let it stop predicting.
We look at the probability distribution it
predicts for the next character.
We pick a character randomly from that
distribution.
So if it predicts, the probability Q is
one in a thousand, we pick a Q one times
in a thousand.
We then tell the net that whatever
character we picked was the character that
actually occurred, and ask it to predict
the next character.
In other words, we're telling it whatever
he guesses is correct.
We let it continue to make characters
until we've got as many as we want, and
then we look at the strings it produces to
see what it knows.
So here's an example of a string produced
by Ilya's network after some burn in.
This was selected from a much longer
passage of text.
But it's a continuous passage, which shows
you that it works pretty well.
You'll notice that it has weird semantic
associations.
So, Opus Paul at rome.
No person would never say that.
But we understand that Opus and Paul and
Rome are all highly interconnected.
You'll notice it doesn't really have any
long range thematic structure.
It pretty much changes topic after each
full stop.
One amazing thing is that is produces very
few non words.
What that means is that, even though it's
predicting probabilities of characters,
As soon as you've got enough characters so
there's only one way to complete it as an
English word, it will predict the next
character almost perfectly.
If that wasn't the case, it would produce
non words.
Even when it does produce a non word, like
the word in red, it's a very good non
word.
I'm not absolutely certain, or I wasn't
when I first saw it, that ephemerable
wasn't an English word.
You'll also notice it's produced a closing
bracket without an opening one.
So it doesn't always balance brackets.
It just does it quite frequently.
You'll also notice at the end, that its
produced an opening quote and then a
closing quote much later.
That's consistent behavior on its part.
It really did produce that closing quote,
because it had an open quote earlier on.
If you look at this text you can see
there's a lot of good local syntax.
So, little strings of three or four words
look perfectly reasonable.
There's also lots of semantic knowledge.
So one thing we can do is we can test the
model by giving it carefully designed
strings to see what it knows.
So I tried giving it a non word.
The word thrunge T H R U N G E is not an
english word.
But most English speakers, when they see
that word, would expect it to be a verb
because of it's form.
So I gave it opening contexts in which
that might be averred, to see what
character was most likely to come next.
So, if you give it Sheila Thrunge and ask
for the next character, the most frequent
one is an s, which suggests that it knows
that Sheila is singular, just from reading
Wikipedia.
If you give it people thrunge the most
frequent next character is a space, not an
s, which suggests that it knows that
people is plural.
I then tried giving it a list of names.
So I used capitals for the names with a
comma in between.
And I put a capital T for Thrunge so it
looked like a name, to see what it would
do with that.
So it actually completed it as a name.
And if you look at the name it made, it's
not a bad name.
It indicates that it knows an awful lot
about names in many languages.
You can also give it the meaning of life
is, and the see what comes next.
If it produced 42, that wouldn't be very
interesting cuz I'm fairly sure that will
be somewhere in Wikipedia.
It produces random things.
But in it's first ten tries, it produced
the meaning of life is literally
recognition.
Which is syntactically and semantically
sensible.
We then trained the model some more and
presented it with a Meaning of Life is
again.
We took the first ten things it produced
and I'm going to show you the most
interesting one.
The completion it produced for the Meaning
of Life is, suggests that by reading
Wikipedia it really is beginning to
understand something about the meaning of
life.
That's probably just wild
over-interpretation, though.
So here's its completion.
So, what does the model know after it's
read all these characters in Wikipedia?
Well, it certainly knows about words.
It produces almost always, English words.
It will produce strings of initials,
typically in capitals.
It can produce numbers and dates and
things like that.
But it doesn't produce non words very
often.
It produces them extremely rarely.
And when it does produce them, they're
typically very plausible non words.
It also knows a lot about proper names,
like Frangelini Del Rey.
It knows about dates and numbers, and the
context in which they occur.
It's good at balancing quotes and
brackets, and in fact it can actually
count brackets.
If you give it no opening brackets, it's
very unlikely to produce a closing
bracket.
If you give it one opening bracket, it's
quite likely to produce a closing bracket
in the next twenty characters or so.
If you give it two opening brackets, it'll
produce a closing bracket very quickly.
Giving it three doesn't seem to make it
any faster.
It clearly knows a lot about syntax
because it's able to produce these little
strings of English words that are
sensible.
But it's very hard to pin down exactly
what form this knowledge has.
It's not like trigram models which have
just learned little sequences of words.
Or rather they have a table that contains
little sequences of words.
It's actually synthesizing strings of
words.
And it's synthesizing them with sensible
syntax.
It's very hard to say though what form
that syntactic knowledge had.
It's not a bunch of rules like a linguist
has.
It's much more like what's in the
linguists head when he speaks a language.
It knows a lot of weak semantic
associations.
So, for example, it only ever produced the
word Wittgenstein once.
And it produced that soon after producing
the word Plato.
So it knows that Plato and Wittgenstein
are associated.
Well, that's a pretty good assumption.
It clearly knows that cabbage is
associated with vegetable.
It doesn't know much about the precise
ways in which these things are associated.
People are like that too if you get them
to respond very fast.
So I'm going to ask you a question, and I
want you to shout out the answer.
So you're sitting there watching this
video.
And this experiment will work, by far, the
best, if you actually shout out the answer
out loud.
And you have to shout it out really fast.
You get rewarded for responding very
quickly.
It doesn't matter what you say.
You just have to respond fast.
So the question is, what do cows drink.
Most people, when given that question,
shout out milk.
Now, most cows don't drink milk most of
the time.
We say milk cuz it's associated with both
drink and with cow.
But it's not logical to say milk.
Recently, Thomas Mikolov and his
collaborators have been training large
recurrent neural networks to predict the
next word in large data sets.
They use the same technique as the feet
forward neural nets.
That first convert a word to a real valued
feature vector.
And then use those feature vectors as
input to the rest of the network.
And they do better than the feed forward
neural nets.
They also do better than the best other
models.
And when you average them with the best
other models, they do better still.
So that's the best language models there
are currently.
One interesting property of the RNNs is
that they require less training data than
the other methods to reach a given level
of performance.
More importantly, as the data sets get
bigger, the RNNs improve faster than the
other methods.
So methods like trigrams, for example, do
get better with bigger data sets but it's
a very slow process.
You need to double the size of the data
set to get a small improvement.
With RNNs, they can make much better use
of the data.
This means it's going to be very hard to
beat them as data sets get bigger.
I think it may be the same story as for
object recognition with large, deep neural
nets.
But once the neural nets get ahead, they
can make better use of faster computers
and bigger data sets.
And so, it's going to be very hard for
other methods to catch up.