In this, video we're going look at various ways to avoid having to use 100,000
different output units in the softmax, if we want to get probabilities for a 100,000
different words. At the end of the video, I'll show an
example of the words, or the word representations that are learned by a
particular method. To see what these representations look
like, what we do is we embed them in a two dimensional space.
And then, we can see which words have extremely similar representations.
That gives us a feel for what the neural network has been able to learn, just in
trying to predict the next word or perhaps, the middle word of a string of
words. So, one way to avoid having a 100,000
output units, is to go through the words one at a time.
So, the input consists of a context of previous words, and we plug in a candidate
word. Then the output of the network is a score
for how good that combination is. How well does that candidate word fit into
that context? Of course, that means we have to run this
net many, many times but most of the work can be shared so the inputs that come into
that final hidden layer from the context, stay the same for all different candidate
words and so we only need to run a small part of the network for each candidate
word. We try all the candidate words one at a
time. And I don't want to [sigh].
So, one way having so, one way to avoid having a 100,000 different output units,
is to use a serial architecture. We put in the context words as before but
now, we also put in a candidate for the next word in the same way as the context
words. Then, we go forward through the net and
what we output, is a score for how good that candidate word is, in that context.
Of course, we have to run forward through this net, many, many times.
But most of the work only needs to be done once.
So, the inputs from the context to that big, big hidden layer are the same for
every different candidate word. The only bit we need to run for each
candidate word is the inputs coming from the candidate word and the final output to
the score. And, of course, that doesn't have many
weights in it. So, we try all the candidate words one at
a time. And by putting in the word as a candidate
at the bottom, we're able to use the learned feature vector that, for that kind
of word, that we learned when it was a context word.
So, we can have the same representation of the candidate word when it's in the
context, and when it's a candidate for the next word.
So, we can have the same representation of the word when it's part of the context,
and when it's a candidate for the next word that we're trying to predict.
Learning in the serial architecture works in the following way.
We first compute the score for each possible candidate word and then we put
all those scores, which we computed sequentially into a big softmax to get
word probabilities. Now, the difference between the word
probabilities and their target probabilities which is normally one for
the correct word and zero for everything else, gives us the cross entropy error
derivatives and we use those derivatives to change the way, in such a way that we
raise the score for the correct candidate and we lower the scores for all of these
high scoring rivals. We can save a lot of time in this
architecture if instead of considering all possible candidate words, we only consider
a small set. Perhaps, candidate words suggested by some
other predictor. For example, we could use the neural net
to revise the probabilities of the words that the Trigram model thinks are likely.
A different way to avoid a great big softmax is to structure the words into a
tree. So, we arrange all of the words in a
binary tree, with the words as its leaves, we then use the context of previous words
to generate a prediction vector v. We compare that prediction vector with a
vector that we learned for each node of the tree.
The way we do the comparison is by taking a scale of product of the, the prediction
vector and the vector that we've learned for the node of the tree.
And then, we apply, apply the logistic function.
And then, we apply the logistic function to that scale of product, and that will
give us the probability of taking the right branch in the tree and one minus
that, gives us the probability of taking the left branch in the tree.
So, the tree looks like this and if that sigma is the logistic function, you can
see at the top of tree that we take the logistic of the prediction vector, times
the vector that we learned ui of the top node, and that tells us the probability
taking the right branch and conversely we take the left branch with one minus that
probability and so on, all the way down the tree to the word we want.
So, when we're learning, we use our contacts to get a prediction vector.
And in this work, we used quite a simple neural network, that simply takes the
stored, learned representations of each word, and uses the feature vector for each
word to provide evidence for the great prediction vector.
We use quite the simple, we use quite a simple neuron at work, for this work,
where we take the feature vector for each word and those feature vectors directly
contribute evidence in favor of a prediction vector.
We simply add up the evidence contributed by those feature vectors and that gives us
the prediction vector. That prediction vector then gets compared
with the vectors that have been learned for all the nodes in the tree on the path
to the correct next word. So, that would be nodes i, j, and m in
this tree. That red path shows you the path to the
word that actually occurred next and those are the only nodes we need to consider
during learning. What we know is that, we'd like the
probability of predicting that word to be as high as possible.
And so we'd like the probability of taking that path to be as high as possible.
So, we'd like the product of the probabilities on the individual elements
of that path to be as high as possible. And that means we'd like the sum of their
log probabilities to be high. So, we benefit from a nice decomposition
here. That when we try and maximize the
probability of picking the correct target word, we're really trying to maximize the
sum of the log probabilities of taking all the branches on the path that leads to
that target word. So, during learning, we only need to
consider the nodes on that correct path. And that's a huge win, that's
exponentially fewer nodes when considering all of the nodes.
So, it's log to the base two of n instead of n.
For each of those nodes, we know the correct branch, cuz we know what the next
word is. We know the current probability of taking
that branch, by comparing the prediction vector with the learned vector of the
node. As so, we can get derivatives for learning
both the prediction vector v and learned vector of that node, u.
This makes training hundreds of times faster.
Unfortunately, it's still slow at test time.
At test time, you need to know the probabilities of many words to help speech
recognizer. And so, you can't just consider one path.
There's a much simpler way to learn feature vectors for words.
This is what by Collobert and Weston. And what they did was learn feature
vectors for words and then showed that the feature vectors they learned were very
good for a whole bunch of different natural language processing tasks.
They're not trying to predict the next word, they're just trying to get good
feature vectors for words, and so, they use both the past context and the future
context. So, they look at a window of eleven words,
five in the past and five in the future. And in the middle of that window, they put
either the correct word, the one that actually occurred in the text, or a random
word. And then they trained a neural net to
produce an output that's high if it's a correct word and low if it's a random
word. The neural net works the same way as
before. We map the individual words to feature
vectors, these word codes, and then we use the feature vectors in the neural net,
possibly with more layers in the neural net to try and predict whether this is the
right or wrong word for that context. So, what they're really doing is judging
whether the middle word is an appropriate word for the five-word context on either
side of it. They trained this up on about 600 million
examples from Wikipedia. And they showed that the vectors they get
are good for a variety of different natural language processing tasks.
One way of getting a sense of the vectors that they learn for words, is to display
the vectors in a two-dimensional map. All we want to do is lay out the word
vectors in such a way that very similar vectors are very close to one another.
So, then you'll discover what words the neural network thinks have similar
meanings. We're going to use a multi-scale method
called t-sne. You can look up t-sne on Google and
discover how it works if you want. And t-sne is able, in addition to putting
very similar words close to each other, it's also able to put similar clusters
close to each other. So, it gives you structure at many
different scales. What we see, is that the learned feature
vectors capture lots of subtle semantic distinctions.
And they do this just by looking at strings of words from Wikipedia.
Nobody tells them anything other than the fact that these eleven words occurred in
the string. There's no extra supervision.
What's remarkable is that, that contextual information, the fact that these words
occurred together, tells you an awful lot about what the word means.
In fact, some people think that's the main way we learn the meaning of words.
So, here's and example. If I give you a sentence with a word
you've never heard before like, She scrammed him with a frying pan.
From that one sentence, you already have a pretty good idea what scrammed means.
It's conceivable that she was trying to impress him with her cooking skills and so
scrammed means impressed or something like that but probably it means something like
walloped. So, here's part of a two-dimensional map
in which we laid out the two and a half thousand commonest words and you'll see
this part of the map is all about games. Not only that, it's got similar kinds of
words together. So, matches and games and races are
together. It's got players and teams and clubs
together. It's got the things you win at games
together, like cups, and bowls, and medals and prizes.
It's got different kinds of games together.
It's done a very good job of taking these words to do with games and finding out
which ones are very similar in meaning. And because it's using very similar
feature vectors for those, it can in any natural language processing task, say,
that if one word was a good word for that context, the other word's probably also a
pretty good word for that context. Here's another part of the map.
This part of the map is entirely about places.
At the top, it has US states. Under that it has some cities.
Mainly ones in North America. And under that other cities, then it has
some countries. So, if you look at the cities this one is
clearly Cambridge. And underneath Cambridge, there's
something else that's very similar to Cambridge.
Here, we see that it's put Toronto with Detroit and Ontario and Boston.
Toronto's in English-speaking Canada. And it's put Quebec, which is in
French-speaking Canada, with Berlin and Paris.
If we look at the bottom, we can see that it thinks Iraq is pretty similar to
Vietnam. Here's another example.
If you look here, these are adverbs. And it's understood that likely and
probably and possibly and perhaps, all mean very similar things.
It's also understood that entirely, completely, fully, and greatly have very
similar meanings. And it's understood various other kinds of
similarity. For example, which and that, or whom and
what, or how and whether and why.