In this video, we're going to look at a practical use for feature vectors that
represent words. The uses in speech recognition systems,
where having a good idea of what somebody might say next is very helpful in
recognizing the sounds they make. We're now going to see an important
practical use for feature factors that describe words.
When we're trying to do speech recognition, it's impossible to identify
phonemes perfectly in noisy speech. The acoustic input just isn't good enough.
It's often ambiguous. There may be several different words that
fit the acoustic signal equally well. We don't notice this a lot of the time,
because we're so good at using the meaning of the utterance to hear the right words.
So if I read the next comment on the slide, I would say we do this
unconsciously when we recognize beach. And you would hear, we do this
unconsciously when we recognize speech. You can actually hear the slight
difference between recognize beach and recognize speech.
But if you're expecting recognize speech, particularly if there's noise around you
wouldn't hear recognize beach. Okay.
So we're very good at doing this. We do it all the time when we do it
unconsciously. That means that speech recognizers have to
know which words are likely to come next and which are not.
Fortunately. Words can be predicted quite well without
having a full understanding of what's being said.
So there's a standard method for predicting the probabilities of the
various words that might come next, it's called the trigram method.
You take a huge amount of text and you count the frequencies of all triples of
words. Then you use these frequencies to make
bets on the relative probabilities of the next word given the previous two words.
So if we've heard the words A and B. We can look at the counts that we have in
our huge body of text. For the sequence ABC, and the sequence
ABD, we can say that the relative probability that the third word will be C
versus the third word will be D, is given by the ratio of the two counts.
Abc and ABD. Until very recently, this was the state of
the art method for getting the probability of the next word to help out the speech
recognizer. We can't use much bigger contexts than two
previous words, because there are just too many possibilities to store, and if we did
use bigger contexts, the counts would be mostly zero.
Even for two word contexts there's many contexts that you will never have heard.
For example, if I say dinosaur pizza, that's probably a string of two words that
you've never heard before. For cases like that, we have to back off
to individual words. So, after dinosaur pizza, you predict the
next word by just seeing what's likely to come after the word pizza, because you've
never heard dinosaur pizza before. What you mustn't do is to say that
probability's a zero just because you haven't seen an example.
That's clearly not true. Now the trigram model fails to use a lot
of obvious information that will help you predict the next word.
Suppose for example, you have seen the sentence the cat got squashed in the yard
on Friday. That should help you predict the words in
the sentence, the dog got flattened in the yard on Monday.
In particular, the trigram model doesn't understand the similarities between words
like cat and dog, or squashed and flattened, or garden and yard, or Friday
and Monday. So it can't use past experience with one
of those words to help it with the other one.
To overcome this limitation, what we need to do is convert the words into a vector
of semantic and syntactic features. And use the features of previous words to
predict the features of the next word. Using a feature representation, allows us
to use much bigger context, that contains many more words.
For example, ten previous words. Yoshua Bengio pioneered this approach for
language models, and his initial network for doing this looks rather familiar.
It is actually very similar to the family trees network.
It's just applied to a real problem, and is much bigger.
So at the bottom you can think of us as putting in the index of a word, and you
could think of that as a set of neurons of which just one is on.
And then the weight from that on neuron will determine the pattern of activity in
the next hidden layer. And so the weights from the active neuron
in the bottom layer will give you the pattern of activity in the layer that has
the distributed representation of the word.
That is it's feature vector. But this is just equivalent to saying you
do table look-up. You have a stored feature vector reach
word, and with learning, you modify that feature vector.
Which is exactly equivalent to modifying the weights coming from a single
active-input unit. After getting distributed representations
of a few previous words, I've only shown two here but you would typically use, say,
five. You can then, use those distributed
representations, via hidden layer, to predict, via huge softmax, what the
probabilities are for all the various words that might come next.
When extra refinement that makes it work better is to use skip-lag connections that
go straight from the input words to the output words.
Because the individual input words are, are individually quite informative about
what the output word might be. Bengio's model was actually slightly worse
than Trigram's at predicting the next word, but it was in the same ballpark, and
if you combined it with Trigram's it improved things.
Since then these language models that use feature vectors for words have been
improved considerably, and they're now considerably better than trigram models.
One problem with having a big softmax output layer, is that you might have to
deal with 100,000 different output works. Because typically in these language
models, the plural of a word is a different word from the singular.
And the various different tenses of a verb are different words from other tenses.
So each unit in the last hidden layer of the net, might have to have a
hundred-thousand outgoing weights. And that means we can only afford to have
a few units there before we start over-fitting.
That's not necessarily true. We might have a huge number of training
cases. So some organization like Google might
have so much training tech that it can afford to have a very big softmax layer.
We could try and make the last hidden less small, so we don't need too many weights.
But then we have the problem that we have to get the 100,000 probabilities of the
various words that might come next fairly accurately right.
It's not just the big probabilities we need.
A very large number of words will have small probabilities.
And the small probabilities are often relevant.
It might be that the speech recognizer Has to decide whether it's two different rare
words, and then it's very relevant which of those is more common in the context,
even though both of them are pretty unlikely.
The question is, is there a better way to deal with such a large number of outputs?
And we'll see several different ways of doing that in the next video.