In this video, I am going to give an overview of various types of models that
have been used for sequences. I'll start with the simplest kinds of
model, which is ultra aggressive models, that just try and predict the next term or
the sequence from previous terms. I'll talk about more elaborate variants of
them using hidden units. And then I'll talk about, more interesting
kinds of models, that have hidden state, and hidden dynamics.
These include linear dynamical systems and hidden Markov models.
Most of these are quite complicated kinds of models, and I don't expect you to
understand all the details of them. The main point of mentioning them is to be
able to show how recurrent your own networks are related to models of that
kind. When we're using machine learning to model
sequences, we often want to turn one sequence into another sequence.
For example, we might want to turn English words into French words or we might want
to take a sequence of sand pressures and turn it into a sequence of word identities
which is what's happening in speech recognition.
Sometimes we don't have a separate target sequence, and in that case we can get a
teaching signal by trying to predict the next term in the input sequence.
So the target output sequence is simply the input sequence with an advance of one
time step. This seems much more natural, than trying
to predict one pixel in an image from all the other pixels or one patch of an image
from the rest of the image. One reason it probably seems more natural
is that for temporal sequences, there is a natural order to do the predictions in.
Whereas for images it's not clear what you should predict from what.
But in fact a similar approach works very well for images.
When we predict the next term in a sequence, it blurs the distinction,
between supervised and unsupervised learning, that I made at beginning of the
course. So we use methods that were designed for
supervised learning to predict the next term.
But we don't require separate teaching signal.
So in that sense, it's unsupervised. I'm now going to give a quick review of
some of the, other models of sequences, before we get on to using recurrent neural
nets to model sequences. So a nice simple model for sequences that
doesn't have any memory is an auto regressive model.
What that does is take some previous terms in the sequence and try and predict the
next term basically as a weighted average of previous terms.
The previous terms might be individual values or they might be whole vectors.
And a linear auto regressive model would just take a weighted average of those to
predict the next term. We can make that considerably more
complicated by adding hidden units. So in a feedforward neural net, we might
take some previous input terms, put them through some hidden units, and predict the
next term. Memory list models are only one subclass
of models that can be used for sequences. We can think about ways of generating
sequences, and one very natural way to generate a sequence is to have a model
that has some hidden state which has its own internal dynamics.
So, the hidden state evolves according to its internal dynamics, and the hidden
state also produces observations, and we get to see those observations.
That's a much more interesting kind of model.
It can store information in its hidden state for a long time.
Unlike the memoryless models, there's no simple band, to how far we have to look
back before we can be sure it's not affecting things.
If the dynamics of the hidden state is noisy and the way it generates outputs
from its hidden state is noisy, then by observing the output of a generative model
like this, you can never know for sure what it's hidden state was.
The best you can do is to infer probability distribution over the space of
all possible hidden state vectors. You can know that it's probably in some
part of the space and not another part of the space, but you can't pin it down
exactly. So with a generative model like this, if
you get to observe what it produces, and you now try to infer what the hidden state
was, in general that's very hard, but there's two types of hidden state model
for which the computation is tractable. That is, there's a fairly straightforward
computation that allows you to infer the probability distribution over the hidden
state vectors that might have caused the data.
Of course when we do this and apply it to real data.
We're assuming that the real data is generated by our model.
So that's typically what we do when we're modeling things.
We assume the data was generated by the model and then we infer what state the
model must have been in, in order to generate that data.
The next three slides are mainly intended for people who already know about the two
types of hidden state model I'm going to describe.
The point of the slides is so that I make it clear how recurrent neural networks
differ from those standard models. If you can't follow the details of the two
standard models, don't worry too much. That's not the main point.
So one standard model is a linear dynamical system.
It's very widely used in engineering. This is a generative model that has real
valued hidden state. The hidden state has linear dynamics,
shown by those red arrows on the right. And the dynamics has Gaussian noise, so
that the hidden state evolves probabilistically.
There may also be driving inputs, shown at the bottom there, which directly influence
the hidden state. So the inputs, influence the hidden state
directly, the hidden state determines the output to predict the next output of a
system like this, we need to be able to infer its hidden state.
And these kinds of systems are used, for example, for tracking missiles.
In fact, one of the earliest uses of Gaussian distributions was for trying to
track planets from noisy observations. Gaussian actually figured out that, if you
assume Gaussian noise, you could do a good job of that.
One nice property that a Gaussian has is that if you linearly transform a gaseon
you get another Gaussian. Because all the noise in a linear dynamic
system is gaseon. It turns out that the distribution over
the hidden state given the observation so far, that is given the output so far, is
also a Gaussian. It's a full covariance Gaussian, and it's
quite complicated to compute what it is. But it can be computed efficiently.
And there's a technique called Kalman Filtering.
This is an efficient recursive way of updating your representation of the hidden
state given a new observation. So, to summarize,
Given observations of the output of the system, we can't be sure what hidden state
it was in, but we can, estimate a Gaussian distribution over the possible hidden
states it might have been in. Always assuming, of course, that our model
is a correct model of the reality we're observing.
A different kind of hidden state model that uses discrete distributions rather
than Gaussian distributions, is a hidden Markov model.
And because it's based on discrete mathematics, computer scientists love
these ones. In a hidden Markov model, the hidden state
consists of a one of N. Choice.
So there a number of things called states. And the system is always in exactly one of
those states. The transitions between states are
probabilistic. They're controlled by a transition matrix
which is simply a bunch of probabilities that say, if you're in state one at time
one, What's the probability of you going to
state three at time two? The output model is also stochastic.
So, the state that the system is in doesn't completely determine what output
it produces. There's some variation in the output that
each state can produce. Because of that, we can't be sure which
state produced a given output. In a sense, the states are hidden behind
this probabilistic veil, and that's why they're called hidden.
Historically the reason hidden units in a neural network are called hidden, is
because I like this term. It sounded mysterious, so I stole it from
neural networks. It is easy to represent the probability
distribution across n states with n numbers.
So, the nice thing about a hidden Markov model, is we can represent the probability
distribution across its discreet states. So, even though we don't know what it,
what state it's in for sure, we can easily represent the probability distribution.
And to predict the next output from a hidden Markov model, we need to infer what
hidden state it's probably in. And so we need to get our hands on that
probability distribution. It turns out there's an easy method based
on dynamic programming that allows us to take the observations we've made and from
those compute the probability distribution across the hidden states.
Once we have that distribution, there is a nice elegant learning algorithm hidden
Markov models, and that's what made them so appropriate for speech.
And in the 1970s, they took over speech recognition.
There's a fundamental limitation of HMMs. It's easiest to understand this
limitation, if we consider what happens when a hidden Markov model generates data.
At each time step when it's generating, it selects one of its hidden states.
So if it's got n hidden states, the temporal information stored in the hidden
state is at most logn) n bits. So that's all it knows about what it's
done so far. So now let's consider how much information
a hidden Markov model can convey to the second half of an utterance it produces
from the first half. So imagine it's already produced the first
half of an utterance. And now it's going to have to produce the
second half. And remember, its memory of what it said
for the first half is in which of the n states it's in.
So its memory only has log n bits of information in it.
To produce the second half that's compatible with the first half, we must
make the syntax fit. So for example, the number intend must
agree. It also needs to make the semantics fit.
It can't have the second half of the sentence be about something totally
different from the first half. Also the intonation needs to fit so it
would look very silly if the, intonation contour completely changed halfway through
the sentence. There's a lot of other things that also
have to fit. The accent of the speaker,
The rate they're speaking at, How loudly they're speaking.
And the vocal tract characteristics of the speaker.
All of those things must fit between the second half of the sentence and the first
half. And so if you wanted a hidden Markov model
to actually generate a sentence, the hidden state has to be able to convey all
that information from the first half to the second half.
Now the problem is that all of those aspects could easily come to a hundred
bits of information. So the first half of the sentence needs to
convey a hundred bits of information to the second half and that means that the
hidden Markov model needs two to the hundreds states and that's just too many.
So that brings us to recurrence your own networks.
They have a much more efficient way of remembering information.
They're very powerful because they combine two properties that have distributed
hidden state. That means, several different units can be
active at once. So they can remember several different
things at once. They don't just have one active unit.
They're also nonlinear. You see, a linear dynamical system has a
whole hidden state vector. So it's got more than one value at a time,
but those values are constrained to act in a linear way so as to make inference easy,
and in a recurrent neural network we allow the dynamics to be much more complicated.
With enough neurons and enough time, a recurring neuron network can compute
anything that can be computed by your computer.
It's a very powerful device. So linear dynamical systems and hidden
mark off models are both stochastic models.
That is the dynamics and the production of observations from the underlying state
both involve intrinsic noise. And the question is do models need to be
like that. Well one thing to notice is that the
posterior probability distribution over hidden states in either a limited anomical
system or hidden markoff model is a deterministic function of the data that
you've seen so far. That is the inference algorithm for these
systems ends up with a probability distribution, and that probability
distribution is just a bunch of numbers, and those numbers are a deterministic
version of the data so far. In a recurrent neural network, you get a
bunch of numbers that are a deterministic function of the data so far.
And it might be a good idea to think of those numbers that constitute the hidden
state of a recurrent neural network. They're very like the probability
distribution for these simple stocastic models.
So what kinds of behavior can recur at your own networks exhibit?
Well, they can oscillate. That's obviously good for things like
motion control, where when you're walking, for example, you want to know regular
oscillation, which is your stride. They can settle to point attractors.
That might be good for retrieving memories.
And later on in the course we'll look at Hopfield nets where they use the settling
to point attractors to store memories. So the idea is you have a sort of rough
idea of what you're trying to retrieve. You then let the system settle down to a
stable point and those stable points correspond to the things you know about.
And so by settling to that stable point you retrieve a memory.
They can also behave chaotically if you set the weights in the appropriate regime.
Often, chaotic behavior is bad for information processing, because in
information processing, you want to be able to behave reliably.
You want to achieve something. There are some circumstances where it's a
good idea. If you're up against a much smarter
adversary, you probably can't outwit them, so it might be a good idea just to behave
randomly. And one way to get the appearance of
randomness is to behave chaotically. One nice thing about R and N's, which, a
long time ago, I thought was gonna make them very powerful, is that an R and N
could learn to implement lots of little programs, using different subsets of its
hidden state. And each of these little programs could
capture a nugget of knowledge. And all of these things could run in
parallel, and interact with each other in complicated ways.
Unfortunately the computational power of recurrent neural networks makes them very
hard to train. For many years, we couldn't exploit the
computational power of recurrent neural networks.
It was some heroic efforts. For example, Tony Robinson managed to make
quite a good speech recognizer using recurrent nets.
He had to do a lot of work implementing them on a parallel computer built out of
transputers. And it was only recently that people
managed to produce recurrent neural networks that outperformed Tony Robinson's