The next few videos are about using the back propagation algorithm to learn a
feature representation of the meaning of the word.
I'm gonna start with a very simple case from the 1980s, when computers were very
slow. It's a small case, but it illustrates the
idea about how you can take some relational information, and use the back
propagation algorithm to turn relational information into feature vectors that
capture the meanings of words. This diagram shows a simple family tree,
in which, for example, Christopher and Penelope marry, and have children Arthur
and Victoria. What we'd like is to train a neural
network to understand the information in this family tree.
We've also given it another family tree of Italian people which has pretty much the
same structure as the English tree. And perhaps when it tries to learn both
sets of facts, the neural net is going to be able to take advantage of that analogy.
The information in these family trees can be expressed as a set of propositions.
If we make up names for the relationships depicted by the trees.
So we're gonna use the relationships son daughter, nephew niece, father mother,
uncle aunt, brother sister and husband wife.
And using those relationships we can write down a set of triples such as, Colleen has
father James, Colleen has mother Victoria and James has wife Victoria.
And in the nice simple families depicted in the diagram, the third proposition
follows from the previous two. Similarly, the third proposition in the
next set follows from the previous two. So the relational learning task, that is,
the task of learning the information in those family trees, can be viewed as
figuring out the regularities in a large set of triples that express the
information in those trees. Now the obvious way to express
irregularities is as symbolic rules. For example, X has mother Y, and Y has
husband Z, implies X has father Z. We could search for such rules, but this
would involve a search through quite a large space, a combinatorially large
space, of discrete possibilities. A very different way to try and capture
the same information is to use a neural network that searches through a continuous
space of real valued weights to try and capture the information.
And the way it's going to do that is we're going to say it's captured the information
if it can predict the third terminal triple from the first two terms.
So at the bottom of this diagram here, We're going to put in a person and a
relationship and the information is going to flow forwards through this neural
network. And what we are going to try to get out of
the neural network after it's learned is the person who's related to the first
person by that relationship. The architecture of this net was designed
by hand, that is I decided how many layers it should have.
And I also decided where to put bottle necks to force it to learn interesting
representations. So what we do is we encode the information
in a neutral way, because there are 24 possible people.
So the block at the bottom of the diagram that says, local encoding of person one,
has 24 neurons, and exactly one of those will be turned on for each training case.
Similarly there are twelve relationships. And exactly one of the relationship units
will be turned on. And then for a relationship that has a
unique answer, we would like one of the 24 people at the top, one of the 24 output
people to turn on to represent the answer. By using a representation in which exactly
one of the neurons is on, we don't accidentally give the network any
similarities between people. All pairs of people are equally
dissimilar. So, we're not cheating by giving the
network information about who's like who. The people, as far as the network is
concerned, are uninterpreted symbols. But now in the next layer of the network,
we've taken the local encoding of person one, and we've connected it to a small set
of neurons, actually six neurons for this. And because there are 24 people, it can't
possibly dedicate one neuron to each person.
It has to re-represent the people as patterns of activity over those six
neurons. And what we're hoping is that when it
learns these propositions, the way in which thing encodes a person, in that
distributive panel activities. Will reveal structuring the task, or
structuring the domain. So what we're going to do is we're going
to train it up on 112 of these propositions.
And we go through the 112 propositions many times.
Slowly changing the weights as we go, using back propagation.
And after training we're gonna look at the six units in that layer that says
distributed encoding of person one to see what they are doing.
So here are those six units as the big gray blocks.
And I laid out the 24 people, with the twelve English people in a row along the
top, and the twelve Italian people in a row underneath.
So each of these blocks you'll see, has 24 blobs in it.
And the blobs tell you the incoming weights for one of the hidden units in
that layer. So going back to the previous slide.
If you look at that layer that says distributed and coding of person one.
There are six neurons there. And we're looking at the incoming weights
of each of those six neurons. If you look at the big gray rectangle on
the top right, you'll see an interesting structure to the weights.
The weights along the top that come from English people are all positive.
And the weights along the bottom are all negative.
That means this unit tells you whether the input person is English or Italian.
We never gave it that information explicitly.
But obviously, it's useful information to have in this very simple world.
Because in the family trees that we're learning about, if the input person is
English, the output person is always English.
And so just knowing that someone's English already allows you to predict one bit of
information about the output. Which is according to saying it halves the
number of possibilities. If you look at the gray blob immediately
below that, the second one down on the right, you'll see that it has four big
positive weights at the beginning. Those correspond to Christopher and Andrew
with our Italian equivalents. Then it has some smaller weights.
Then it has two big negative weights, that correspond to Collin, or his Italian
equivalent. Then there's four more big positive
weights, corresponding to Penelope or Christina, or their Italian equivalents.
And right at the end, there's two big negative weights, corresponding to
Charlotte, or her Italian equivalent. By now you've probably realized that, that
neuron represents what generation somebody is.
It has big positive weights to the oldest generation, big negative weight to the
youngest generation, and intermediate weights which are roughly zero to the
intermediate generation. So that's really a three-value feature,
and it's telling you the generation of the person.
Finally, if you look at the bottom gray rectangle on the left hand side, you'll
see that has a different structure, and if we look at the top row and we look at the
negative weights in the top row of that unit.
It has a negative weight to Andrew, James, Charles, Christine and Jennifer and now if
you look at the English family tree you'll see Andrew, James, Charles, Christine, and
Jennifer are all in the right hand branch of the family tree.
So that unit has learned to represent which branch of the family tree someone is
in. Again, that's a very useful feature to
have for predicting the output person, because if you know it's a close family
relationship, you expect the output to be in the same branch of the family tree as
the input. So the networks in the bottleneck have
learned to represent features of people that are useful for predicting the answer.
And notice, we didn't tell it anything about what features to use.
We never mentioned things like nationality or brunch or family tree or generation.
It figured out that those are good features for expressing the regularity in
this domain. Of course, those features are only useful
if the other bottlenecks, the one for relationships, and the one near the top of
the network before the output person, use similar representations.
And the central layer is able to say how the features of the input person and the
features of the relationship predict the features of the output person.
So for example if the input person is a generation three, and the relationship
requires the output person to be one generation up, then the output person is a
generation two. But notice to capture that rule, you have
to extract appropriate features at the first hidden layer, and the last hidden
layer of the network. And you have to make the units in the
middle, relate those features correctly. Another way to see that the network works,
is to train it on all but a few of the triples.
And see if it can complete those triples correctly.
So does it generalize? So there's 112 triples, and I trained it
on 108 of them and tested it on the remaining four, I did that several times
and it got either two or three of those right.
That's not so bad for a 24 way choice, so it's true it makes mistakes, but it didn't
have much training data, there's not enough triples in this domain to really
nail down the regularities very well. And it does much better than chance.
If you train it on a much bigger data set, it can generalize from a much smaller
fraction of the data. So if you have thousands and thousands of
relationships you only need to show a small percentage before it can start
guessing the other ones correctly. That research was done in the 1980s, and
was a way of showing that back-propagation could learn interesting features.
And it was a toy example. Now we have much bigger computers, and we
have databases of millions of relational facts.
Many of which of the form A, R, B, A has relationship R to B, we could imagine
training a net to discover feature vector representations of A and R, that allow it
to predict the feature vector representation of B.
If we did that, it would be a very good way of cleaning a database.
It wouldn't necessarily be able to make perfect predictions.
But it could find things in the database that it thought were highly implausible.
So if the database contained information, like, for example, Bach was born in 1902.
It could probably realize that was wrong, 'cuz Bach's a much older kind of person,
and everything else he's related to is much older than 1902.
Instead of actually using the first two terms to predict the third term, we could
use the whole set of terms, three of them in this case, but possibly more, and
predict the probability that the fact is correct.
To train a net to do that, we'd need examples of a whole bunch of correct
facts, and we'd ask it to give a high output.
We'd also need a good source of incorrect facts, and we'd ask it to give a low
output when we're told it was something that was false.