In this video, I'm going to explain a very different way of using Hopfield's energy
function. We add some hidden units to the network.
And what we are trying to do is make the states of those hidden units represent an
interpretation of the perception input that's shown on the visible units.
So, the idea is that the weights on between units represent constraints on
good interpretations. And by finding a low energy state, we find
a good interpretation of the input data. Hopfield nets combine two ideals, the idea
of that you can find a local energy minimum by using a network of
symmetrically connected binary threshold units, and the idea that these local
energy minima might correspond to memories.
There's a different way of using the ability to find local minima.
Instead of using the net to store memories, we can use it to construct
interpretations of the sensory input. So, the idea is that we have the input
represented by some visible units. And we can structure an interpretation of
that input in the set of hidden units. So, the interpretation or explanation of
the input is going to be a binary configuration over the hidden units.
The energy of the whole system will represent the badness of that
interpretation. So, to get good interpretations according
to our current model of the world, which is in the energy function, we need to find
low energy states of the hidden units, given, the input represented by the
visible units. I want to give an example of this to make
the idea clearer, in order to give the example, I need to go into a little bit of
detail, about what you can infer, when you see a 2-D line in an image.
What does that tell you about the three-dimensional world?
So, a 2-D line in an image could have been caused by many different three-dimensional
edges in the world. If this blue dot is your eyeball and the
red lines are two lines of sight coming from the center of your eyeball,
Then the black line is a possible 3-D edge that would lead to a two-dimensional line
on your retina. Here's another 3-D edge that would lead to
exactly the same thing on your retina. And here's another one.
All of these different 3-D edges have
exactly the same appearance in the image. That's because we've lost the information
about how far away the ends of the line are along that line of sight.
We know the end is somewhere along the line of sight but we don't know the depth.
So, if we assume that a straight 3-D edge in the world is the cause of a straight
2-D line in the image, then we've lost two degrees of freedom of that 3-D edge, its
depth at each end. So, there is a whole family of 3-D edges
that all correspond to the same 2-D line. You can only see one of these 3-D edges at
a time because they all get in the way of each other.
So now, we're in a position to see a little example of what you might be able
to do, if you can use the fact that you can find low energy states of a network of
binary units, to help you find interpretations of sensory input.
So, here's the example. You imagine we see a line drawing, and we
want to interpret it as a three-dimensional thing.
So, the data we have, let's suppose, is a bunch of 2-D lines like the lines shown in
the picture. And for each possible line, we will have
set aside a neuron. Don't worry for now about the fact that,
that will require too many neurons. So, for every possible 2-D line, we have
neuron. In any one picture, only a few of the
possible lines will be present. And so , we'll activate just a few of
those neurons. So, I've shown two edges in that picture,
activating two of the neurons. And those are neurons that represent 2-D
lines. They're the data.
Now, what we're going to do is have a whole bunch of 3-D line units, one for
each possible 3-D line or 3-D edge. So, each of the 2-D line units could be
the projection of many different possible 3-D lines. We therefore need to make the
2-D line unit excite all those 3-D lines, but we also need to make them all compete
with one another, cuz you can only see one of them at a time. So, here's an example
where I have a stack of 3-D line units, the green connections are excitrity
connections coming from the 2-D line unit, all of them with equal weight, saying, if
this line unit is present, I'm going to try and turn on all those.
But in addition, we need competition between those so that only one of them
will turn on, and that's what the red lines represent. And we do that for each
2-D line unit. So, I'm just showing it to you for the 2-D line units that happen to
be active at present. And again, don't worry about the fact that
this would need far to many units. Now, the story is not quite complete.
We've now wired into the neural network the information about projection that I
showed on the previous slide, I.e.,
The neural network in those green and red connections understands that each 2-D line
can cross upon to many 3-D edges, but only one of them should be present at a time.
But now, we know a lot about how 3-D edges connect.
For example, when we see two 2-D lines connect in the image, we think it's almost
certain that they correspond to edges that have the same depth at the point where the
lines connect. So, let's suppose that the two 3-D edges
I've joined there correspond to having the same depth at the point where the two 2-D
lines join. That means they should support each other.
It doesn't have to be like that. You could have a very funny viewpoint
where one line ends at a different depth from the other and you just happen to be
at the viewpoint from which they coincide on your retina.
But that's very unlikely. So, we're going to need to use the fact
that we expect 2-D lines that coincide in the image to correspond to 3-D edges that
agree on the depth of that point. So, we'll put in a lot of connections like
that. But there's an even stronger fact we can
use which is that in our manufactured world, we expect that quite often, 3-D
edges will join in a right angle. And so, for two particular 3-D edges that
happen to agree in depth and join at a right angle,
We'll put in a particularly strong connection.
And I've indicated that by a thicker green line.
So, by putting in lots of connections like that, we can indicate how we expect 3-D
edges to go together to form a coherent 3-D object.
And now, we have a network that contains information about how edges go together in
the world and about how edges project to cause lines in the image.
And so, if we give that network an image, it should be able to come up with an
interpretation of the image. And for the image I'm showing you, there's
two quite different interpretations. It's called a Necker cube, and if you look
at it long enough, it will flip in depth on you.
And this network would have two pretty much equally deep energy minima that
correspond to those two interpretations of the Necker cube.
Remember, this is all just a analogy so you understand the idea of using low
energy states as interpretations of perceptial data.
To actually build a proper model of what happens when the Necker cube flips will be
a lot more complicated than this. So, if we decide we're going to use low
energy states to represent good interpretations, then we have two issues.
The first is to do with search and I'm going to deal with that in the next video.
The search question is, how do we avoid the hidden units getting trapped in poor
local minima of the energy function? The poor minima represent interpretations
that are sub-optimal, given our current model and the weights of the network.
Can we do anything better than simply going downhill in energy from some random
starting state? The second issue which seems even more
difficult is how do we learn the weights on the connections between hidden units
and between visible units and hidden units.
Is the sum simple learning algorithm for adjusting all those weights so that we get
sensible perception interpretations? And notice here we haven't got a
supervisor anywhere. We're just showing it input and we would
like it to construct tons of activity in the hidden units that represent sensible
interpretations. This seems like a rather tall order.