In this video, I'm going to talk about some recent work on learning a joint
model of captions and feature vectors that describe images.
In the previous lecture, I talked about how we might extract semantically
meaningful features from images. But we were doing that with no help from
the captions. Obviously the words in a caption ought to
be helpful in extracting appropriate semantic categories from images.
And similarly, the images ought to be helpful in disambiguating what the words
in the caption mean. So the idea is we're going to try in a
great big net that gets its input, stand to computer vision feature vectors
extractive for images and pack up words representations of captions and learns
how the two input representations are related to each other.
At the end of the video I'll show you a movie of the final network using words to
create feature vectors for images and then showing you the closest image in its
data base. And also using images to create bytes of
words. I'm now going to describe some work by
Natish Rivastiva, who's one of the TAs for this course, and Roslyn Salakutinov,
that will appear shortly. The goal is to build a joint density
model of captions and of images except that the images represented by the
features standardly used in computeration rather than by the ropic cells.This needs
a lot more computation than building a joint density model of labels and digit
images which we saw earlier in the course.
So what they did was they first trained a multi-layer model of images alone.
That is it's really a multi-layer model of the features they extracted from
images using the standard computer vision features.
Then separately, they train a multi-layer model of the word count vectors from the
captions. Once they trained both of those models,
they had a new top layer, that's connected to the top layers of both of
the individual models. After that, they use further joint
training of the whole system so that each modality can improve the earlier layers
of the other modality. Instead of using a deep belief net, which
is what you might expect, they used a deep Bolton machine, where the symmetric
connections bring in all pairs of layers. The further joint training of the whole
deep Boltzmann machine is then what allows each modality to change the
feature detectors in the early layers of the other modality.
That's the reason they used a deep Boltzmann machine.
They could've also used a deep belief net, and done generative fine tuning with
contrastive wake sleep. But the fine tuning algorithm for deep
Boltzmann machines may well work better. This leaves the question of how they
pretrained the hidden layers of a deep Boltzmann machine.
because what we've seen so far in the course is that if you train a stack of
restricted Boltzmann machines and you combine them together into a single
composite model what you get is a deep belief net not a deep Boltzmann machine.
So I'm now going to explain how, despite what I said earlier in the course, you
can actually pre-trail a stack of restrictive Boltzmann machines in such a
way that you can then combine them to make a deep Boltzmann machine.
The trick is that the top and the bottom restrictive bowser machines in the stack
have to trained with weights that it twices begin one directions the other.
So, the bottom Boltzmann machine, that looks at the visible units is trained
with the bottom up weights being twice as big as the top down weights.
Apart from that, the weights are symmetrical.
So, this is what I call scale symmetrical.
But the bottom up weights are always twice as big as their top down
counterparts. This can be justified, and I'll show you
the justification in a little while. The next restrictive Boltzmann machine in
the stack, is trained with symmetrical weights.
I've called them two W, here rather then W for reasons you'll see later.
We can keep training with restrictive bowsler machines like that with genuinely
symmetrical weights. But then the top one in the stack has
be-trained with the bottom up weights being half of the top down weights.
So again, these are scale symmetric weights, but now, the top down weights
are twice as big as the bottom up weights.
That's the opposite of what we had when we trained the first restricted Bolton
machine in the stack. After having trained these three
restricted Bolton machines, we can then combine them to make a composite model,
and the composite model looks like this. For the restricted Bolton machine in the
middle, we simply halved its weights. That's why they were 2W2 to begin with.
For the one at the bottom, we've halved the up-going weights but kept the
down-going weights the same. And for the one at the top we've halved
the down-going weights and kept the up-going weights the same.
Now the question is: Why do we do this funny business of halving the whites?
The explanation is quite complicated but I'll give you a rough idea of what's
going on. If you look at the layer H1.
We have two different ways of inferring the states of the units in h1, in the
stack of restricted bolts and machines on the left.
We can either infer the states of H1 bottom up from V or we can infer the
states of H1 top down from H2. When we combine these Boltzmann machines
together, what we're going to do is we're going to an average of those two ways of
inferring H1. And to take a geometric average, what we
need to do, is halve the weights. So we're going to use half of what the
bottom up model says. So that's half of 2W1.
And we're going to use half of what the top down model says.
That's half of 2W2. And if you look at the deep Boltzmann
machine on the right, that's exactly what's being used to infer the state of
H1. In other words, if you're given the
states in H2, and you're given the states in V, those are the weights you'll use
for inferring the states of H1. The reason we need to halve the weights
is so that we don't double count. You see, in the Boltzmann machine on the
right. The state of H2 already depends on V.
At least it does after we've done some settling down in the Boltzmann Machine.
So if we were to use the bottom up input coming from the first restricted
Boltzmann Machine in the stack. And we use the top down input coming from
the second Boltzmann Machine in the stack, we'd be counting the evidence
twice.'Cause we'd be inferring H1 from V. And we'd also be inferring it from H2,
which, itself, depends on V. In order not to double count the
evidence, we have to halve the weights. That's a very high level and perhaps not
totally clear description of why we have to half the weights.
If you want to know the mathematical details, you can go and read the paper.
But that's what's going on. And that's why we need to halve the
weights. So that the intermediate layers can be
doing geometric averaging of the two different models of that layer, from the
two different restricted Boltzmann machines in the original stack.