In this video, I'm going to show a simple
example of a restricted Boltzmann machine
learning a model of images of handwritten
twos.
After it's learned the model, we'll look
at how good it is at reconstructing twos.
And we'll look at what happens if we give
it a different kind of digit and ask it to
reconstruct that.
We'll also look at the weights we get if
we train a restricted Boltzmann machine
that's considerably larger on all of the
digit classes.
It lends a wide variety of features, which
between them are very good at
reconstructing all the different classes
of digits, and also are quite a good model
of those digit classes.
That is, if you take a binary vector, this
image of a hundred digit, the model will
be able to find low energy states,
compatible with that image and if you give
it an image that's a long way away from
being an image of a hundred digit, the
model will not be able to find low energy
states compatible with that image.
I'm now gonna show how a relatively simple
RBM can learn to build a model of images
of the digit two.
The images of sixteen pixels by sixteen
pixels.
And it has 50 binary hidden units that
they're gonna learn to become interesting
feature detectors.
So when it's presented with the data case,
the first thing that it does is use the
weight and the connection from pixel to
feature like this, to activate the
features like this.
That is for each of the binary neurons, it
makes the sarcastic decision about whether
it should deduct the state of one or zero.
It then uses the binary pans for
activation to reconstruct the data.
That is, for each pixel, it makes a binary
decision about whether it should be a one
or a zero.
It then reactivates the binary feature
detectors using the reconstruction to
activate them rather than the data.
The weights are changed by incrementing
the weights between an active pixel and an
active feature detector when the network
is looking at data.
And that will lower the energy of the
global configuration of the data, and
whatever hidden pattern went with it.
And it decrements the weights, between an
active pixel and an active feature
detector, when it's looking at a
reconstruction, and that would raise the
energy of the reconstruction.
Near the beginning of learning when the
weights are random, the reconstruction
would almost certainly have lower energy
than the data.
Because the reconstruction is what the
network likes to reproduce on the visible
units, given the hidden pattern of
activity.
And obviously it likes to reproduce
patterns that have low energy according to
its energy function.
And you can think of what learning does as
changing the weights so the data is low
energy.
And the reconstructions of the data are
generally higher energy.
So let's start with some random weights
for the 50 feature detectors.
We'll use small random weights and each of
these squares shows you the weights to the
pixels coming from a particular feature
detector.
The small random weights are used to break
symmetry.
That because the update were stochastic,
we don't really need that.
After seeing a few hundred examples of
digits, and digesting the weights a few
times, the weights are beginning to form
patterns.
If we do it again, you can see that many
of the feature detectors are detecting the
pattern of a hole two.
They're fairly global feature detectors.
And those feature detectors are getting
stronger,
And stronger, and now some of them begin
to localize, and they're getting more
local, and even more local, and even more
local, and these are the final weights and
you can see that each neuron has become a
different feature detector and most of the
feature detectors are fairly local.
If you look at the feature detector in the
red box, for example, it's detecting the
top of a two,
And it's happy when the top of a two is
where its white pixels are and where
there's nothing where the black pixels
are.
So it's representing where the top of the
two is.
Once we've learned the model, we can look
at how well it reconstructs digits.
And we'll give it some test digits that it
hasn't seen before.
So we'll start by giving it a test example
of a two.
And its reconstruction is pretty faithful
to the test example.
It's slightly blurry.
The test example has a hook at the top and
that's been blurred after the
reconstruction, but it's a pretty good
reconstruction.
The more interesting thing we can do, is
give it a test example from a different
digit class.
So if we give it an example of the three
to reconstruct.
What it reconstructs actually looks more
like a two then like a three.
All of the feature detectors is learned,
are good for representing twos, but it
doesn't have feature detectors for things
like representing that cusp in the middle
of the three.
So it ends up reconstructing something,
but obeys the regularities of a two,
better than it represents the regularities
of a three.
In fact, the network tries to see
everything as a two.
So here's some feature detectors that were
learned in the first hidden layer of a
model that uses 500 hidden units to model
all ten digit classes.
And this model has been trained for a long
time with contrastive divergence.
It has a big variety of feature detectors.
If you look at the one in the blue box,
that's obviously going to be useful for
detecting things like eights If you look
at the one in the red box, it's not what
you expect to see.
It likes to see pixels on very near the
bottom there, and it really doesn't like
to see pixels on in a road that's 21
pixels above the bottom.
What's going on here is that the data is
normalized and so the digits can't have a
height of greater than twenty pixels.
And that means if you know that there's a
pixel on where those big positive weights
are, there can't possibly be a pixel on,
where those negative weights are.
So this is picking up on the long range
regularity that was introduced by the way
we normalized the data.
Here's another one that's doing the same
thing for the fact that the data can't be
wider than twenty pixels.
The feature detected in the green box is
very interesting.
It's for detecting where the bottom of a
vertical stroke comes and it will detect
it in a number of different positions and
then refuse to detect it in the
intermediate positions.
So it's very like one of the least
significant digits in a binary number, as
you increase the magnitude of a number it
goes on again, and off again, and on
again, and off again and it shows that
this is developing quite complex ways of
representing where things are.