In this video, I'm going to talk about convolusional neural networks for hundred
and digit recognition. This was one of the big success stories of
neuron networks in the 1980s. The deep convolutional nets developed by
Yann LaCun and his collaborators did a really good job of recognizing handwriting
and were actually used in practice. They're one of the few examples from that
period of deep neural nets that it was possible to train on computers that
existed then, and that performed really well.
Convolutional neural networks are based on the idea of replicated features.
So, because objects move around and show up on different pixels, if we have a
feature detector that's useful in one place in the image, it's likely that the
same feature detector will be useful somewhere else.
So, the idea is to build many different copies of the same feature detector in all
the different positions. If you look on the right I've shown you
three feature detectors, which are replicas of each other.
Each of them has weights to nine pixels. And those weights are identical between
the three different feature detectors. So the red arrow has the same weight on it
for all three feature detectors. And when we learn we keep those red arrows
all having the same weight as each other and we keep the green arrows having all
the same weight as each other. Even though the red and green arrows will
have different weights. We could also try replicating across scale
and orientation but that's much more difficult and expensive and probably not a
good idea. Replication across position greatly
reduces the number of free parameters that you have to learn.
So those 27 pixels that you see in those three replicated detectors only have nine
different weights. Now we don't just want to use one feature
type. So we're going to have many maps.
Each map will have replicas of the same feature, features that are constrained to
be identical in different places. And then different maps will learn to
detect different features. This allows each patch of the image to be
represented by features of many different types.
Replicated features fit in nicely with back propagation that is it's easy to
learn using back propagation. In fact its easy to modify the back
propagation algorithm incorporate any linear constraint between the weights.
So what we do is we compute the gradients as usual.
But then we modify the gradients, so that if the weight satisfied the linear
constraint before the weight update, they'll also satisfy the linear constraint
after the weight update. So, the simplest example is we want two
weights to be equal. We want w1 to equal w2.
That would be true if we start off with W1 equal to W2.
And then we make sure that the change in W1 is always equal to the change in W2.
The way we do that is we compute the gradient of the arrow with respect to W1.
And the gradient with respect to W2. And then we use the sum or average of
those two gradients for both W1 and W2. By using weight constraints like that, we
can force back propagation to learn replicated feature detectors.
There's quite a lot of confusion in the literature about what replicated feature
detectors are actually achieving. Many people claim they're achieving
translation invariance. And that's not true.
Well at least it's not true in the activities of the neurons.
So if you look at the activities, what replication features achieve is
equivariance not invariance. An example should make that clear.
Here's an image, and the black dots are the activated neurons.
Here's a translated image. And notice the black dots have also
translated. So the image changed and the
representation also changed by just as much as the image.
That's equivariance not invariance. There is somethings invariant, and that's
the knowledge. So if you learn replicative feature
detectors, if you know how to detect a feature in one place, you'll know how to
detect that same feature in another place. And it's important to note that we're
achieving equivariance in the activities and invariance in the weights.
If you want to achieve some invariance in the activities, what you need to do is
pool the acts replicative feature detectors.
So you can get a small amount of translation in variance at each level of a
deep net, by averaging full neighboring replicated detectors.
One advantage of this is that it reduces the number of inputs for the next layer.
So that we can have more different maps, allowing us to learn more different kinds
of features in the next layer. It actually works slightly better to take
the maximum of four neighboring feature detectors, rather than an average, but
there is a problem. And the problem is that after several
levels of doing this kind of pooling, we've lost precise information about where
things are. That's okay if we just want to recognize
that it's a face. The fact that we've got a few eyes, and a
nose, and a mouth floating about in vaguely the same position is very good
evidence that it's a face. But if you want to recognize whose face it
is, you need to use the precise spatial relationships between the eyes and between
the nose and the mouth. And that's been lost by these
convolutional neural nets. I'll come back to that issue later on.
So the first impressive example of a convolution on your own end was done by
Yann Lecun and his collaborators who developed a really good recognizer for a
hundred digits. In it had many hidden layers.
In each layer, it had many maps of replicated units.
And it had pooling between layers. So you pool adjacent replicated units
before you send them to the next layer. But I also used a wide net that could cope
with several characters at once. And that would work even if the characters
overlapped. So you didn't have to segment out
individual characters before you fed them to their net.
And something which, people often forget, is that they used a clever way of training
a complete system. They weren't just training a recognizer of
individual characters. They were training a complete system, so
that you put in pixels at one end and you get out whole zip codes at the other end.
And in training that system they used a method that would now be called maximum
margin. But when they did it, it was way before
maximum margin had been invented. The net they used was at one point
responsible reading about for ten percent of the checks in North America.
So it was of great practical value. There were some very nice demos on that,
on Yann's workpaget. You should really go look at them.
Look at all of them. Because they show you just how.
Well it copes with variations in size, orientation, position, overlap of digits,
and all sorts of background noise that would, would kill most methods.
The architecture of LeNet-5 looks like this.
There's an input, which is pixels. And then there's a whole sequence of
feature maps followed by sub sampling. So in the C1 feature map, the six
different maps, each which is 28 by 28. Of those maps contain small features that
just look at I think three by three pixels.
And their weights are constrained together.
So per map there's only about nine parameters.
That makes learning much more efficient. It means you need much less data.
Then after the feature map, there's what they call sub-sampling which is now called
pooling. And so, you pool together the outputs of a
bunch of neighboring replicated features in C1.
And that gives you a smaller map, which will then provide the input to the next
layer, which is discovering more complicated replicated features.
As you go up this hierarchy, you get features that are much more complicated,
but are more invariant to position. Here's the errors that LeNet5 made.
And this shows you the data that it's dealing with is quite tricky.
There's 10,000 test cases, and these are the 82 errors that it makes.
So it's doing better than 99% correct. Nevertheless, most of the errors it makes
are the things that people find quite easy to recognize.
So there's some way to go still. Nobody knows the human error rate on this
data. But it's probably twenty to 30 errors.
Of course the might-be digits that LeNet5 got right and you would get wrong.
So you have to be careful in estimating the error rate.
You can't just look at these 82 and ask which ones you'll get right and which ones
you'll get wrong. You have to worry about all those other
ones that Lynette Five might've got right and you might've got wrong.
I'm now want to go to a very general point, about how to inject prior knowledge
in machine learning, and it applies particularly to neural networks.
We can put in prior knowledge as it is done in the net five, by the design of the
network. We can have local connectivity.
We can have weight constraints. Or we can choose neuro-activites that are
particularly appropriate for the task we're doing.
This is much less intrusive than trying to hand-engineer the futures.
But it still prejudices the network towards a particular way of solving the
problem that we had in mind. We have an idea about how to do object
recognition by gradually making bigger and bigger features.
And by replicating these features across space.
And we force the network to do it that way.
There is an alternative way to put in prior knowledge that gives the network a
much freer hand. What we can do is use our prior knowledge
to get a whole lot more training data. One of the first examples of this was work
by Hofmann and Tresp on trying to model what happens in a steel mill.
They wanted to know the relationship between what comes out of the steel mill
and various input variables, and they actually had an, big old Fortran simulator
that would allow them to simulate the steel mill.
Of course, the simulator wasn't reality. It was making all sorts of approximations.
So they had real data, and also a simulator.
And what they did was run the simulator in order to create some synthetic data.
We then added that to the real data, and showed that they could do better than just
using the real data alone. If I remember right, their great, big
Fortran simulator was only worth a few dozen extra real examples, but
nevertheless, they made the point. Of course, if you generate a lot of
synthetic data, it may make learning take much longer.
So in terms of the speed of learning, it's much more efficient to put in knowledge by
using things like connectivity and weight constraints, as was done in Lynette five.
But as computers get faster, this other way of putting in knowledge, by generating
synthetic examples, begins to look better and better.
In particular, it allows optimization to discover clever ways of using the
multilayer network that we didn't think of.
If fact, we might never fully understand how it does it.
If we just want good solutions to a problem, that might be fine.
So using the idea of synthetic data, there's a brute force approach to
handwritten digit recognition. Lenet5 uses knowledge about invariances to
design the connectivity and the weight sharing and the pooling.
And that achieves about 80 errors. Adding a lot more tricks, including
synthetic data, [UNKNOWN] was able to get that down to about 40 errors.
A group in Switzerland, led by [UNKNOWN] went to town with injecting knowledge by
putting in synthetic data. They put a lot of work into creating very
instructive synthetic data. So for every real training case, they
transformed it to make many more training examples.
They then trained a large net with many units per layer, many layers on a graphic
processor unit. The graphics processor unit gave them a
factor of thirteen computation. And because of all the synthetic data they
put in, it didn't overfit. If they just use a large net with a GPU.
It would have been a disaster that over fitted terribly that they would have done
on the training data but terribly on the test data.
So they were really combining three tricks.
Put your effort in to generating lots of synthetic data then train a large net on a
gpu. They managed to achieve 35 errors like
that. So here's the 35 errors that they got.
The top printed digit is the right answer. And the bottom two digits are their top
two answers. What you'll notice is that they nearly
always get the right answer in their top two.
There's only five cases where they don't. With some more work by building several
different models like this and then using a consensus to decide what the digit was,
they managed to get down to about 25 errors.
And that must be around about the human error rate.
One question this work raises is how do you tell if a model makes 30 errors is
really better than a model that makes 40 errors.
Is that significantly different? Rather surprisingly, it turns out it
depends on which errors they make. The numbers then provide you enough
information. You have to know which ones they get wrong
and which ones they get right. So this statistical test called the
McNemar test that uses the particular errors and is far more sensitive than just
using the numbers. Let me give you an example.
If you look at this two by two tape. It shows you, in the top left hand corner,
how many examples Model one got wrong and Model two also got wrong.
That's 29. And in the bottom right, it shows you how
many examples Model one got right and Model two also got right.
And in the Magnema Test, you can just ignore those numbers in black.
All you're interested in is ones where Model one got it right and Model two get
it wrong, or Model two got it right and Model one get it wrong.
And if you look at that, there's an eleven to one ratio, and it turns out that's
pretty significant. Model two is definitely better than model
one. That didn't happen by accident, almost
certainly. By contrast if you look at this table,
again. Model one is making 40 hours, model two is
making 30 hours, but now model one is winning fifteen times when model two loses
and model 2's winning 25 times when model one loses.
That difference is not very significant so we wouldn't be confident that model two is
better than model one.