It was always a question for speculation
whether the kinds of nets developed for
recognizing handwritten digits could
actually be scaled up to what vision
people call a real task.
That is, recognizing objects in high
resolution color images when the scene is
cluttered.
So that you have to do things like
segmentation, you have to deal with 3D
viewpoint, you have to deal with 5-foot
list.
Many different objects surrounding, you're
not quite sure which is the intended one,
and so on.
Since the start of this course, we've got
some interesting new results on that.
So in my first lecture, I described the
network developed by Alex Krizhevsky and
showed that it was good at object
recognition.
But at that point it hadn't been
benchmarked against the best computer
vision systems,
Now it has.
People worked on Emenise for many years,
gradually improving their ability of these
networks to recognize handwritten digits.
Many computer vision researchers thought
this was a waste of time if you wanted to
be able to recognize real objects in color
images, because they thought the lessons
learned from Emnise would not generalize
to that domain.
That was a fairly reasonable thing to
think.
Here's a number of reasons why it's a much
more difficult task.
First of all, there's many, many more
different kinds of objects.
Even if we only recognize a thousand
classes, that's still a factor of a
hundred.
Secondly there is many more pixels even if
we use trans sampled images that are only
256 by 256 with color pixels that's still
100 or 300 times of many pixels Another
factor is that in real scenes you have to
deal with the fact you've got a two
dimensional image of a three dimensional
reality so a lot of information is being
lost.
And real scenes have clutter of a kind
that doesn't occur in handwriting.
In handwriting you can have overlapping
letters and that requires segmentation but
you don't have things like occlusion of
large parts of objects by opaque other
objects.
You don't have many different kinds of
objects in the same scene.
And you don't have a little lighting
variations that you get in real scenes.
So the question is will the same kind of
convolution neural network that proved to
be so good on recognizing hand written
digits work for real color images In the
domain of real color images we probably do
need to wire in some prior knowledge.
Because, if we try and do it in the sera
san way with no knowledge wired in,
putting in all the knowledge by generating
extra training examples.
The computational problem is still too
large for current computers.
So there was a recent competition.
And it was on a data base called ImageNet.
ImageNet actually has many more than a
machine images but a subset of 1.2
millimeters was chosen and the
classification task was to correctly label
those images.
Now the images were hand labelled with a
thousand different classes but this wasn't
very reliable.
There could be an image that has two of
those thousand different objects in it and
only one of them is labeled.
So, to make the task feasible the computer
vision system is allowed to make five
bets.
And it's set to get it right if one of
those bets corresponds to the label that a
person has given the image.
There's also a localization task.
The reason for the localization task is
that many computer vision systems use a
bag of features approach.
For the whole image or for say, a quadrant
of the image they know what the features
are, but they don't know where they are.
This allows them to recognize objects but
without knowing exactly where they are.
That's very unlike how people behave
except people with a curious kind of brain
damage called balance syndrome where they
can recognize objects and not be sure
where they are.
So for the localization task you have to
place a box around an object once you've
recognized it and to get it right your box
must have at least a 50% overlap with the
correct box.
On this task, people tried some of the
best existing computer vision methods.
So, leading groups from Oxford and the
French National Research Labs Inria and
Xerox's European Research Center and
various other universities tried this task
and discovered it's very hard.
The computer vision systems typically use
complicated multi-stage systems.
The early stages of these systems are
typically hand-tuned by optimizing a few
parameters using some of the data.
And, the top stage of these systems is
always a learning algorithm.
But they don't learn all the way through.
In the way that a deep neural net does
when its trained to do back propagation.
They don't have end-to-end learning, where
the parameters used in the early feature
detectors are being influenced by how
useful they are for making final decision
about classes.
So here are some examples from the test
set to show you what the data is like.
You already sow some examples in the first
lecture, but here's some more.
So you can see that it's fairly obvious
what the object is in that image, but a
lot of it's missing.
It doesn't have ears, it doesn't have
legs.
The predictions are the un-normalized
probabilities of Alex Krizhevsky's
deep-neural-network.
And you can see it's confident that, that
is a cheetah And if it's not a cheetah, it
thinks it almost sudden a leopard.
It also understands there's other
possibilities, like a snow leopard, it's
the wrong color for a snow leopard, or an
Egyptian cat.
Here's an example the other way around,
here there's many objects in the image and
the object of interest is only a very
small fraction of the pixels.
The network correctly says bullet train.
But it also has other bets, like subway
train or electric locomotive, which are
presentable bets.
If you look at the image, there's lots of
other things that could be labeled, like
the roof which occupies a much larger
fraction of the image than the train or
the pillar that's supporting the roof or
the pedestrian.
Or the large apartment block in the
background.
In these kinds of images you really have
to be able cope with the fact that there's
lots of alternative targets.
Last image shows a different kind of
example where there is no background
clutter.
The object is quite well isolated,
probably a picture from a catalog or
something.
And the network doesn't get it right for
its first bet, but it does get it in its
top five bets.
But here the network isn't confident about
anything.
These are the relative probabilities, and
the network correctly realizes it doesn't
really know.
And if you look at the other
possibilities, they're all perfectly
plausible.
If you screw your eyes up so you can't see
the image too well, you can see how it
might think it was a frying pan or a
stethoscope.
So how did the systems do on this data?
Here's the error rates for the computer
vision systems.
One thing you'll notice is that the best
systems are all very similar.
So the University of Tokyo managed to get
26.1%, and here what I'm doing is just
reporting the best system from each group.
Oxford University, which has a very good
computer vision group, generally
recognized to be possibly the best group
in Europe, again got in the 26 percents
and the French National Research Labs in
the Xerox Park Center, which are, again, a
very good computer vision groups, got 27%.
So you'll guess from this that it is going
to be hard to be 26%, and if you do beat
26% you're comparable with the very best
computer vision systems.
So Alex Krizhevsky's neural net got
sixteen percent error.
It's a huge gap.
Normally, in these competitions you don't
see big gaps like that.
So Alex Krizhevsky's network works like
this.
It's a very deep convolutional neural net
of the type pioneered by Yann Le Cun was
first used for digit recognition and then
Yann later applied it to recognizing real
objects.
And we're using all the lessons that we
learned by Yann's group from [UNKNOWN]
group and various other groups, developing
these deep neural nets for doing real
vision.
It has seven hidden layers, which is
deeper than usual and that's not counting
some of the max-pooling layers.
The early layers are convolutional.
We could probably get away with using just
local receptive fields, without tying any
weights, if we had a much bigger computer.
But by making them convolutionary, you cut
down the parameters a lot, so you cut down
the amount of training data you need a lot
which cuts down the amount of computation
time a lot.
The last two loads were globally connected
and that's where most of the parameters
are.
I think there's about sixteen million
parameters between each pair of those
layers.
What the last two layers are doing is
looking for combinations of local features
that were extracted by the early layers.
And obviously this commonly tourily many
combinations to look for.
And that's why you need a lot of
parameters there.
The activation functions were rectified
linear units in every hidden layer.
These train much faster than logistic
units and they're more expressive.
Most of the people seriously applying deep
in your own networks to real images to the
object recognition of I switch direct fi
linear units.
We also used competitive normalization,
within a layer to suppress the activity of
a unit, if other units that are looking
nearby localities are very active.
This helps a lot with variations in
intensity.
So, you might have an edge detector, which
gets somewhat active due to some fairly
faint edge.
And that's pretty much irrelevant, if
there is much more intense things around.
There's other tricks that we used to
significantly improve the generalization
of this net.
First of all, we use the trick of
enhancing the data by using
transformations.
So here's a skit for down-sampled the
images in the competition to 256 by 256.
But instead of using those whole images
Alex Krizhevsky took random 224 by 224
patches from those images.
Which gave him hugely more images to train
on.
And helped him deal with translation and
variance.
Even though they're convolutional nets
that's still a help.
He also used left-right reflections of the
images, which again doubled the amount of
data.
He didn't use up dime reflections.
Because, gravity's very important.
Left right reflections don't really change
what things look like much unless they're
things like writing.
At test time, he doesn't just use one
patch.
He uses a number of different patches, the
four corners, the middle, that gives him
five, and then the left right reflections
of all those, that gives him ten.
He runs all ten through the network and
then combines their opinions.
In the top layers, where most of the
parameters are, he uses a new
regularization technique, called drop-out,
which is very effective.
And stops the network over fitting.
That's worth several percent in his
results.
I'll describe drop pouch at some length in
the later lecture.
But for now, the basic idea of drop out is
that each time you present a training
example, you omit half the hidden units
from a layer.
This means that the other hidden units in
that layer, the survivors, can't rely on
the their com rates being present.
They can't learn to fix up the errors left
over by the other hidden units in that
layer, cuz the other hidden units might
not be there no matter be fixing up an
error that doesn't exist.
So they have to become more individualist.
They have to individually do useful things
but they still have to do useful things
that are different from what the other
survivors do.
So drop out is stopping too much
cooperation between the hidden units.
And a lot of cooperation is very good for
fitting the training data.
But if the test distribution is
significantly different, then all that
cooperation causes over-fitting.
Alex couldn't have done this work without
significant hardware, but the hardware
only costs a few thousand dollars now.
Alex is a very good programmer, and he
used a very efficient implementation of
convolution and neural nets on two Invidia
GTX 580 graphics processors.
Each of these has over 500 fast little
cores, which are very good at doing
arithmetic and not much good at anything
else.
The GP use are very good at doing matrix,
matrix multiplies.
So if you stack together the vector of
activities of a hidden layer, over many
training cases, that gives you a matrix.
And now you multiply that by matrix of
weights to figure out the activities in
the next hidden layer for all those
training cases.
And if both those matrices are big, the
GPU's give you a huge advantage.
They give you about a factor of 30.
They also have very high bandwidth to
memory, and that's needed for neural nets.
Cause in neural nets you keep wanting to
know another weight so that you can
multiply it by an activity.
And there's millions of these weights, so
you can't keep them all in the cache.
Using all that hard brac, he could train
his final network, in a week.
And you could also combine results from
ten, ten different patches of TestTime
very quickly.
So Test Time you can run it at just about
the frame rate.
In the future we are going to be able to
spread this kind of network over a large
number of calls.
As calls become cheaper, people at Google
are already experimenting with that.
And if we can communicate the stakes fast
enough we are going to be able to do much
bigger networks on many more calls.
Google has already simulated networks with
1.7 billion connections and I think that
it's only going to get bigger.
As the cores get cheaper and the data sets
get bigger, these big deep neural nets are
gonna improve much faster than the
old-fashioned computer vision systems,
because they don't involve much hand
engineering, and they can make very good
use of huge data sets and huge amounts of
computation.
So the fact that we've already opened up a
big gap I think means there's no looking
back.
I think from now on all the best object
recognition systems, at least of static
images, will use big deep neural nets.
There are other application domains where
we've learned the same lesson so Vladimir
Nee.
Used a net with local fields but without
convolution to extract roads from aerial
images.
These are cluttered aerial images of urban
scenes.
Again he uses multiple layers of rectified
linear units.
And he takes a relatively large image
patch, and predicts for the central 16x16
pixels whether each of those pixels is a
piece of road or not a piece of road.
The nice thing about this task is that
there's a lot of label training data
available.
That's because maps tell you where the
centre lines of roads are and roads are
roughly fixed width.
So from the vectors in the map that tell
you where the centre line of the road is
you can estimate which pixels are probably
road.
Nevertheless, the task is very hard.
There's the normal kind of vision
problems: so roads are occluded by
buildings because a plane isn't looking
straight down when it takes the
photograph.
They're occluded by trees.
They're also occluded by cars that are
sitting on the road.
The shadow effects from building, the
major lighting changes depending on
whether it's a sunny day or a cloudy day
for example and there's minor view point
changes.
So the plane is basically looking
downwards, but in any large photo it can't
be looking straight downwards at every
pixel.
The worst problems in this data are the
incorrect labels.
You get incorrect labels because the maps
aren't perfectly registered.
For most purposes, you don't need a map to
be registered better than a few meters.
The pixels are about one meter square in
this data.
And so if the registration of the map is
off by three meters, you're going to get
at least three of the labels wrong for
pixels, across every road.
Another, severe problem, is that the
people making maps have to make arbitrary
decisions about what counts as a road and
what counts as a laneway.
So, in may of the maps, you look at
something, and you've no idea whether
that's gonna be considered to be a road or
a lane-way.
And so you simply don't know what label
it's gonna get from the map.
Big neural nets trained on big image
patches, using millions of examples, are,
I think, the only real hope for doing a
good job at this task.
It's very hard to find out what people can
do.
So, here is what the data looks like.
This is a part of Toronto.
If you know Toronto, you can tell that by
the angle of the roads.
And, above the image of the part of
Toronto, I put two patches extracted from
that image.
And if you look at those patches, you can
see it's not trivial to tell which the
road pixels are.
On the right, is the output of [UNKNOWN]
system.
Green is correctly identified pixels of
road, and red means things that his system
thought might be road, but actually
aren't.
Actually that thing is a parking lot but
you can see why he might have thought it
was a road.