In this video, I'm going to describe some of the things that make it difficult to
recognize objects in real scenes. We're incredibly good at this, and so it's
very hard for us to realize how difficult it is to take a bunch of numbers that
describe the intensities of pixels and go from there to the label of an object.
There's all sorts of difficulties. We have to segment the object out.
We have to deal with variations in lighting, in viewpoint.
We have to deal with the fact that the definitions of objects are quite
complicated. It's also possible that to get from an
image to an object requires huge amounts of knowledge, even for the lower level
processes that involve segmentation and dealing with viewpoint and lighting.
If that's the case, it's gonna be very hard for any hand engineered program to be
able to do a good job of those things. There are many reasons why it's hard to
recognize objects and images. First of all, it's hard to segment out an
object from the other things in an image. In the real world, we move around, and so
we have motion cues. We also have two eyes, so we have stereo
cues. You don't get those in static images.
So it's very hard to tell which pieces go together as parts of the same object.
Also, parts of an object can be hidden behind other objects, and so, you often
don't see the whole of an object. You're so good at doing vision that you
don't often notice this. Another thing that makes it very hard to
recognize objects is that the intensity of a pixel is determined as much by the
lighting as it is by the nature of the object.
So, for example, a black surface in bright light will give you much more intense
pixels than a white surface in very gloomy light.
Remember, to recognize an object you've got to convert a bunch of numbers, that is
the intensities of the pixels, into a class label.
And, these intensities are varying for all sorts of reasons that have nothing to do
with the nature of the object. Or nothing to do with the identity of the
object. Objects can also deform in a variety of
ways. So for relatively simple things like
handwritten digits there's a wide variety of different shapes that have the same
name. A two for example could be very italic
with just a cusp instead of a loop or it could be a very loopy two with a, a big
loop and very round. It's also the case that for many types of
object, the class is defined more by what the object's for, than by its visual
appearance. So consider chairs.
There's a huge variety of things we call chairs, from armchairs to modern chairs
made with steel frames and wood backs to basically anything you can sit on.
One other thing that makes it hard to recognize objects, is that we have
different viewpoints. So, there's a wide variety of viewpoints
from which we can recognize a 3D object. Now, changes in viewpoint cause changes in
the images that standard machine learning methods cannot cope with.
The problem is that information hops about between the input dimensions.
So, typically envision the input dimensions correspond to pixels, and, if
an object moves in the world and you don't move your eyes to follow it, the
information about the object will occur on different pixels.
That's not the kind of thing we normally have to deal with in machine learning.
Just to stress that point, suppose we had a medical database in which one of the
inputs is the age of a patient and another input is the weight of the patient.
And we start doing machine learning, and then we realize that some coder has
actually changed which input I mentioned is coding which property.
So for one of the coders they've put weight where they should have put age, and
they put age where they should have put weight.
Obviously, we wouldn't just carry on doing our learning.
We'd try and do something to fix that. That's going to make everything go wrong.
I call that phenomenon dimension hopping. When information jumps from one input
dimension to another. And that's what Viewpoint does and it's
something we need to fix. And preferably we'd like to fix it in a
systematic way.