In the previous video,
you saw how you can get a neural network to output four numbers of bx,
by, bh, and bw to specify
the bounding box of an object you want a neural network to localize.
In more general cases,
you can have a neural network just output X and
Y coordinates of important points and image,
sometimes called landmarks, that you want the neural networks to recognize.
Let me show you a few examples.
Let's say you're building a face recognition application and for some reason,
you want the algorithm to tell you where is the corner of someone's eye.
So that point has an X and Y coordinate,
so you can just have a neural network have
its final layer and have it just output two more numbers which
I'm going to call our lx and ly
to just tell you the coordinates of that corner of the person's eye.
Now, what if you want it to tell you all four corners of the eye,
really of both eyes.
So, if we call the points, the first, second,
third and fourth points going from left to right,
then you could modify the neural network now to output l1x,
l1y for the first point and l2x,
l2y for the second point and so on,
so that the neural network can output
the estimated position of all those four points of the person's face.
But what if you don't want just those four points?
What do you want to output this point,
and this point and this point and this point along the eye?
Maybe I'll put some key points along the mouth,
so you can extract the mouth shape and tell if the person is smiling or frowning,
maybe extract a few key points along the edges
of the nose but you could define some number,
for the sake of argument, let's say 64 points or 64 landmarks on the face.
Maybe even some points that help you define the edge of the face,
defines the jaw line but by selecting a number of landmarks
and generating a label training sets that contains all of these landmarks,
you can then have the neural network to tell you where are
all the key positions or the key landmarks on a face.
So what you do is you have this image,
a person's face as input,
have it go through a convnet and have a convnet,
then have some set of features,
maybe have it output 0 or 1,
like zero face changes or not and then have it also output l1x,
l1y and so on down to l64x, l64y.
And here I'm using l to stand for a landmark.
So this example would have 129 output units,
one for is your face or not?
And then if you have 64 landmarks,
that's sixty-four times two,
so 128 plus one output units and
this can tell you if there's a face as well as where all the key landmarks on the face.
So, this is a basic building block for recognizing
emotions from faces and if you played with the Snapchat and the other entertainment,
also AR augmented reality filters like
the Snapchat photos can draw a crown on the face and have other special effects.
Being able to detect these landmarks on the face,
there's also a key building block for the computer graphics effects that warp
the face or drawing various special effects like putting a crown or a hat on the person.
Of course, in order to treat a network like this,
you will need a label training set.
We have a set of images as well as labels Y where people,
where someone will have had to go through and
laboriously annotate all of these landmarks.
One last example, if you are interested in people pose detection,
you could also define a few key positions like the midpoint of the chest,
the left shoulder, left elbow, the wrist,
and so on, and just have a neural network to annotate
key positions in the person's pose as well and by having a neural network output,
all of those points I'm annotating,
you could also have the neural network output the pose of the person.
And of course, to do that you also need to specify on
these key landmarks like maybe l1x and l1y
is the midpoint of the chest down to maybe l32x,
l32y, if you use 32 coordinates to specify the pose of the person.
So, this idea might seem quite simple of just adding a bunch of output units
to output the X,Y coordinates of different landmarks you want to recognize.
To be clear, the identity of landmark one must be
consistent across different images like maybe
landmark one is always this corner of the eye,
landmark two is always this corner of the eye,
landmark three, landmark four, and so on.
So, the labels have to be consistent across different images.
But if you can hire labelers or label yourself a big enough data set to do this,
then a neural network can output all of these landmarks which is going to
used to carry out other interesting effect such as with the pose of the person,
maybe try to recognize someone's emotion from a picture, and so on.
So that's it for landmark detection.
Next, let's take these building blocks and use
it to start building up towards object detection.