In this video, we're gonna look at the
limitations of perceptrons.
These limitations stem from the kinds of
features you use.
If you use the right features, you could
do almost anything.
If you use the wrong features, they're
extremely limited in what the learning
part purpose that font can do.
And that's what cause perceptrons to guard
favor, and it emphasizes that the
difficult bit of learning is to learn the
right features.
There's still a lot you can do with
learning, even if you do not learn
features.
For example, if you want to say whether a
sentence is a plausible English sentence,
you could hand define a huge number of
features, and then learn how to write them
in order to decide whether particular
sentence is likely a good English
sentence.
But, in the long run you need to learn
features.
So the reason that neural network research
came to a halt in the late 1960s and early
1970s is that perceptrons were shown to be
very limited, and we're now gonna
understand what those limitations are.
If you'd like to choose the features by
hand, and if you use enough features, you
can make the perceptron do almost
anything.
Suppose for example we have binary input
vectors.
And we create a separate feature unit that
gets activated by exactly one of those
binary input vectors.
We'll need exponentially many feature
units.
But now we can make any possible
discrimination on binary input vectors.
So for binary input vectors there's no
limitation if you're willing to make
enough feature units.
But of course, that's not a very good
strategy for solving a practical problem
because you need an awful lot of feature
units and it won't generalize.
You can't look at a subset of all possible
cases and have any hope of getting the
remaining cases right because those
remaining cases require new feature units
and you don't know what weights to put on
those feature units.
Once you've decided the hand coded
features.
That is once they've been determined,
there are very strong limitations on what
a perceptron can learn to do.
So here's a classic example.
What we're interested in is what can you
learn to do with the binary threshold
decision unit that is by changing its
weights.
And we're going to show that there's very
simple things that it can learn to do.
So the simplest example is consider a
problem in which there's two positive
cases and two negative cases.
And the features, just single bit
features, that have values either one or
zero.
So the two positive cases consist of both
features being on.
In which case the right answer's one.
Or both features being off.
In which case the right answer's one.
And the two negative cases are when one
feature's on and the other one's off.
In which case the right answer is zero.
So all we're asking the binary threshold
unit to do is decide whether the two
features have the same value.
And they can't even learn to do that.
We can prove that algebraically.
Those four input/output pairs that I
showed you give rise to four inequalities,
and it's impossible to satisfy them.
So the first positive case, when the two
feature values are one the output should
be one.
That gives us the inequality that: one
times W1 plus one times W2 is gonna be
greater than the threshold.
So we give an output a one.
Then the second positive case gives us the
inequality that zero times W1 plus zero
times W2, must also be greater than the
threshold.
And the negative cases give us the
inequalites that one times W1 plus zero
times W2, must be less than the threshold,
and similarly zero times W1 plus one times
W2, must be less than the threshold.
Now if you take those first two
inequalities and you add them up, you get
the W1 plus W2 must be greater than twice
the threshold.
And if you take the second two
inequalities and you add them up, you get
W1 plus W2 must be less than twice the
threshold.
So there's clearly no way to satisfy all
four inequalities.
Or to put it another way, if you look at
the binary decision unit where we're going
to put the threshold as a negative weight
on an input line that always has value of
one.
If you take that binary threshold unit
shown at the bottom right, there's no way
to set the threshold in the two weights,
so it gets all four cases right.
We can also see this geometrically.
So we're going to imagine a data space
now, in which the axis correspond to
components of an input vector.
So in this space each point corresponds to
a data point.
And, a weight vector is going to find a
plane in this space.
So it's just the opposite of what we're
doing with weight space.
In weight space we made each point be a
weight vector, and we used a plane, to
define an input case.
Of course that plane was defined by an
input vector.
Now what we're going to do is we're going
to make each point be an input vector and
we're going to use a wait vector to define
a plane in the data space.
The plane defined by the weight vector is
going to be perpendicular to the weight
vector and it's going to miss the origin
by a distance equal to the threshold.
So here's a picture.
You see the four data cases there, and for
the two data cases in red, we need to give
an output of zero.
And with the two data cases in green, we
need to put an output of one.
That me, means we need the green cases to
be on the side of the weight plane where
the output is one and we need the red
cases to be on the side where the output
is zero, and we obviously cannot arrange
the weight plane so that's true.
We call a set of cases like that, where
there's no hyperplane that will separate
the cases where we want the answer to be
one, from the cases where we want the
answer to be zero.
We call that a set of training cases
that's not linearly separable.
And even more devastating example for
perceptrons because it's much more general
is when we try and discriminate simple
patterns that have to retain the identity
when you translate them with wrap-around.
I'll give you an example of what that
means in a minute.
But the idea is that we want to recognize
a pattern and we want to recognize it even
when it's translated.
So suppose we just use pixels as the
features.
The question is can a binary threshold
unit discriminate between two different
patterns.
We'll call one positive example and the
other's negative examples if they've got
the same number of pixels in them.
And the answer is no it can't discriminate
two patterns out of the same number of
pixels if that discrimination has to work
when the patterns are translated and if
the patterns can wrap-around when
translate.
So, if you look at these examples of
pattern A, in a one-dimensional image.
Pattern A has four pixels that are on.
Those four black pixels.
It's like a little bar code.
And it's the same pattern when we
translate it a bit to the right.
And we're going to allow ourselves to
translate the pattern so it goes off the
right hand end, and comes back on the left
hand end.
So the third example is the same pattern
that's been translated with some
wrap-around.
And pattern B, it also has four patterns,
but four pixels, but in a different
arrangement.
And in the third example of pattern B,
it's been translated with wrap-around.
So that's still an example of pattern B.
And for two sets of patterns like that, a
binary threshold unit cannot learn to
discriminate them.
And here's the proof.
What we're going to do is we're going to
consider that for the positive examples we
have pattern A in all possible
translations.
Now since pattern A has four on pixels,
that means if we look at any pixel on the
retina, there'll be four different
positions in which we can put pattern A
that will activate that pixel.
So each pixel will be activated by four
different translations of pattern A.
That means that the total input received
by the decision unit, over all those
various translations of pattern A, would
be four times the sum of all the weights
in the perceptron, because each pixel will
activate the decision unit four different
times.
And so summed over all those patterns will
get four times the sum of the weights.
Now consider pattern B.
We're going to make the negative cases be
pattern B, in all possible translations.
And again, each pixel will be activated by
four different translations of pattern B.
So the total input of the decision unit
receives and, over all those different
translations of pattern B, will again be
four times the sum of all the weights.
But the perceptron, in order to
discriminate correctly, has to have
weights so that every single case of
pattern A provides more input to the
decision unit than every single case of
pattern B.
And that's clearly impossible if when you
sum of all these cases, all those
different versions of pattern A and all of
those different versions of pattern B,
provide exactly the same amount of input
to the decision unit.
So we've proved that a perceptron cannot
recognize patterns under translation if we
allow wrap-around.
That's a particular case of Minsky and
Papert's group invariance theorem.
And that result is devastating for
perceptrons, it was historically
devastating.
Because the whole point of pattern
recognition is to recognize patterns that
undergo transformations and see that
they're still the same pattern, despite
the transformation.
Like for example, translation.
And when Minsky and Papert showed that a
perceptron couldn't do that if the
transformations formed a group, that is
the learning part of a perceptron couldn't
learn to do that, it became clear that the
claims that have been made for what
perceptrons could learn were somewhat
exaggerated, and that to get them to do
anything interesting, you had to choose
just the right features to make it fairly
easy for the last stage to learn the
classification.
So the translations within our prime form
a group and, Minsky and Papert proved a
general theorem for transformations that
form a group, are making it impossible,
for a perceptron.
For the learning part of a perceptron to
do the recognition.
The perceptron architecture can still do
the recognition, but you have to organize
the features so they do the difficult
part.
So we have to have multiple feature units
that recognize informative sub patterns
that tell you something about what class
it is, and we have to have separate
feature units for each position of those
informative sub patterns, if we're trying
to recognize under translation.
So the tricky part of pattern recognition
has to be solved by the hand-coded feature
detectors, not the learning procedure.
The temporary conclusion from this is that
perceptrons are no good and therefore
neural networks are no good.
The longer term conclusion is that neural
networks are only gonna be really powerful
if we can learn the feature detectors.
It's not enough just to learn weight sum
feature detectors, we have to learn the
feature detectors themselves.
And the second generation of neural
networks, which we'll come to in the next
lecture, was all about how you learn the
feature detectors.
But it took twenty years before people
figured out how to do that.
So, networks without hidden units are very
limited in what they can learn to model.
If we add more layers of linear units, it
doesn't help because everything is linear.
We can make them much more powerful by
putting in hand coded hidden units but
they're not really hidden units because we
hand coded them.
We told them what to do.
It's not enough just to have fixed output
non-linearities.
What we need is multiple layers of
adaptive non-linear hidden units.
And the problem is how can we train such
nets.
We need a way to adapt all the weights not
just the last layer like in a perceptron,
and that's hard.
In particular, leaning the weights go in
to the hidden units, that's equivalent to
learning features.
And that's the hard thing to do.
Because nobody is telling us directly,
what the hidden unit should be doing, when
they should be active and, when they
should not be active.
And the, real problem is, how do we figure
out how to learn these weights go into
hidden units so that the hidden units turn
into the kinds of feature detectors we
need for solving a problem, when nobody is
telling us what the featured detector
should be.