In this figure, we're going to get a
geometrical understanding of what happens
when a perceptron learns.
To do this, we have to think in terms of a
weight space.
It's a high dimensional space in which
each point corresponds to a particular
setting for all the weights.
In this phase, we can represent the
training cases as planes and learning
consists of trying to get the weight
vector on the right side of all the
training planes.
For non-mathematicians, this may be
tougher than previous material.
You may have to spend quite a long time
studying the next two parts.
In particular, if you're not used to
thinking about hyperplanes and high
dimensional spaces, you're going to have
to learn that.
To deal with hyperplanes in a
14-dimensional space, for example, what
you do is you visualize a 3-dimensional
space and you say, fourteen to yourself
very loudly.
Everybody does it.
But remember, that when you go from
13-dimensional space to a 14-dimensional
space, your creating as much extra
complexity as when you go from a 2D space
to a 3D space.
14-dimensional space is very big and very
complicated.
So, we are going to start off by thinking
about weight space.
This is the space that has one dimension
for each weight in the perceptron.
A point in the space represents a
particular setting of all the weights.
Assuming we've eliminated the threshold,
we can represent every training case as a
hyperplane through the origin in weight
space.
So, points in the space correspond to
weight vectors and training cases
correspond to planes.
And, for a particular training case, the
weights must lie on one side of that
hyperplane, in order to get the answer
correct for that training case.
So, let's look at a picture of it so we
can understand what's going on.
Here's a picture of white space.
The training case, we're going to think of
one training case for now, it defines a
plane, which in this 2D picture is just
the black line.
The plane goes through the origin and it's
perpendicular to the input vector for that
training case, which here is shown as a
blue vector.
We're going to consider a training case in
which the correct answer is one.
And for that kind of training case, the
weight vector needs to be on the correct
side of the hyperplane in order to get the
answer right.
It needs to be on the same side of the
hyperplane as the direction in which the
training vector points.
For any weight vector like the green one,
that's on that side of the hyperplane, the
angle with the input vector will be less
than 90 degrees.
So, the scaler product of the input vector
with a weight vector will be positive.
And since we already got rid of the
threshold, that means the perceptron will
give an output of what?
It'll say yes, and so we'll get the right
answer.
Conversely, if we have a weight vector
like the red one, that's on the wrong side
of the plane, the angle with the input
vector will be more than 90 degrees, so
the scalar product of the weight vector
and the input vector will be negative, and
we'll get a scalar product that is less
than zero so the perceptron will say, no
or zero, and in this case, we'll get the
wrong answer.
So, to summarize, on one side of the
plane, all the weight vectors will get the
right answer.
And on the other side of the plane, all
the possible weight vectors will get the
wrong answer.
Now, let's look at a different training
case, in which the correct answers are
zero.
So here, we have the weight space again.
We've chosen a different input vector, of
this input factor, the right answer is
zero.
So again, the input case corresponds to a
plane shown by the black line.
And in this case, any weight vectors will
make an angle of less than 90 degrees with
the input factor, will give us a positive
scalar product, [unknown] perceptron to
say yes or one, and it will get the answer
wrong conversely.
And the input vector on the other side of
the plain, will have an angle of greater
than 90 degrees.
And they will correctly give the answer of
zero.
So, as before, the plane goes through the
origin, it's perpendicular to the input
vector, and on one side of the plane, all
the weight vectors are bad, and on the
other side, they're all good.
Now, let's put those two training cases
together in one picture weight space.
Our picture of weight space is getting a
little bit crowded.
I've moved the input vector over so we
don't have all the vectors in quite the
same place.
And now, you can see that there's a code
of possible weight vectors.
And any weight vectors inside that cone,
will get the right answer for both
training cases.
Of course, there doesn't have to be any
cone like that.
It could be there are no weight vectors
that get the right answers for all of the
training cases.
But if there are any, they'll lie in a
cone.
So, what the learning algorithm needs to
do is consider the training cases one at a
time and move the weight vector around in
such a way that it eventually lies in this
cone.
One thing to notice is that if you get a
good weight factor, that is something that
works for all the training cases, it'll
lie on the cone.
Ad if you had another one, it'll lie on
the cone.
And so, if you take the average of those
two weight vectors, that will also lie on
the cone.
That means the problem is convex.
The average of two solutions is itself a
solution.
And in general in machine learning if you
can get a convex learning problem, that
makes life easy.