In this video, we're going to, look at a proof that the perceptron learning
procedure will eventually get the weights into the cone of feasible solutions.
I don't want you to get the wrong idea about the course from this video.
In general, it's gonna be about engineering, not about proofs of things.
There'll be very few proofs in the course. But we get to understand quite a lot more
about perceptrons when we try and prove that they will eventually get the right
answer. So we going to use our geometric
understanding of what's happening in weight space as subset from learns, to get
a proof that the perceptron will eventually find a weight vector if it gets
the right answer for all of the training cases, if any such vector exists.
And our proof is gonna assume that there is a vector that gets the right answer for
all training cases. We'll call that a feasible vector.
An example of a feasible vector is shown by the green dot in the diagram.
So we start with the wait vector that's getting some of the training cases wrong,
and in the diagram we've shown a training case that is getting wrong.
And what we want to show. This is the idea for the proof.
Is that, every time he gets a training case wrong, it will update the current
weight vector. In a way that makes it closer to every
feasible weight factor. So we can represent the squared distance
of the current weight factor from a feasible weight factor, as the sum of a
squared distance along the line of the input factor that defines the training
case, and another squared difference orthogonal to that line.
The orthogonal squared distance won't change, and the squared distance along the
line of the input factor will get smaller. So our hopeful claim is that, every time
the perceptor makes a mistake. Our current weight factor is going to get
closer to all feasible weight factors. Now this is almost right, but there's an
unfortunate problem. If you look at the feasible weight vector
in gold, it's just on the right side of the plane that defines one of the training
cases. And the current weight factor is just on
the wrong side, and the input vector is quite big.
So when we add the input factor to the count weight factor, we actually get
further away from that gold feasible weight factor.
So our hopeful claim doesn't work, but we can fix it up so that it does.
So what we're gonna do is we're gonna define a generously feasible weight
factor. That's a weight factor that not only gets
every training case right, but it gets it right by at least a certain margin.
Where the margin is as big as the input factor for that training case.
So we take the cone of feasible solutions, and inside that we have another cone of
generously feasible solutions. Which get everything right by at least the
size of the input vector. And now, our proof will work.
Now we can make the claim, that every time the perceptron makes a mistake.
The squared distance to all of the generously feasible weight vectors would
be decreased by at least the squared length of the input vector, which is the
update we make. So given that, we can get an informal
sketch of a proof of convergence. I'm not gonna try and make this formal.
I'm more interested in the engineering than the mathematics.
If your mathematician i'm sure you can make it formal yourself.
So, every time the preceptor makes a mistake, the current weight vector moves
and it decreases its squared distance from every feas, generously feasible weight
vector, by at least the squared length of the current input vector.
And so the squared distance to all the generously feasible weight vectors
decreases by at least that squared length and assuming that none of the input
vectors are infinitesimally small. That means that after a finite number of
mistakes the weight vector must lie in the feasible region if this region exists.
Notice it doesn't have to lie in the generously feasible region, but it has to
get into the feasible region to make, to stop it making mistakes.
And that's it. That's our informal sketch of a proof that
the perceptron convergence procedure works.
But notice, it all depends on the assumption that there is a generously
feasible weight vector. And if there is no such vector, the whole
proof falls apart.