This video introduces the learning algorithm for a linear neuron.
This is quite like the learning algorithm for a perceptron, but it achieves
something different. In a perceptron, what's happening is the
weight, so always getting closer to a good set of weights.
In a linear neuron, the outputs are always getting closer to the target outputs.
The perception convergence procedure works by ensuring that when we change the
weights, we get closer to a good set of weights.
That type of guarantee cannot be extended to more complex networks.
Because in more complex networks when you average two good set of weights, you might
get a bad set of weights. So for multilayer neural networks, we
don't use the perceptron learning procedure.
And to prove that when they're learning something is improving, we don't use the
same kind of proof at all. They should never have been called
multilayer perceptrons. It's partly my fault and I'm sorry.
For multilayer nets we're gonna need a different way to show that the learning
procedure makes progress. Instead of showing that the weights get
closer to a good set of weights, we're gonna show that the actual output values
get closer to the target output values. This can be true even for non-convex
problems in which averaging the weights of two good solutions does not give you a
good solution. It's not true for perceptual learning.
In perceptual learning, the outputs as a whole can get further away from the target
outputs even though the weights are getting closer to good sets of weights.
The simplest example of learning in which you're making the outputs get closer to
the target outputs is learning in a linear neuron with a squared error measure.
Linear neurons, which are also called linear filters in electrical engineering,
have a real valued output that's simply the weighted sum of their outputs.
So the output Y, which is the neuron's estimate of the target value, is the sum
over all the inputs i of a weight vector times an input vector.
So we can write it in summation form or we can write it in vector notation.
The aim of the learning is to minimize the error summed over all training cases.
We need a measure of that error and to keep life simple, we use the square
difference between the target output and the actual output.
So one question is why don't we just solve it analytically.
It's straightforward to write down a set of equations with one equation per
training case, and to solve for the best set of weights.
That's the standard engineering approach, and so why don't we use it?
The first answer, and the scientific answer, is we'd like to understand what
real neurons might be doing, and they're probably not solving a set of equations
symbolically. An engineering answer is that we want a
method that we can then generalize to multilayer, nonlinear networks.
The analytic solution relies on it being linear and having a squared error measure.
An iterative method, which we're gonna see next, is usually less efficient, but much
easier to generalize to more complex systems.
So I'm now gonna go through a toy example that illustrates an iterative method for
finding the weights of a linear neuron. Suppose that every day, you get lunch at a
cafeteria. And your diet consists entirely of fish,
chips, and ketchup. Each day, you order several portions of
each, but on different days, it's different numbers of portions.
The cashier only shows you the total price of the meal, but after a few days, you
ought to be able to figure out what the price is for each portion of each kind of
thing. In the iterative approach, you start with
random guesses for the prices of portions. And then you adjust these guesses so that
you get a better fit to the prices that the cashier tells you.
Those are the observed prices of whole meals.
So each meal, you get a price and that gives you a linear constraint on the
prices of the individual portions. It looks like this, the price of the whole
meal is the number of portion of fish, x fish, times the cost of a portion of fish,
w fish. And the same for chips and ketchup.
So the prices of the portions are like the weights of a linear neuron.
And we can think of the whole weight vector as being the price of a portion of
fish, the price of a portion of chips, and the price of a portion of ketchup.
We're going to start with guesses for these prices and then we're going to
adjust the guesses slightly, so that we agree better with what the cashier says.
So let's suppose that the true weights that the cashier using to figure out the
price, are 150 for a portion of fish, 50 for portion of chips and a 100 for a
portion of Ketchup. For the meals shown here, that will lead
to a price of 850. So that's going to be our target value.
That suppose that we start with guesses, but each portion costs 50.
So for the meal with two portions of fish, five of chips, and three of ketchup, we're
going to initially think that the price should be 500.
That gives us a residual error of 350. The residual error is the difference
between what the cashier says and what we think the price should be with our current
weights. We're then gonna use the delta rule for
revising our prices of portions. We make the change in a weight, delta WI
be equal to a learning rate, epsilon times the number of portions of the i-th thing,
times the residual error. The difference between the target and our
estimate. So if we make the learning rate be one
over 35, so the maths stays simple, then the learning rate times the residual error
for this particular example is ten. And so, our change in the weight for fish
will be two times ten. We'll increase that weight by twenty.
Our change in the weight for chips will be five times ten.
And our change in the weight for ketchup will be three times ten.
That'll give us new weights of 70, 100, and 80.
And notice, the weight for chips actually got worse.
There's no guarantee with this kind of learning that the individual weights will
keep getting better. What's getting better is the difference
between what the cashier says and our estimate.
So now, we're going to derive the delta rule.
We start by defining the arrow measure, which is simply our squared residual
summed over all training cases. That is the squared difference between the
target and what the neural net predicts. Or the linear neuron predicts.
Squared, in some liberal training cases. And we put a one-half in front, which will
cancel the two, when we differentiate. We now differentiate that error measure
with respect to one of the weights, WI. To do that differentiation we need to use
the chain rule. The chain rule says that how the error
changes as we change a weight, will be how the output changes as we change the
weight, times how the error changes as we change the output.
The chain rule is easy to remember, you just cancel those two DYs but you can only
do that when there's no mathematicians looking.
The reason the first one, DY by DW is written with a curly D is because it's a
partial derivative. That is, there's many different weights
you can change to change the output. And here, we're just considering the
change to weight i. So, DY by DWi, is actually equal to Xi,
and that's because Y is just Wi times Xi, and DE by DY, is just T minus Y, because
when we differentiate that T minus Y squared, and use the half to cancel the
two we just get T minus Y. So our learning rule is now, we change the
weights by an amount that's equal to the learning rate epsilon times the derivative
of the error with respect to a weight, to E by DWi.
And with a minus sign in front cuz we want the error to go down.
And that minus sign cancels the minus sign in the line above and we get that.
The change in a weight is the sum of all training cases of the learning rate times
the input value times the difference between the target and actual outputs.
Now we can ask how does this learning procedure, this delta rule, behave?
Does this, for example, eventually get the right answer?
There may be no perfect answer. It may be that we give the linear neuron a
bunch of training cases with desired answers.
And there's no set of weights that'll give the desired answer.
There's still some set of weights that gets the best approximation on all those
training cases, minimizes that error measure.
Some that are all training cases. And if we make the learning rate small
enough and we learn for long enough, we can get as close as we like to that best
answer. Another question is, how quickly do we get
towards the best answer. And even for a linear system.
The learning can be quite slow in this kind of intricate learning.
If two input dimensions are highly correlated, its very hard to tell how much
of the sum of the weight on both input dimensions should be attributed to each
input dimension. So if for example, you always get the same
number of portions of ketchup and chips is, we can't decide how much of the price
is due to the ketchup and how much is used to the chips.
And if they're almost always the same, it can take a long time for the learning to
correctly attribute the price to the ketchup and the chips.
There's an interesting relationship between the delta rule and the learning
rule for perceptrons. So, if you, you use the online version of
the delta rule, but we change the weights after each training case, it's quite
similar to the perceptron learning rule. In perceptron learning, we increment or
decrement the weight vector by the input vector, but we only change the input
vector when we make an error. In the online version of the delta rule,
we increment or decrement the weight vector by the imperfector.
But we scale that by both the residual error and the learning rate.
And one annoying thing about this is we have to choose a learning rate.
If we choose a learning rate that's too big, the system will be unstable.
And if we choose a learning rate that's too small, it will take an unnecessarily
long time to, to learn a sensible set of weights