To extend the learning rule for a linear
neuron to a learning rule we can use for
multilayer nets of nonlinear neurons, we
need two steps.
First, we need to extend the learning rule
to a single nonlinear neuron.
And we're going to use logistic neurons,
although many other kinds of nonlinear
neurons could be used instead.
We're now going to generalize the learning
rule for a linear neuron to a logistic
neuron, which is a non linear neuron.
So, a logistic neuron, computes its logic,
z, which is its total input, its, its bias
plus the sum over all its input lines of
the value of, on an input line xi times
the weight on that line, wi.
It then gives an output y that's a smooth
nonlinear function of that logit.
As shown in the graph here, that function
is approximately zero when z is big and
negative, approximately one when z is big
and positive, and in bet, in between, it
changes smoothly and nonlinearly.
The fact that it changes continuously
gives it nice derivatives, which make
learning easy.
So to get the derivatives of a logistic
neuron with respect to the weight, which
is what we need for learning, we first
need to compute the derivative of the
logit itself, that is the total input with
respect to our weight, that's very simple.
The logit is just a bias plus the sum of
all the input lines of the failure on the
input lines times the weight.
So, when we differentiate with respect to
wi, we just get xi.
So, the derivative of the logit with
respect to wi is xi, and similarly, the
derivative of the logit with respect to xi
is wi.
The derivative of the output with respect
to the logic is also simple if you express
it terms of the output.
So, the output is one / one + e^-z. And dy
by dz is just y into one - y. That's not
obvious.
For those of you who like to see the math,
I've put it on the next slide.
The math is tedious but perfectly
straightforward so you can go through it
by yourself.
Now, we've got the derivative, the output
with respect to the logic and the
derivative, the logit with respect to the
weight, we can start to figure out the
derivative, the output with respect to the
weight.
We just use the chain rule again.
So, dy by dw is dz by dw times dy by dz.
And dz by dw, as we just saw, is xi, dy by
dz is y into one minus y.
And so, we now have the learning row for a
logistic neuron.
We've got dy by dw, and all we need to do
is use the chain rule once more, and
multiply it by de by dy.
And we get something that looks very like
the delta rule.
So, the way the arrow changes is we change
the weight, de by dwi, is just the sum of
all the row of training cases and of the
value on input line xin times the
residual, the difference between the
target and the output, on the actual
output of the neuron.
But it's got this extra term in it, which
comes from the slope of the logistic
function, which is yn into one - yn.
So, a slight modification of the delta
rule gives us the gradiant decent learning
rule for training a logistic unit.