In this video, we're going to look at the soft max output function.
This is a way of forcing the outputs of a neural network to sum to one so they can
represent a probability distribution across discreet mutually exclusive
alternatives. Before we get back to the issue of how we
learn feature vectors to represent words, we're gonna have one more digression, this
time it's a technical diversion. So far I talked about using a square area
measure for training a neural net and for linear neurons it's a sensible thing to
do. But the squared error measure has some
drawbacks. If for example the design acuities are
one, so you have a target of one, and the actual output of a neuron is one
billionth, then there's almost no gradient to allow a logistic unit to change.
It's way out on a plateau where the slope is almost exactly horizantal.
And so, it will take a very, very long time to change its weights, even though
it's making almost as big an error as it's possible to make.
Also, if we're trying to assign probabilities to mutually exclusive class
labels, we know that the output should sum to one.
Any answer in which we say, the probability this is A is three quarters
and the probability that it's a B is also three quarters is just a crazy answer.
And we ought to tell the network that information, we shouldn't deprive it of
the knowledge that these are mutually exclusive answers.
So the question is, is there a different cost function that will work better?
Is there a way of telling it that these are mutually exclusive and then using a,
an appropriate cost function? The answer, of course is, that there is.
What we need to do is force the outputs of the neural net to represent a probability
distribution across discrete alternatives, if that's what we plan to use them for.
The way we do this is by using something called a soft-max.
It's a kind of soft continuous version of the maximum function.
So the way the units in a soft-max group work is that they each receive some total
input they've accumulated from the layer below.
That's Zi for the i-th unit, and that's called the logit.
And then they give an output Yi that doesn't just depend on their own Zi.
It depends on the Zs accumulated by their rivals as well.
So we say that the output of the i-th neuron is E to the Zi divided by the sum
of that same quantity for all the different neurons in the softmax group.
And because the bottom line of that equation is the sum of the top line over
all possibilities, we know that when you add over all possibilities you'll get one.
That is, the sum of all the Yi's must come to one.
What's more, the Yi's have to lie between zero and one.
So we force the Yi to represent a probability distribution over mutually
exclusive alternatives just by using that soft max equation.
The soft max equation has a nice simple derivative.
If you ask about how the YI changes as you change the Zi, that obviously depends on,
all the other Zs. But then the Yi itself depends on all the
other Zs. And it turns out, that you get a nice
simple form, just like you do for the majestic unit, where the derivative of the
output with respect to the input, for an individual neuron in a softmax group, is
just Yi times one minus Yi. It's not totally trivial to derive that.
If you tried differentiating the equation above, you must remember that things turn
up in that normalization term on the bottom row.
It's very easy to forget those terms and get the wrong answer.
Now the question is, if we're using a soft max group for the outputs, what's the
right cost function? And the answer, as usual, is that the most
appropriate cost function is the negative log probability of the correct answer.
That is, we want to maximize the log probability of getting the answer right.
So if one of the target values is a one and the remaining ones are zero, then we
simply sum of all possible answers. We put zeros in front of all the wrong
answers. And we put one in front of the right
answer and that gets us the negative log probability of the correct answer, as you
can see in the equation. That's called the cross entropy cost
function. It has a nice property that it has a very
big gradient when the target value is one and the output is almost zero.
You can see that by considering a couple of cases.
So value of one in a million is much better than a value of one in a billion,
even though it differs by less than a millionth.
So when you make the output value, you increase by less than one millionth.
The value of C improves by a lot. That means it's a very, very steep
gradient for C. One way of seeing why a value of one in a
million is much better than a value of one in a billion, if the correct answer is one
is that if you believe the one in a million, you'd be willing to bet but odds
of one in a million, then you'd lose $one million.
If you thought the answer was one in a one billion you'd, you'd lose $one billion
making the same bet. So we get a nice property that.
That cost function, C has a very steep derivative when the answer is very wrong
and that exactly bounces the fact that the way which the advert changes is to change
the import, the Y or the Z is very flat when the once is very wrong.
And when you multiply the two together to get the derivative of cross entropy with
respect to the logic going into i put unit i.
You use the chain rule so that derivative is how fast the cost function changes as
you change the output of the unit times how fast the output of the unit changes as
you change Zi. And notice we need to add up across all
the Js, because when you change the i, the output of all the different units changes.
The result is just the actual output minus the target output.
And you can see that when the actual target outputs are very different, that
has a slope of one or -one. And the slope is never bigger than one or
-one. But the slope never gets small until the
two things are pretty much the same. In other words, you're getting pretty much
the right answer.