In this video, I'm going to talk about the
Bayesian interpretation of weight
penalties.
In the full Bayesian approach, we try to
compute the posterior probability of every
possible setting of the parameters of a
model.
But there's a much reduced form of the
Bayesian approach, where we simply say,
I'm going to look for the single set of
parameters that is the best compromise
between fitting my prior beliefs about
what the parameters should be like, and
fitting the data I've observed.
This is called Maximum alpha Posteriori
learning and it gives us a nice
explanation of what's really going on,
when we use weight decay to control the
capacity of a model.
I'm now going to talk a bit about what's
really going on, when we minimize the
squared error during supervised maximum
likelihood learning.
Finding the weight vector that minimizes
the squared residuals, that is the
differences between the target value and
the value predicted by the net,
Is equivalent to finding a weight vector
that maximizes the log probability density
of the correct answer.
In order to see this equivalence, we have
to assume that the correct answer is
produced by adding Gaussian noise to the
output of the neural net.
So the idea is, we make a prediction by
first running the neural net on the input
to get the output.
And then adding some Gaussain noise.
And then we ask, what's the probability
that when we do that, we get the correct
answer?
So the model's output is the center of a
Gaussian and what we're interested in is
having the target value of high
probability under that Gaussian because
the probability producing the value t,
given that the network gives an output of
y is just the probability density of t
under a Gaussian centered at y.
So the math looks like this: let's suppose
that the output of the neural net on
training case c is yc and this output is
produced by applying the weights W to the
input c. The probability that we'll get
the correct target value when we add
Gaussian noise to that output yc is given
by a Gaussian centered on yc.
So we're interested in the probability
density of the target value, under a
Gaussian centered at the output of the
neural net.
And on the right here, we have that
Gaussian distribution, with mean yc.
We also have to assume some variance, and
that variance will be important later.
If we now take logs and put in a minus
sign,
We see that the negative log probability
density of the target value tc given that
the network outputs yc, is a constant that
comes from the normalizing term of the
Gaussian plus the log of that exponential
with the minus sign in which is just (tc2
- yc)^2 divided by twice the variance of
the Gaussian.
So what you see is that, if our cost
function is the negative log probability
of getting the right answer, that turns
into minimizing a squared distance.
It's helpful to know that whenever you see
a squared error being minimized, you can
make a probabilistic interpretation of
what's going on, and in that probabilistic
interpretation, you'll be maximizing the
log probability under a Gausian.
So the proper Bayesian approach, is to
find the full posterior distribution over
all possible weight vectors.
If there's more than a handful of weights,
that's hopelessly difficult when you have
a non-linear net.
Bayesians have a lot of ways of
approximating this distribution, often
using Monte Carlo methods.
But for the time being, let's try and do
something simpler.
Let's just try to find the most probable
weight vector.
So the single setting of the weights
that's most probable given the prior
knowledge we have and given the data.
So what we're going to try and do is find
an optimal value of W by starting with
some random weight vector, and then
adjusting it in the direction that
improves the probability of that weight
factor given the data.
It will only be a local optimum.
Now, it's going to be easier to work in
the log domain than in the probability
domain.
So if we want to minimize a cost, we
better use negative log props.
Just an aside about why we maximize sums
of log probabilities, or minimize sums of
negative log probs,
What we really want to do is maximize the
probability of the data, which is
maximizing the product of the
probabilities of producing all the target
values that we observed on all the
different training cases.
If we assume that the output errors on
different cases are independent, we can
write that down as the product over all
the training cases, of the probability of
producing the target value, tc, given the
weights.
That is the product of the probability of
producing tc, given the output that we're
going to get from out network, if we give
it input c and it has weights W.
The log functions monotonic, and so it
can't change where the maxima are.
So instead of maximizing a product of
probabilities, we can maximize the sums of
log probabilities, and that typically
works much better on a computer.
It's much more stable.
So we maximize the log probability of the
data given the weights, which is simply
maximizing the sum over all training cases
of the log probability of the output for
that training case, given the input and
given the weights.
In Maximum a Posteriori learning, we're
trying to find the set of weights that
optimizes the trade off between fitting
our prior and fitting the data.
So that's base theorem.
If we take negative logs to get a cost, we
get that the negative log of the
probability of the weights given the data,
is the negative log of the prior term, and
the negative log of the data term, and an
extra term.
So, that last extra term, is an integral
overall possible weight vectors.
And so that doesn't affect W.
So we can ignore it when we're optimizing
W.
The term that depends on the data is the
negative log probability is given W, and
that's our normal error term.
And then term that only depends on W is
the negative log probability of W under
its prior.
Maximizing the log probability of a weight
is related to minimizing a squared
distance, in just the same way as
maximizing the log probability of
producing correct target value is related
to minimizing the square distance.
So minimizing the squared weights is
equivalent to maximizing the log
probability of the weights under a
zero-mean Gaussian prior.
So here's a Gaussian.
It's got a mean zero, and we want to
maximize the probability of the weights,
or the log probability of the weights.
And to do that, we obviously want w to be
close to the mean zero.
The equation for the Gaussian is just like
this, where the mean is zero so we don't
have to put it in.
And the log probability of w is then the
squared weights scaled by twice the
variance, plus a constant that comes from
the normalizing term of the Gaussian.
And isn't affected when we change w.
So finally we can get to the basing
interpretation of weight decay or weight
penalties.
We're trying to minimize the negative log
probability of the weights given in the
data and that involves minimizing a term
that depends on by the turn the weights
namely how will we shift the targets and
determine that depends only on the
weights.
Is derived from the log probability of the
data given the weights, which if we assume
Gaussian noise is added to the output of
the model to make the prediction, then
that log probability is the squared
distance between the output of the net on
the target value scaled by twice the
variance of that Gaussian noise.
Similarly, if we assume we have a Gaussian
prior for the weights, the log probability
of a weight under the prior is the squared
value of that weight scaled by twice the
variance of the Gaussian prior.
So now let's take that equation and
multiply through by two sigma squared D.
So, we got a new cost function. And the
first term, when we multiply through turns
into simply the sum of all training cases
of the squared difference between the
output of the net and the target.
That's the squared error that we typically
minimize in the neural net.
The second term now becomes, the ratio of
two variances times the sum of the squares
of the weights.
And so what you see is, the ratio of those
two variances is exactly the weight
penalty.
So we initially thought of weight
penalties as just a number you make up to
try and make things work better.
Where you fix the value of the weight
penalty by using a validation set.
But now we see that if we make this
Gaussian interpretation where we have a
Gaussian prior and we have a Gaussian
model of the relation of the output of the
net to the target,
Then the weight penalty is determined by
the variances of those Gaussians.
It's just the ratio of those variances.
It's not an arbitrary thing at all within
this framework.