In this video, I'm going to describe a
method developed by David MacKay in the
1990s for determining the weight penalties
to use in a neural network without using a
validation set.
It's based on the idea that we can
interpret weight penalties as doing map
estimation so that the magnitude of the
weight penalty is related to the tightness
of the prior distribution over the
weights.
Mackay showed we can empirically fit both
the weight penalties and the assumed noise
in the output of the neural net to get a
method for fitting weight penalties that
does not require a validation set and
therefore, allows us to have different
weight penalties for different subsets as
the connections in a neural network,
something that will be very expensive to
do using validation sets.
Mackay went on to win competitions using
this kind of method.
I'm now going to describe a simple and
practical method developed by David MacKay
for making use of the fact that we can
interpret weight penalties as the ratio of
two variances.
After we've learned a model to minimize
squared error we can find the best value
for the output variance and the best value
is found by simply using the variance of
the residual errors.
We can also estimate the variance in the
Gaussian prior of the weight.
We have to start with some guess about
what this variance should be.
Then, we do some learning, and then we use
a very dirty trick called empirical Bayes.
We set the variance of our prior to be the
variance of the weights of the model
learned because that's the variance that
will make those weights most likely.
This really violates a lot of the
presuppositions of the Bayesian approach.
We're using the data to decide what our
prior beliefs are.
So, once we've learned the weights, we fit
a zero mean Gaussian to the
one-dimensional distribution of the
learned weights.
And then, we take the variance of that
Gaussian, and we use that for our prior.
Now, one nice thing about that is, is the
different subsets of weights.
Like in different layers, for example, we
could learn different variances for the
different layers.
We don't need a validation set so we can
use all of the non-test data for training.
And because we don't need validation sets
to determine the weight penalties in
different layers, we can actually have
many different weight penalties.
This will be very hard to do with
validation sets.
So, here's MacKay's method.
You start by guessing the noise variance
and the weight prior variance.
Actually, all you have to really do is
guess their ratio.
Then, you do some gradient descent
learning, trying to improve the weights.
Then, you reset the noise variance to be
the variance of the residual errors and
you reset the weight prior variance to be
the distribution of the actually learned
weight.
And then, you go back around this loop
again.
So, this actually works quite well in
practice.
And MacKay won several competitions this
way.