In this video, I'm going to return to the idea of full Baysian learning, and explain
a little bit more about how it works. And then in the following video, I'm going
to show how it can be made practical. In full Bayesian learning, we don't try
and find a single best setting of the parameters.
Instead, we try and find the full posterior distribution over all possible
settings. That is, for every possible setting, we
want a posterior probability density. And all those densities, we want to add up
to one. It's extremely computationally intensive
to compute this for all, but the simplest models.
So, in the example earlier, we did it for a biased coin which just has one
parameter, which is how biased it is. But in general, for a neural net, it's
impossible. After we've computed the posterior
distribution across all possible settings of the parameters, we can then make
predictions by letting each different setting of the parameters make its own
prediction. And then, averaging all those predictions
together, weighting by their posterior probability.
This is also very computationally intensive.
The advantage of doing this is that if we use the full Bayesian approach, we can use
complicated models even when we don't have much data.
So, there's a very interesting philosophical point here.
We're now used to the idea of overfitting, When you fit a complicated model to a
small amount of data. But that's basically just a result of not
bothering to get the full posterior distribution over the parameters.
So, frequentists would say, if you don't have much data, you should use a simple
model. And that's true.
But it's only true if you assume that fitting a model means finding the single
best setting of the parameters. If you find the full posterior
distribution, that gets rid of overfitting.
If there's very little data, the full posterior distribution will typically give
you very vague predictions, because many different settings of the parameters that
make very different predictions will have significant posterior probability.
As you get more data, the posterior probability will get more and more focused
on a few settings of the parameters, and the posterior predictions will get much
sharper. So, here's a classic example of
overfitting. We've got six data points and we fitted a fifth order polynomial and so
it should go exactly through the data, which it more or less does.
We also featured a straight line which only has two degrees of freedom.
And so, which model do you believe? The model that has six coefficients and
fits the data almost perfectly, or the model that only has two coefficients and
doesn't fit the data all that well. It's obvious that the complicated model
fits better, but you don't believe it. It's not economical, and it also makes
silly predictions. So, if you look at the blue arrow,
If that's the input value and you're trying to predict the output value, the
red curve will predict a value that's lower than any of the observed data
points, which seems crazy, whereas the green line will predict a sense of the
value. But everything changes, if instead of
fitting one fifth order polynomial, we start with a reasonable prior of the fifth
order polynomials, for example, the coefficient shouldn't be to big.
And then, we compute the full posterior distribution over fifth order polynomials.
And I've shown you a sample from this distribution in the picture, where a
thickened line means higher probability in the posterior.
So, you will see some of those thin curves, miss a few of the data points by
quite a lot, but nevertheless, they're quite close to most of the data points.
Now, we get much vaguer, but much more sensible predictions.
So, where the blue arrow is, you'll see the different models predict very
different things. While, on average, they make a prediction
quite close to the prediction made by the green line.
From a Bayesian prospective, there's no reason why the amount of data you collect
should influence your prior beliefs and the complexity of the model.
A true Baysian would say, you have prior beliefs about how complicated things might
be and just because you haven't collected any data yet, it doesn't mean you think
things are much simpler. So, we can approximate full Baysian
learning in a neural net, if the neural net has very few parameters.
The idea is we put a grid over the parameter space,
So each parameter is only allowed a few return to values and then we take the
cross product of all those values for all the parameters.
And now, we get a number of grid points in the parameter space.
And in each of those points, we can see how well our model predicts the data, that
is, if we're doing supervised learning, how well a model predicts the target
outputs. And we can say that the posterior
probability in that grid-point is the product of how well it predicts the data,
how likely it is under the prior. And with the whole thing normalized, so
that the posterior probability is [UNKNOWN].
This is still very expensive, but notice it has some attractive features.
There's no gradient descent involved, and there's no local optimum issues.
We're not following a path in this space, We're just evaluating a set of points in
this space. Once we've decided on the posterior
probability to assign to each grid-point, We then use them all to make predictions
on the test data. That's also expensive.
But when there isn't much data, it'll work much better than maximum likelihood or
maximum a posteriori. So, the way we predict the test output,
given the test input, is we say, the probability of the test output, given the
test input, Is the sum overall the grid points of the
probability that, that grid-point is a good model, is the sum over all
grid-points of the probability of that grid-point, given the data and given our
prior, times the probability that we will get that test output,
Given the input and given the grid-point. In other words, we have to take into
account, the fact that we might add noise to the output of the net before producing
the test answer. So, here's a picture of full Bayesian
learning. We have a little net here, that has four
weights and two biases. If we allowed, nine possible values for
each of those weights and biases, There would be nine to the six grid+points
in the parameter space. It's a big number but we can cope with it.
For each of those grid-points, we compute the probability of the observed outputs on
all the training cases. We multiply by the prior for the
grid-point, which might depend on the values of the weights, for example.
And then, we re-normalize to get the posterior probability over all the
grid-points. Then we make predictions using those
grid-points, but weight to each of their predictions by its posterior probability.