In this video, I'm gonna describe the
Bayesian Approach to fitting models, using
a simple coin tossing example.
If you're already know about the Bayesian
Approach, you can skip this video.
The main idea behind the Bayesian Approach
is instead of looking for the most likely
setting of the parameters of the model, we
should consider all possible settings of
the parameters and try to figure out for
each of those possible settings, how
probable it is, given the data we
observed.
The Bayesian framework assumes that we
always have a prior distribution for
everything.
That is, for any event that you might care
to mention, I have to have some prior
probability that, that event might happen.
The problem might be very vague.
So what's happening is, our data gives us
a likelihood term.
We combine it with our prior and then we
get a posterior.
The likelihood term favors settings of our
parameters that make the data more likely.
It can disagree with the prior.
And in the limit, if we get enough data,
however unlikely the prior is, the data
can overwhelm it.
And in the end, with enough data, the
truth will out.
That is, even if your prior's wrong,
you'll end up with the right hypothesis.
But that may take an awful lot of data if
you thought that things were very unlikely
under your prior.
So let's start with a coin tossing
example.
Suppose you don't know anything about
coins except that they can be tossed and
when you toss a coin you get either a head
or a tail.
And we're also going to assume you know
that each time you do that it's an
independent decision.
So our model of a coin is going to have
one parameter P.
This parameter P determines the
probability that the coin will produce a
head.
What happens now if we see 100 tosses and
there are 53 heads.
What is a good value for P.
Well obviously you're tempted to say.53.
But what's the justification for that?.
The frequentest answer, which is also
called maximum likelihood, is to pick the
value of p that makes the observations
most probable.
And that value of p is.53.
It's not obvious that's true, let's derive
that.
So the probability of a particular
sequence that contains 53 heads and 47
tails could be written out by writing down
p every time you toss a head.
And 1-p every time you toss a tail.
And then if we collect all the p's
together, and all the 1-p's together, we
get p^53, and 1-p^47.
If we now ask, how does the probability of
observing that data depend on p, we can
differentiate with respect to p, and we
get the expression shown here, and if we
then set that derivative to zero.
We discover that the probability of the
data is maximized by setting P to be.53.
So that's maximum likelihood.
But there's some problems with using
maximum likelihood to decide on the
parameters of a model.
Suppose for example, we only toss the coin
once and we got one head.
It doesn't really make sense to say we
think the probability of the coin coming
down heads in future is one.
That would mean we'd be willing to bet
that infinite odds that it can't come down
tails.
And that seems ridiculous.
It's sort of intuitively obvious that a
much better answer is not.5.
But how can we justify that.
More importantly, we can ask, is it
reasonable to give a single answer.
We don't know much.
We don't have much data, and so we're
unsure about what the value of P is.
So what we really ought to do is refuse to
give a single answer and instead give a
whole probability distribution across
possible answers.
An answer like 0.5 is fairly likely.
An answer like one is maybe still pretty
unlikely if we have some prior belief that
coins come down heads half the time.
So, now I'm going to go through an example
where we start with some prior
distribution over parameter values, and
we'll pick a prior distribution that's
easy to work with.
Not one that necessarily fits what we
really believe about commons.
And then we'll show how that prior
distribution get modified by data if we
adopt the Bayesian Approach.
So, we're gonna start with a prior
distribution that says all the different
values of p are equally likely.
We believe that coins come biased to
various extents.
And any amount of bias is equally likely.
So that some coins come down heads half
the time.
Other coins come downs heads all the time.
And those two kinds of coins are equally
likely.
We now observe a coin coming down heads.
So what we do now, is for each possible
value of p, we take its prior probability
and we multiply by the probability that we
would have observed ahead, given that,
that value of p is the correct one.
So, for example, if we take the value of P
= one which says coins come down heads
every time then the probability of
observing a head would be one.
There would be no alternative.
And if we take the value of P to be zero
the probability of observing a head would
be zero.
And if we take it to b.5, the probability
of observing having b is.5.
So we take that red line, that's our
prior, and we multiply each point on that
by the probability of observing a head
according to that hypothesis.
And now we get the sloping-like, that's a
unnormalized posterior.
It's unnormalized because the area under
that line doesn't add up to one.
And of course for a probability
distribution, the probabilities of all the
alternative events have to add to one.
So the last thing we do is re-normalize
it.
We scale everything up so the area under
the curve is one.
And now if we started with the uniform pri
distribution of the P we end up with this
triangular posterior distribution of the P
having observed one head.
Now let's do it again.
And this time let's suppose we get a tail.
So, the prior distribution that we start
with now is the post-serial distribution
we had after observing our one head.
And now, the green line shows the
probability that we will get a tail
according to each of those hypotheses that
correspond to a value of P.
So, for example, if P is one, the
probability we would observe at time is
zero.
So we have to multiply our prior by our
likelihood term.
And now we get a curve like that.
Then we have to re-normalize to make the
area be one.
And that's now a posterior distribution,
after having observed one head and one
tail.
I notice it's a pretty sensible
distribution.
After observing one of each, we know that
P can't be either zero or one, and it also
seems very sensible that the most likely
thing is now in the middle.
If we do this again another 98 times, and
keep applying the same strategy of
multiplying the posterior we had after the
last task, by the likelihood of observing
that event, given the various different
settings of the parameter p.
And let's suppose we get 53 heads and 47
tails in all.
Then we'll end up with a curve that looks
like this.
It'll have its mean at 53..
Because we started with the uniform prior.
And it'll be fairly sharply peaked by 53..
But it'll allow other values like 49. is a
perfectly reasonable value under this
curve.
Not quite as lengthy as 53., but very
reasonable.
Whereas a value of 25. is extremely
unlikely under this curve.
So we can summarize all that with base
therm.
Determine the middle of this equation is
the joint probability of a set of
parameters W, and some data D.
And for supervised learning, the data is
gonna consist of the target values.
So we assume we are given inputs and the
data values consist of the target values
that are associated with those iinputs.
That what we observe.
That joint probability can be re-written
as the product of an individual
probability and a conditional probability.
So on the right, we're written it as p of
w times p of d given w, and on the left,
I've written it as p of d times p of w
given d.
Now we can divide both sides by p of d.
And this gives us base there, I mean it's
usual form.
Base theorem says that the posterior
probability of a particular value of W,
given the data D, is just the prior
probability of that particular value of W
times the probability given that
particular value of W, that you would have
observed the data you observed.
And that has to be normalized by P of D.
The probability of the data which is
simply the integral over all possible
values of W, of P of W, P of D, given W.
The bottom line needs to be the sum of the
top line a row of possible values w in
order for this to be a probability
distribution that adds to one.
Because that p of d has integrated over
all possible values of w, it's not
affected by picking a particular value of
w on the left-hand side.
So when we're looking for the best value
of w, for example, we can ignore p of d.
It doesn't depend on w.
The other two terms on the right-hand
side, however, do depend on w.