[MUSIC] So, instead of using training error
to assess our predictive performance. What we'd really like to do is analyze
something that's called generalization or true error. So, in particular, we really want
an estimate of what the loss is averaged over all houses that we
might ever see in our neighborhood. But really, in our dataset we only have
a few examples of houses that were sold. But there are lots of other houses that
are in our neighborhood that we don't have in our dataset, or other houses that
you might imagine having been sold. Okay, so to compute this estimate over all
houses that we might see in our dataset, we'd like to weight these house pairs,
so the pair of house attributes and the house sale's price. By how likely that pair is to
have occurred in our dataset. So to do this we can think about
defining a distribution and in this case over square feet
of houses in our neighborhood. So what this little
cartoon is trying to show is a distribution over
the real line of square feet. But you can think of it as just
a really dense, in a sense histogram, counting how many houses that we might
see with a given square feet for every possible square feet value. Okay and so what this picture is
showing is a distribution that says we're very unlikely to see
houses with very small or low number of square feet,
very small houses. And we're also very unlikely to
see really, really massive houses. So there's some bell curve to this,
there's some sweet spot of kind of typical houses in our neighborhood, and
then the likelihood drops off from there. Likewise what we can do is define
a distribution that says for a given square footage of a house, what's the distribution over
the sales price of that house? So let's say the house
has 2,640 square feet. Maybe I expect the range of
house prices to be somewhere between $680,000 to maybe $950,000. That might be a typical range. But of course, you might see much
lower valued houses or higher value, depending on the quality of that house. And that's what this distribution
here is representing. Okay, so formally when we go to
define our generalization error, we're saying that we're taking
the average value of our loss weighted by how likely those
pairs were in our dataset. So specifically we estimate our model
parameters on our training data set so that's what gives us w hat. That defines the model we're using for
prediction, and then we have our loss function,
assessing the cost of predicting f, this f sub w hat at our square
foot x when the true value was y. And then what we're gonna do is we're
gonna average over all possible xy's. But weighted by how likely they
are according to those distributions over square feet and value given square feet. Okay, so let's go back to these plots of
looking at error verses model complexity. But in this case let's quantify
our generalization error as a function of this complexity. And to do this, what I'm showing
by this crazy blue region here. And, it has different gradation
going from white to darker blue, is the distribution of houses that
I'm likely to see in my dataset. So, this white region here,
are the houses and now we just made it not white,
but hopefully we still see. These are the houses that I'm very,
very likely to see, and then as I go further away from
this I get to less likely house sale prices given
a specific square foot value. And so what I'm gonna do when I look
at thinking about generalization error is I'm gonna take my fitted
function where remember this green line was fit on the training data
which are these blue circles. And then I'm gonna say, how well does it
predict houses in this shaded blue region, weighted by how likely they are,
how close to that white region. If you imagine in 3D,
there are these distributions popping up off of this shaded grey and
shaded blue area. Maybe I can try and draw it. Maybe the distribution
at a given square foot, okay that doesn't look good at all,
let me try and do it again. Then it looks something like this,
the houses with xt square feet. And so when I think about how well my
prediction is doing at xt, this x here, I'm looking at the difference between
this and all points along this line. Weighted by how likely they are in the
general population of houses I might see. And then I do that across this entire
region of possible square feet. Okay, so
what I see here is this constant model who really doesn't approximate things well
except maybe in this region here. So overall it has a reasonably
high generalization error and I can go to my more complex,
just fitting a line through the data. And I see I have better performance, but
still not doing great in these regions. So my generalization error dropped a bit,
but when I get to this higher complexity quadratic fit things are starting to look
a bit better, maybe not great out in these regions here, so again,
the generalization error drops. Then I get to this much
higher order polynomial, and when we were looking at training error,
the training error was lower, right? But now, when we think about
generalization error, we actually see that the generalization error is gonna go
up relative to the simpler model, because if we look at this region here,
it's doing really horribly. So, we might get a generalization error
that's actually larger than the quadratic, and then we can fit even a higher order
polynomial, and we get this really, really crazy fit. And it's doing horrible basically
everywhere, except maybe at these very, very small little regions
where it's doing okay. So in this case we get dramatically
bad generalization there. Okay, so this is starting to
match a lot more of our intuition behind what might be
a good fit to this data. So, let's think about just
drawing the curve over all possible models now that we've
fit these few specific points. So our generalization error in general will have some
shape where it's going down. And then we get to a point where
the error starts increasing. Sorry, that should have
been a smoother curve. The error starts increasing
because we're getting to these overly complex models that fit
the training data really well but don't generalize to other
houses that we might see. But importantly, in contrast to training error we can't
actually compute generalization error. Because everything was relative
to this true distribution, the true way in which the world works. How likely houses are to appear in our
dataset over all possible square feet and all possible house values. And of course, we don't know what that is. So, this is our ideal picture or
our cartoon of what would happen. But we can't actually go along and
compute these different points. [MUSIC]