[MUSIC] Okay, so the issue we're facing here with
this crazy 13th order polynomial fit is something called overfitting. So in particular, what we've done
is we've taken a model and really, really, really honed in to
our actual observations, but it doesn't generalize well to
thinking about new predictions. And so the issues actually go beyond
just making really crazy predictions. And we're gonna discuss this in a lot
more detail in the regression course. But I wanna mention that this is a real
problem with any machine learning model or statistical model that you might consider. So in these cases,
we wanna fit a model to data, but we don't want that model to be so
specified exactly to the one data set that we have that it doesn't generalize
well to new observations we might get. Okay, so let's go back to this
13th order degree polynomial fit. And a question is,
do we actually believe this? Do we believe that this might
be a reasonable fit to the data? And I think as I alluded to before,
probably not. So although it minimizes
the residual sum of squares, it ends up leading to
very bad predictions. Because I'm sitting here and thinking,
well, this quadratic fit that we had, even though it didn't minimize my residual
sum of squares as much as that 13th order polynomial, it still, my gut feeling
is, somehow this is a better model. Okay, so a question is,
what's going on here, and how do we think about choosing the right
model order or model complexity? Well, what we want,
is we want good predictions. Of course, that's what we're aiming for. But we can't actually observe the future. Right, so we can't actually observe
that prediction that we want to make and say did we do a good job or
not until we actually go ahead and do it. So when we're thinking
about choosing our model, somehow we have to work just
with the data that we have. So how can we think about trying to
choose a good model in this case? Well, what we can do is we can
think about simulating predictions. So we're gonna take our
data set that we have, and we're gonna remove some of the houses. So those are the grayed-out houses here. These are gonna be removed temporarily. And we're gonna fit our model
on the remaining houses. So all of these guys we're gonna
use to fit our model using exactly the kind of methods
that we talked about before. And then what we're gonna
do is we're gonna predict. So I'll go through an erase these
x's now and put question marks. And say from the model that I just
learned on the circled houses, what values do I predict for
these question marks? And then I can compare to
the actual observed values, because these houses are in my data set. Okay, so I can use this as a proxy for
doing the types of real predictions that I wanna do on
data that I haven't yet collected. Of course, this type of method only
is gonna work well if I have enough observations to think about fitting
on versus testing my predictions on. Okay, so let's introduce
a little bit of terminology. Well, the houses that we use to fit our
model, we call that the training set. And the houses that we're
using as a proxy for our predictions, those that we're
holding out, we call the test set. Okay, so let's dig a little bit more
into how we're gonna do this analysis. And the first thing that we can do is look
at something called the training error. So we're gonna examine every
house in our test data set. So let's look at this red color here. So all of our training
houses are represented with these blue circles here,
and these are the only houses we're gonna look at when we're thinking
about defining our training error. So in particular, we're gonna look at what are the errors
that we make on these houses? So this is just the residual sum of
squares on the houses in our training data set, and
that's called the training error. So in particular, the training error
looks exactly like what we had for our residual sum of squares calculation,
but we're only including the houses that
are present in our training data set. Okay, so then for any given model,
such as a linear fit through the data, quadratic fit, or so on, what we can do is
we can think about estimating our model parameters as those that
minimize the training error. So that's equivalent to what we talked
about before of minimizing the residual sum of squares. But again, here we're only looking at
the houses in our training data set. Okay, so then that's how
we get our estimated w hat, our estimated model parameters. But then what we wanna do is we wanna take
these estimated model parameters, and we wanna say how good are we doing? And remember what we said,
what we're gonna do, is we're gonna look at our
held out observations, okay? So here, these gray circles
are the houses that are in our test set. So these are houses that were
not used to fit this model. And we're gonna say how well are we
predicting these actual house sales? Okay, and so what were our predictions? Well, remember when we thought
about making a prediction, we just used the value of the fit,
so just the point on the line. So to assess how well we're predicting
these held-out observations, our test data,
we're gonna look at something that, again, looks exactly like
residual sum of squares. But it's called our test error, where we take these estimated
model parameters w hat, and we sum over our residual sum of squares over all
houses that are in our test data set. Okay, so that's our test error. But what we can think about is we can
think about how does test error and training error vary as a function
of the model complexity? [MUSIC]