[MUSIC]
Now let's turn to this important question of how to choose the lambda tuning
parameter.
And what we mentioned last module was that if we have some tuning parameter that
controls model complexity, then we can think for
every value of that tuning parameter, we can fit our model on our training data.
Then we can assess the performance of that fitted model on a validation set,
and we can tabulate this for all values of lambda that we might consider.
And choose the specific model complexity according to the error on this validation
set, and then assess the performance of the selected model on our test set.
Well, now what we've seen is rich regression is a special case
of an objective where there's a tuning parameter, lambda,
that's controlling model complexity.
And we'd like to see how we can select this tuning parameter.
So, of course we can use exactly this procedure that we described last module.
And that's assuming that we have sufficient data to do
this training validation test split.
But now let's ask the question of, what if we don't have enough data to reasonably do
a divide into these three different sets?
What can you do?
And we're gonna talk about this in the context of rich regression, but again,
this holds for any tuning parameter lambda that you might have in selecting
between different model complexities.
Or any other tuning parameter controlling your model.
Okay, so we're assuming that we're starting with a smallish dataset.
And as always, we need to break off some test dataset that we're gonna hide, okay?
We always need to have some test set that's never touched during training or
validation of our model.
So now we took this smallish dataset and
we have even a smaller dataset to do our training and model validation.
So how are we gonna do this?
Well, we wanna do this in some way that's a bit smarter than
the naive approach of just forming our validation set.
So let's just remember this naive approach, where we took whatever data
was remaining after we split off our test set and we defined some validation set.
And a question is, in this case, when we have just a small amount of data, so
necessarily this validation set will just be a small number of observations,
is this sufficient for comparing between different model complexities and
accessing which one is best?
Well, no, clearly the answer is no.
We're saying that it's just a small set.
It's not gonna be representative of the space of things that we
might see out there.
Okay, so what can we do better?
We're stuck with just this dataset.
Well, did we have to use the last set of tabulated observations as
the observations to define this validation set?
No, I could of used the first few observations, or next set of observations,
or any random subset of observations in this dataset.
And a question is,
which subset of observations should I use as my validation set?
And the answer, and this is the key insight, is use all of the data subsets.
Because if you're doing that,
then you can think about averaging your performance across these validation sets.
And avoid any sensitivity you might have to one specific choice of validation set
that might give some strange numbers because it just has a few observations.
It doesn't give a good assessment of comparison between different model
complexities.
[MUSIC]