[MUSIC] So how are we gonna do this? How are we gonna use all of our
data as our validation set? We're gonna use something
called K-fold cross validation. Where the first step,
it's just a preprocessing step. Where we're gonna take our data, and
divide it into K different blocks. And, we have N total observations, so,
every block of data is gonna have N over K observations, and these observations
are randomly assigned to each block. Okay, so, this is really key that
we're taking our tabulated data. And in this image, even though it looks
like it just might be parceling out a table of data, the data in each one
of these blocks is randomly assigned. And for all steps of the algorithm
that I'm gonna describe now, we're gonna use exactly
the same data split. So, exactly the same assignments of observations to each one
of these different blocks. Okay, then for
each one of these K different blocks, we're gonna cycle through treating
each block as the validation set. And using all the remaining
observations to fit the model for every value of lambda. So in particular, we're gonna start by
saying for a specific value of lambda and we're gonna do a procedure for
each of the K blocks and at the end we're gonna cycle
through all values of lambda. So for right now, assume that we're
looking at a specific lambda value out of a set of possible values we might look at. And now we're gonna cycle through
each one of our blocks where at the first iteration we're gonna fit
our model using all the remaining data. That's gonna produce something
that I'm calling w hat lambda, so indexed by this lambda
that we're looking at. So we're considering the first
block as our validation set. Then we're gonna take
that fitted model and we're gonna assess it's performance
on this validation site. That's gonna result in some error
which I'm calling error sub one. Meaning the error on the first block
of data for this value of lambda. Okay, so I'm gonna keep track,
of the error for the value of lambda for each block, and then I'm gonna do this for
every value of lambda. Okay, so
I'm gonna move on to the next block, treat that as my validation set,
fit the model on all the remaining data, compute the error of that fitted
model on that second block of data. Do this on a third block,
fit data on all the remaining data, assess the performance on the third block, and
cycle through each of my blocks like this. And at the end, I've tabulated my error across each of these K different
blocks for this value of lambda. And what I'm gonna do is I'm gonna compute
what's called the cross validation error of lambda, which is simply an average of the air that
I had on each of the K different blocks. So now I explicitly see how my
measure of air, my summary of air for the specific value lambda
uses all of the data. it's an average across the validation
sets in each of the different blocks. Then, I'm gonna repeat this procedure for
every value that I'm considering of lambda and I'm gonna choose the lambda that
minimizes this cross validation error. So I had to divide my data into K
different blocks in order to run this K full cross validation algorithm. So a natural question is what
value of K should I use? Well you can show that the best
approximation to the generalization error of the model is given when
you take K to be equal to N. And what that means is that every
block has just one observation. So this is called leave-one-out
cross validation. So although it has the best approximation
of what you're trying to estimate, it tends to be very
computationally intensive, because what do we have to do for
every value of lambda? We have to do N fits of our model. And if N is even reasonably large, and if it's complicated to fit our model
each time, that can be quite intensive. So, instead what people tend
to do is use K = 5 or 10, this is called 5-fold or
10-fold cross validation. Okay, so this summarizes our cross
validation algorithm, which is a really, really important algorithm for
choosing two name parameters. And even though we discussed this option
of forming a training validation and test set, typically you're in a situation
where you don't have enough data to form each one of those. Or at least you don't know if you
have enough data to have an accurate approximation of generalization error as
well as assessing the difference between different models, so typically what
people do is cross validation. They hold out some test set and
then they do either leave one out, 5-fold, 10-fold cross validation to choose
their tuning parameter lambda. And this is a really critical step
in the machine learning workflow is choosing these tuning parameters in
order to select a model and use that for the predictions or
various tasks that you're interested in. [MUSIC]