[MUSIC] Okay, so
let's consider the resulting objective, where I'm gonna try and
search over all possible w vectors. To find the ones that minimize the sum of
residual sum of squares plus the square of the two norm of w. So that's gonna be my w hat,
my estimated model parameters. But really what I'd like to do, is I'd
like to be able to control how much I'm weighing the complexity of the model
as measured by this magnitude of my coefficient, relative to
the fit of the model. I'd like to balance between
these two terms, and so I'm gonna introduce another parameter. And this is called a tuning parameter. With the model, it's a lambda, and this is
balancing between this fit and magnitude. So let's see what happens
if I choose lambda to be 0. Well, if I choose lambda to be 0, this magnitude term that we've
introduced completely disappears and my objective reduces down just to
minimizing the residual sum of squares. Which was exactly the same
as my objective before. So, this reduces to minimizing residual sum of squares of w as before. So this is our old solution, Which leads to some w
hat which I'm gonna call w hat superscript LS for least squares. Because what we were doing
before is commonly referred to as the least squares solution. So I'm gonna specifically represent
the parameters associated with that old procedure we're doing
as the least squares parameters. On the other hand, what if I completely crank up that
tuning parameter to be infinity? So I have a really, really massively
large weight on this magnitude term. Massively large being infinitely large. So as large as you can possibly imagine. So what happens to any solution
where w hat is not equal to 0? So, For solutions where w hat does not equal 0. Then the total cost is what? Well I get something that's non-0
times infinity plus something, my residual sum of squares,
whatever that happens to be. But the sum of that is infinity. Okay, so my total cost is infinite. On the other hand,
what if w hat is exactly equal to 0? Then if w hat equals 0, then total cost is equal
to the residual sum of squares of this 0 vector. And that's some number, but
it's probably not infinity. Actually it's not infinity, so the minimizing solution here is
always gonna be w hat equals 0. Cuz that's the thing that's gonna minimize
the total cost over all possible w's. Okay, so just to recap, we said that
if we put that tuning parameter all the way to 0, make it very,
very small, all the way to 0. Then we return to our
previously square solution and if we crank that parameter all
the way up to be infinite. In that limit, we get all of our
coefficients being exactly 0, okay? But we're gonna be operating in a regime
where lambda is somewhere in between 0 and infinity. And in this case, Then we know that the magnitude of our
estimated coefficients, they're gonna be less than or equal to the magnitude of our
least squares coefficients. In particular,
the two norm will be less than. But we also know it's gonna be
greater than or equal to 0. So we're gonna be somewhere
in between these two regions. And a key question is,
what lambda do we actually want? How much do we want to bias away
from our least square solution, which was subject to potentially
over-fitting, down to this really simple, the most trivial model you can
consider which is nothing, no model? So, well not no model,
no coefficients in the model. What's the model if all
the coefficients are 0? Just noise, we just have y equals epsilon,
that noise term. Okay, so we're gonna think about somehow
trading off between these two extremes. Okay, I wanted to mention that this
is referred to as Ridge regression. And that's also known as
doing L2 regularization. Because, for reasons that we'll describe
a little bit more later in this module, we're regularizing the solution
to the old objective that we had, using this L2 norm term. [MUSIC]