[MUSIC]
For any specific value of lambda,
we get some balance between this residual sum of squares, and this two norm.
And so what I'm gonna do,
is in this movie, I'm gonna add these two contour plots together.
I'm gonna add, so let me write this down.
Add contour
plots together,
where I'm getting residual sum of squares of w plus lambda 2 norm of w.
Where here the residual sum of squares were these ellipses,
centered about my least squares solution, and there's to norm,
where these circles centered about zero.
And lambda is some weighting on how much I'm including
that two norm penalty or the cost.
And what I'm going to do is I'm going to show a movie as a function of lambda.
So movie, function of
increasing lambda,
where I have my ellipses, and I'm weighting more and
more heavily these contours that are coming from the circle.
The circle terms from this two norm penalty.
Okay, so this is the movie right here, and
my lovely assistant Carlos, will click the mouse to play the movie.
[LAUGH] Since I don't know how to control it from
the tablet, unfortunately.
[LAUGH] Thank you Fanna.
That reference was probably lost on most people.
And in doing so, you didn't get me describing the movie so
let's watch it again.
But what we see this x, let me be clear,
that the x is going to mark the optimal solution,
For a specific lambda and
we're varying lambda so this x is gonna move.
Okay where's the x gonna start?
Well when lambda's equal to zero, we're starting out our lee squared solution and
as lambda increases we know that as lambda goes to infinity
the coefficients are gonna shrink to zero.
But let's visualize the path that it takes as we increase lambda.
Okay so let's play this movie again, gonna start it early square the solution and
we see that the magnitude of our coefficients W0 and
W1 are shrinking smaller and smaller towards zero.
So, maybe we'll play that once more just to visualize this and what we say.
Again, this was just the tail end of the movie.
Is this shrinking magnitude of the coefficients?
Carlos is very excited about this movie, so we're gonna watch it one more time.
It's pretty cool.
We've never actually seen somebody do this visualization.
We think it's really intuitive.
So again, as that land of penalty is increasing,
the magnitude of the coefficients are getting shrunk.
Okay, well now let's talk about what the solution looks like for
a given value of lambda.
Oops, sorry let me turn my pen on.
So for a specific lambda value.
We have some balance between residual sum of squares and
the magnitude of our coefficients.
Lambda's automatically doing some trade-off between the two.
So some balance
Between RSS and our two norm.
And specifically, in this plot, this is our solution.
So, it has some RSS that happens to be, five thousand two hundred fifteen,
that's what the number on this contour is indicating and it has some tune arm,
which has value 4.75, and so
this lambda has chosen the specific trade off and we see that our solution
is somewhere here, which has shrunk from where our least square solution was.
Let's remember our least square solution was somewhere around here.
And the optimal for lambda equals infinity was that zero.
So it's somewhere in between these two values.
And if we had chosen a different value of lambda, let's say a larger value of
lambda We would of had a different solution.
And when I'm drawing all these contours, what I'm saying is,
let me just go back to the original one before this this drawing.
What I'm saying is, every other point along this circle,
has exactly the same residual sum of squares.
But larger l2 norm of w and everywhere
along this circle has exactly the same w2 norm,
sorry, l2 norm of w, but it has larger residual sum of squares.
So that's why this is the optimal trade off for this lambda.
Then, like I drew here, if I chose a larger lambda,
I will get a solution that preferred a smaller two norm and
a larger residual sum of squares.
So, this would be solution for
a larger lambda value.
Okay, so this is just a little visualization of what a ridge regression
solution looks like.
[MUSIC]