[MUSIC]
Okay, so now that all of you are optimization
experts we can think about applying these optimization notions and optimization algorithms that we described
to our specific scenario of interest. Which is searching over
all possible lines and finding the line that best fits our data. So the first thing that's important to
mention is the fact that our objective is Convex. So you can go and
show that this is a convex function. And what this is implies
is that the solution to this minimization problem is unique. We know there's a unique
minimum to this function. And likewise, we know that our gradient descent
algorithm will converge to this minimum. Gradient descent algorithm will converge to the minimum. Okay, so this is good news for
trying to find the line that best fits our data because this is a very, it sounds
like a very complicated problem at least. We have to search over all possible lines. And, of course we couldn't possibly go and
test each one of these lines, and it's very, very helpful that
there is this property, so we can use these very straightforward
algorithms that we described to go and solve for the solution to this problem. Okay, so let's return to
the definition of our cost, which is the residual sum of
squares of our two parameters, (wo,w1), which I've
written again right here. But before we get to taking the gradient,
which is gonna be so crucial for our gradient descent algorithm, let's just
note the following fact about derivatives, where if we take the derivative
of the sum of functions over some parameter, some variable w. Then we can write this equivalently, so this sigma notation,
it's just shorthand for this. Full sum over the N terms. So N different functions, g1 to gN,
the derivative distributes across the sum. And we can rewrite this as
the sum of the derivative, okay? So the derivative of the sum of functions
is the same as the sum of the derivative of the individual functions so
in our case, We have that gi(w). The gi that I'm writing here is
equal to (yi-[w0+w0+w1xi]) squared. And we see that the residual sum of squares is indeed a sum over n different functions of w0 and w1. And so in our case when
we're thinking about taking the partial of the residual
sum of squares. With respect to, for example w0, this is going to be equal to the sum,
I equals 1 to N, of the partial with respect to W0. (Yi- [W0 + W1Xi]) squared, and the same holds for W1. Okay, so now let's go ahead and
actually compute this gradient. And the first thing we're gonna
do is take the derivative or the partial with respect to the W0. Okay, so I'm gonna use this fact that, I showed on the previous slide
to take the sum to the outside. And then I'm gonna take the partial
with respect to the inside. So, I have a function raised to a power. So i'm gonna bring that power down. I'm gonna get a 2 here,
rewrite the function. That's W1 XI, and
now the power is just gonna be 1 here. But then, I have to take
the derivative of the inside. And so what's the derivative of
the inside when I'm taking this derivative with respect to W0? Well, what I have is I have
a -1 multiplying W0, and everything else I'm just
treating as a constant. So, I need to multiply by -1. So I'll just rewrite this. I'm gonna multiply the 2 by this -1. I'm gonna pull it outside the sum. So I'm gonna get -2 sum i equals one to n. Yi-w0+w1xi, promise I'll try and minimize how many derivations we're
doing in the slides like this. But I think this is really instructive. So, let's go ahead and
now take the derivative or the partial with respect to W1. So in this case, again I'm pulling the sum out same thing happens
where I'm gonna bring the 2 down. And I'm gonna rewrite the function here, the inside part of the function,
raise it just to the 1 power. And then, when I take the derivative
of this part, this inside part, with respect to W1, What do I have? Well all of these things
are constants with respect to W1, but what's multiplying W1? I have a -xi so
I'm going to need to multiply by -xi. Okay, so again rewriting this. I'm going to take the -2
to the outside of the sum. And I'm gonna have yi-w0 +w1xi. It's a good thing you guys
can put me on two speed and just get through all of this very quickly. But here we're multiply by xi and
we have this extra term in the derivative with respect to w1 okay,
well let's put this all together and look, I went ahead and, I typed it out so, I
don't need to write it here, that's good. So this is the gradient of our
residual sum of squares, and it's a vector of two dimensions because
we have two variables, w0 and w1. Now what can w think about doing? Well of course we can think about
doing the gradient descent algorithm. But let's hold off on that because what
do we know is another way to solve for the minimum of this function? Well we know we can, just like we talked
about in one D, taking the derivative and setting it equal to zero,
that was the first approach for solving for the minimum. Well here we can take the gradient and
set it equal to zero. [MUSIC]