In the previous video, we gave a mathematical definition of gradient
descent. Let's delve deeper, and in this video, get better intuition about what the
algorithm is doing, and why the steps of the gradient descent algorithm might make
sense. Here's the gradient descent algorithm that we saw last time. And, just
to remind you, this parameter, or this term, alpha, is called the learning rate.
And it controls how big a step we take when updating my parameter theta J. And
this second term here is the derivative term. And what I want to do in this video
is give you better intuition about what each of these two terms is doing and why, when put
together, this entire update makes sense. In order to convey these intuitions, what
I want to do is use a slightly simpler example where we want to minimize the
function of just one parameter. So, so we have a, say we have a cost function J of
just one parameter, theta one, like we did, you know, a few videos back. Where
theta one is a real number, okay? Just so we can have 1D plots, which
are a little bit simpler to look at. And let's try to understand why gradient
descent would do on this function. [sound]. So, let's say here's my function.
J of theta one, and so that's my, and where theta one is a real number. Right,
now let's say I've initialized gradient descent with theta one at this location.
So image that we start off at that point on my function. What gradient descent will
do, is it will update. Theta one gets updated as theta one minus Alpha times DD
theta one J of theta one, right? and oh an just as an aside you know this, this
derivative term, right? If you're wondering why I changed the notation from
these partial derivative symbols. If you don't know what the difference is between
these partial derivative symbols and the dd theta don't worry about it. Technically
in mathematics we call this a partial derivative, we call this a derivative,
depending on the number of, of parameters in the function J, but that's a
mathematical technicality, so, you know For the purpose of this lecture, think of
these partial symbols, and DD theta one as exactly the same thing. And, don't worry
about whether there are any differences. I'm gonna try to use the mathematically
precise notation. But for our purposes, these notations are really the same thing.
So, let's see what this, this equation will do. And so we're going to compute
this derivative of, I'm not sure if you've seen derivatives in calculus before. But
what a derivative, at this point, does, is basically saying, you know, let's. Take
the tangent to that point, like that straight line, the red line, just,
just touching this function and let's look at the slope of this red line. That's
where the derivative is. It says what's the slope of the line that is just
tangent to the function, okay, and the slope of the line is of course is just
right, you know just the height divided by this horizontal thing. Now. This line has
a positive slope, so it has a positive derivative. And so, my update to theta is
going to be, theta one gives the update that theta one minus alpha times some positive
number. Okay? Alpha, the learning rate is always a positive number. And so
I'm gonna to take theta one, this update as theta one minus something. So I'm gonna
end up moving theta one to the left. I'm gonna decrease theta one and we can see
this is the right thing to do because I actually went ahead in this direction you
know to get me closer to the minimum over there. So, gradient descent so far seems
to be doing the right thing. Let's look at another example. So let's take my same
function j. Just trying to draw the same function j of theta one. And now let's say
I had instead initialized my parameter over there on the left. So theta one is
here. I'm gonna add that point on the surface. Now, my derivative term, d, d
theta one j of theta one, when evaluated at this point, gonna look at right. The
slope of that line. So this derivative term is a slope of this line. But this
line is slanting down, so this line has negative slope. Right? Or alternatively I
say that this function has negative derivative, just means negative slope at
that point. So this is less than equal to zero. So when I update theta, then if
theta is updated as theta minus alpha times a negative number. And so I have
theta one minus a negative number which means I'm actually going to increase theta,
right? Because this is minus of a negative number means I'm adding something to theta
and what that means is that I'm going to end up increasing theta. And so we'll
start here and increase theta, which again seems like the thing I want to do to try
to get me closer to the minimum. So, this hopefully explains the intuition behind
what the derivative term is doing. Let's next take a look at the learning rate term
alpha, and try to figure out what that's doing. So, here's my gradient descent
update rule. Right, there's this equation And let's look at what can happen, if
Alpha is either too small, or if Alpha is too large. So this first example, what
happens if Alpha is too small. So here's my function J. J of theta. Lets
just start here. If alpha is too small then what I'm going to do is gonna
multiply the update by some small number. So end up taking, you know, it's like a baby step
like that. Okay, so that's one step [inaudible]. Then from this new point
we're gonna take another step. But if the alpha is too small lets take another
little baby step. And so if And so if my learning rate is too small. I'm gonna end
up, you know, taking these tiny, tiny baby steps to try to get to the minimum and I'm
gonna need a lot of steps to get to the minimum and so. If alpha's too small, can
be slow because it's gonna take these tiny, tiny baby steps. And it's gonna need
a lot of steps before it gets anyway close to the global minimum. Now,
how about if the alpha is too large. So here's my function J of theta.
Turns out if alpha is too large, then grading descent can overshoot a minimum
and may even fail to converge or even diverge. So here is what I mean. Let's say [inaudible] ireful minimum So the derivative council
It's actually close to the minimum. So the derivative points to the right, but if alpha is too big, I'm gonna
take a huge step. Maybe I'm gonna take a huge step like that. Right? So I end up taking a huge step.
Now, my cost function has got worse. cause it starts off from this value then now my value has gotten worse. Now my
derivatives, you know, points to the left, it's actually decrease theta. But look, if my learning rate is too big,
I may take a huge step going from here all the way out there so I end up. going all
there. Right? And if my learning rate was too big I can take another huge step on the
next acceleration and kind of overshoot and overshoot and so on until you notice
I'm actually getting further and further away from the minimum. And so if alpha is
too large it can fail to converge or even diverge. Now, I have another question for
you. So, this is a tricky one. And when I was first learning this stuff, it actually
took me a long time to figure this out. What if your pre-emptive theta one is
already at a local minimum? What do you think one step of gradient descent will
do? So let's suppose you initialize theta one at a local minimum. So you know
suppose this is your initial value of theta one over here and it's already at a local
optimum or the local minimum. It sends out that at local optimum your derivative
would be equal to zero. Since it's that slope where it's that tangent point so the
slope of this line will be equal to zero and thus this derivative term. Is equal to
zero. And so, in your gradient descent update, you have theta one, gives update
that theta one, minus alpha times zero. And so, what this means is that, if you're
already at a local optimum, it leaves theta one unchanged 'cause this, you know,
gets the update's theta one equals theta one. So if your parameter is already at a local
minimum, one step of gradient descent does absolutely nothing. It doesn't change
the parameter, which is, which is what you want. Cuz it keeps your solution at the
local optimum. This also explains why gradient descent can converge the local
minimum, even with the learning rate Alpha fixed. Here's what I mean by that. Let's
look at an example. So here's a cost function J with theta. That maybe I want
to minimize and let's say I initialize my algorithm my gradient descent algorithm, you know,
out there at that magenta point. If I take one step of gradient descent you know,
maybe it'll take me to that point cuz my derivative's pretty steep out there, right?
Now I'm at this green point and if I take another step of gradient descent, you
notice that my derivative, meaning the slope, is less steep at the green point when
compared to at the magenta point out there, right? Because as I approach the
minimum my derivative gets closer and closer to zero as I approach the minimum.
So, after one step of gradient descent, my new derivative is a little bit smaller.
So I wanna take another step of gradient descent. I will naturally take a somewhat
smaller step from this green point than I did from the magenta point. Now I'm at the new
point, the red point, and then now even closer to global minimums, so the
derivative here will be even smaller than it was at the green point. So when I take
another step of gradient descent, you know, now my derivative term is even smaller, and so
the magnitude of the update to theta one is even smaller, so you can take
small step like so, and as gradient descent runs. You will automatically take smaller
and smaller steps until eventually you are taking very small steps, you know, and you
find the converge to the to the local minimum. So, just to recap. In gradient
descent as we approach the local minimum, grading descent will automatically take
smaller steps and that's because as we approach the local minimum, by definition,
local minimum is when you have this derivative equal to zero. So as we
approach the local minimum this derivative term will automatically get smaller and
so gradient descent will automatically take smaller step. So, this is what
gradient descent looks like, and so actually there is no need to decrease alpha
overtime. So, that's the gradient descent algorithm, and you can use it to minimize,
to try to minimize any cost function J. Not the cost function J to be defined for
linear regression. In the next video, we're going to take the function J, and
set that back to be exactly linear regression's cost function. The, the
square cost function that we came up with earlier. And taking gradient descent, and
the square cost function, and putting them together. That will give us our first
learning algorithm, that'll give us our linear regression algorithm.