When you implement back propagation you'll find that there's a test called
creating checking that can really help you make sure
that your implementation of back prop is correct.
Because sometimes you write all these equations and you're just not 100% sure if
you've got all the details right and internal back propagation.
So in order to build up to gradient and checking,
let's first talk about how to numerically approximate computations of gradients and
in the next video, we'll talk about how you can implement
gradient checking to make sure the implementation of backdrop is correct.
So lets take the function f and replot it here and remember this is
f of theta equals theta cubed, and let's again start off to some value of theta.
Let's say theta equals 1.
Now instead of just nudging theta to the right to get theta plus epsilon,
we're going to nudge it to the right and
nudge it to the left to get theta minus epsilon, as was theta plus epsilon.
So this is 1, this is 1.01, this is 0.99 where, again,
epsilon is same as before, it is 0.01.
It turns out that rather than taking this little triangle and
computing the height over the width, you can get a much better estimate of
the gradient if you take this point, f of theta minus epsilon and this point,
and you instead compute the height over width of this bigger triangle.
So for technical reasons which I won't go into, the height over width of this bigger
green triangle gives you a much better approximation to the derivative at theta.
And you saw it yourself, taking just this lower triangle in the upper right
is as if you have two triangles, right?
This one on the upper right and this one on the lower left.
And you're kind of taking both of them into account
by using this bigger green triangle.
So rather than a one sided difference, you're taking a two sided difference.
So let's work out the math.
This point here is F of theta plus epsilon.
This point here is F of theta minus epsilon.
So the height of this big green triangle is f of theta plus epsilon
minus f of theta minus epsilon.
And then the width, this is 1 epsilon, this is 2 epsilon.
So the width of this green triangle is 2 epsilon.
So the height of the width is going to be first the height, so
that's F of theta plus epsilon minus F of theta minus epsilon divided by the width.
So that was 2 epsilon which we write that down here.
And this should hopefully be close to g of theta.
So plug in the values, remember f of theta is theta cubed.
So this is theta plus epsilon is 1.01.
So I take a cube of that minus 0.99 theta cube of that divided by 2 times 0.01.
Feel free to pause the video and practice in the calculator.
You should get that this is 3.0001.
Whereas from the previous slide, we saw that g of theta,
this was 3 theta squared so when theta was 1, so
these two values are actually very close to each other.
The approximation error is now 0.0001.
Whereas on the previous slide, we've taken the one sided
of difference just theta + theta + epsilon we had gotten 3.0301 and
so the approximation error was 0.03 rather than 0.0001.
So this two sided difference way of
approximating the derivative you find that this is extremely close to 3.
And so this gives you a much greater confidence that g of theta is
probably a correct implementation of the derivative of F.
When you use this method for grading, checking and back propagation,
this turns out to run twice as slow as you were to use a one-sided defense.
It turns out that in practice I think it's worth it to use this other method because
it's just much more accurate.
The little bit of optional theory for
those of you that are a little bit more familiar of Calculus, it turns out that,
and it's okay if you don't get what I'm about to say here.
But it turns out that the formal definition of a derivative is for
very small values of epsilon is f of theta plus epsilon minus f of theta
minus epsilon over 2 epsilon.
And the formal definition of derivative is in the limits of exactly
that formula on the right as epsilon those as 0.
And the definition of unlimited is something that you learned if you
took a Calculus class but I won't go into that here.
And it turns out that for a non zero value of epsilon,
you can show that the error of this approximation is on the order
of epsilon squared, and remember epsilon is a very small number.
So if epsilon is 0.01 which it is here then epsilon squared is 0.0001.
The big O notation means the error is actually some constant times this, but
this is actually exactly our approximation error.
So the big O constant happens to be 1.
Whereas in contrast if we were to use this formula, the other one,
then the error is on the order of epsilon.
And again, when epsilon is a number less than 1, then epsilon is actually
much bigger than epsilon squared which is why this formula here is actually
much less accurate approximation than this formula on the left.
Which is why when doing gradient checking, we rather use this two-sided difference
when you compute f of theta plus epsilon minus f of theta minus epsilon and then
divide by 2 epsilon rather than just one sided difference which is less accurate.
If you didn't understand my last two comments, all of these things are on here.
Don't worry about it.
That's really more for those of you that are a bit more familiar with Calculus, and
with numerical approximations.
But the takeaway is that this two-sided difference formula is much more accurate.
And so that's what we're going to use when we do gradient checking in the next video.
So you've seen how by taking a two sided difference,
you can numerically verify whether or not a function g, g of theta that someone
else gives you is a correct implementation of the derivative of a function f.
Let's now see how we can use this to verify whether or
not your back propagation implementation is correct or
if there might be a bug in there that you need to go and tease out