[MUSIC]
But instead, let's think about the update just to a single feature.
And think about what that looks like, and
give some intuition for why the update has this form.
So, let's go through and derive what the update is just for a single feature.
Of course, we could go through this matrix notation.
And figure out what is the update for that J row.
But, let's go through.
It's a little bit simpler to go back to this form of our residual summit squares
and drive it directly.
So, I'm just gonna rewrite this more explicitly,
so when we start taking derivatives it's easier to see what's going on.
So, we have the sum of our I equals 1,
2N and we have YI minus.
Now, lets write out what this vector inner product is and what is it?
Well, it's simply our fid.
So, we have W0 times H0 of XI minus
W1H1 of XI minus all the way to
our last feature, W capital D H
capital D of XI squared.
Okay, so remember when we're taking a gradient, well, what's the gradient?
It's just a vector of a bunch of partials
with respect to W0 then W1, all the way up to W capital D.
So, let's just look at one element, which is the partial of this residual sum of
squares with respect to WJ, and that will give us our update for this J entry.
So, the partial derivative of this function with respect to WJ.
You go through and remember what we did in this simple linear regression model.
We keep this sum on the outside, and
then we take the derivative of this function with respect to WJ.
So, the two is going to come down.
We're going to get this function repeated again.
So, YI minus W0H0XI minus W1H1XI
dot dot dot minus WDHDXI squared, and
then we have to multiply what's the coefficient,
sorry, it's no longer squared, it's to the 1 power.
Well, let's go back to what I was asking, what's the coefficient associated with WJ?
Well, it's minus HJ, so it's negative of our Jth feature.
Sorry, my pen stopped writing for
a second there, minus HJ of XI.
Okay, so let's write this more compactly.
We can write this as I = 1 to N of, sorry,
I'll bring out the minus two to the outside here.
Minus two times this sum,
where the sum we're going to have HJ of XI, inside the sum.
That's this term.
And then, we're going to multiply by this function which I'm gonna return to vector
notation, just h(xi), transpose w.
But in this case, there's no square.
Okay, so I'm gonna plug this into
the update to the Jth feature weight,
where I take the previous weight of that feature and
subtract off a step size times the partial,
which is -2 sum i=1 to n hj xi times yi- h transpose of xi,
this being this vector times w also a vector.
Where specifically we're looking at W from this Tth iteration.
Okay.
So, again, we can add a little bit of interpretation here where this part here,
this is, I'm taking the features, all the features for
my Ith observation, multiplying by the entire W vector.
So, this is my predicted value of my Ith observation using W of T.
Okay.
So, let's rewrite this.
So, here, I've just written exactly what I had on the previous slide.
Now, let's think about interpreting this update here.
In particular, let's assume that WJ
corresponds to the coefficient associated with number of baths.
So, let's say the Jth features,
let's just assume is number of bathrooms.
And what happens if in general, I'm overestimating.
Sorry, not overestimating.
I'm underestimating the impact of the number of baths on my predicted
value of the house.
So, what that means is, if I look along this bathrooms direction, and
I look at the slope of this hyperplane, I'm saying that it's
not steep enough so increasing bathrooms actually has more impact
on the value of the house than my currently estimated model thinks it does.
Okay so what's gonna happen?
So, if underestimating
the impact of number of bathrooms.
So, what this corresponds to is
W hat J iteration T is too small,
then what I'm gonna have is that
my observations in general are larger
than my predicted observations,
so this term here on average,
we'll be positive.
Okay, but we're taking this average, multiplying
by the feature that's number bathrooms.
So, on average.
Weighted by number of bathrooms,
where this here is number of bathrooms for house I.
And that's what we're multiplying that by.
So this, sorry, that should say then, will be positive.
And what's the impact of that?
The impact of that is this whole term here,
what we're adding to WJ, will be positive.
So, we're gonna increase WJ hat.
I'll just say WJ, sorry,
I don't know how to annotate this to make it clear what I'm saying.
I'll say WJ T plus one will be greater than WJ T.
We are increasing the value.
And let's talk very quickly about this weighing by the number of baths here.
Why are we weighing this by the number of baths?
Well, of course, the observations that have more of the feature,
more numbers baths should weigh more heavily in our assessment of the fit.
So, that's why whenever we look at the residual,
we weight by the value of the feature that we're considering.
Okay, so
this gives us a little bit of intuition behind this gradient descent algorithm,
particularly looking just feature by feature and what the algorithm looks like.
[MUSIC]