[MUSIC]
Okay, well maybe we can just take our retrogression solution, and
just take all the little coefficients and just say they're 0, just get rid of those.
We're gonna call those thresholding those away.
We're gonna choose some value where, below that value of the magnitude of
the coefficient is below the threshold that we choose,
just going to say it's not in the model.
So, let's explore this idea a little bit.
So, here I'm just showing an illustration of a little cartoon of what
the weights might look like on a set of features in our housing application.
And, I'm choosing some threshold which is this dashed black line.
And if the magnitude exceeds that threshold,
then I'm gonna say that features in my model.
So here in, pink or fuchsia.
Carlos, what color is this?
>> Fuchsia.
This is Carlos's color scheme.
He's very attached to it, fuchsia.
So in fuchsia, I'm showing the features that have been selected to be
in my model after doing this thresholding of my ridge regression coefficients.
Might seem like a reasonable approach, but let's dig into this a little bit more.
And in particula,r let's look at two very related features.
So if you look at this list of features, you see, in green,
I've highlighted number of bathrooms and number of showers.
So, these numbers tend to be very, very close to one another.
Because lots of bathrooms have showers, and
as the number of showers grow, clearly the number of bathrooms grow
because you're very unlikely to have a shower not in a bathroom.
But what's happened here?
Well, our model has included nothing having to do with bathrooms, or showers,
or anything of this concept.
So that doesn't really make a lot of sense.
To me it seems like something having to do with how many bathrooms are in the house
should be a valuable feature to include when I'm assessing the value of the house.
So what's going wrong?
Well, if I hadn't included number of showers.
Let's just for simplicity's sake,
treat the number of showers as exactly equivalent to the number of bathrooms.
It might not be exactly equivalent, but they're very strongly related.
But like I said for simplicity, let's say they're exactly the same.
So if I hadn't included number of showers in my model to begin with,
in the full model, then when I did my ridge search, it would've placed that
weight that had been on number of showers, on the number of bathrooms.
Because remember, it's a linear model, we're summing over
weight times number of bathrooms plus weight times number of showers.
So if number of bathrooms equals number of showers,
it's equivalent to the sum of those two weights just times number of bathrooms,
excluding number of showers from the model.
Okay so, the point here is that if I hadn't included this redundant feature,
number of showers, what I see now visually, is that number of bathrooms
would have been included in my selected model doing the threshholding.
So, the issue that I'm getting at here.
It's not specific to the number of bathrooms and number of showers.
It's an issue that, if you have a whole collection, maybe not two,
maybe a whole set of strongly related features.
More formally, statistically I will call these strongly correlated features.
Then ridge regression is gonna prefer a solution that places a bunch of smaller
weights on all the features, rather than one large weight on one of the features.
Because remember the cost
under the ridge regression model is the size of that feature squared.
And so if you have one really big one, that's really gonna blow up that cost,
that L2 penalty term.
Whereas the fit of the model is gonna be basically about the same.
Whether I distribute the weights over redundant features or
if I put a big one on just for one of them and zeros elsewhere.
So what's gonna happen is I'm going to get a bunch of these small weights over
the redundant features.
And if I think about simply thresholding,
I'm gonna discard all of these redundant features.
Whereas one of them, or
potentially the whole set, really were relevant to my prediction task.
So hopefully, it's clear from this illustration that just taking ridge
regression and thresholding out these small weights,
is not a solution to our feature selection problem.
So instead we're left with this question of,
can we use regularization to directly optimize for sparsity?
[MUSIC]