Hi everyone.
This video is dedicated to the following advanced feature engineering techniques.
Calculating various statistics of one feature grouped by
another and features derived from neighborhood analysis of a given point.
To make it a little bit clearer,
let's consider a simple example.
Here we have a chunk of data for some CTR task.
Let's forget about target variable and focus on human features.
Namely, User_ID, unique identifier of a user,
Page_ID, an identifier of a page user visited,
Ad_price, item prices in the ad,
and Ad_position, relative position of an ad on the web page.
The most straightforward way to solve this problem is to label
and call the Ad_position and feed some classifier.
It would be a very good classifier that could take into
account all the hidden relations between variables.
But no matter how good it is,
it still treats all the data points independently.
And this is where we can apply feature engineering.
We can imply that an ad with
the lowest price on the page will catch most of the attention.
The rest of the ads on the page won't be very attractive.
It's pretty easy to calculate the features relevant to such an implication.
We can add lowest and highest prices for every user and page per ad.
Position of an ad with the lowest price could also be of use in such case.
Here's one of the ways to implement statistical features with paid ads.
If our data is stored in the data frame df,
we call groupby method like this to get maximum and minimum price values.
Then store this object in gb variable,
and then join it back to the data frame df. This is it.
I want to emphasize that you should not stop at this point.
It's possible to add other useful features not
necessarily calculated within user and page per.
It could be how many pages user has visited,
how many pages user has visited during the given session,
and ID of the most visited page,
how many users have visited that page,
and many, many more features.
The main idea is to introduce new information.
By that means, we can drastically increase the quality of the models.
But what if there is no features to use groupby on?
Well, in such case,
we can replace grouping operations with finding the nearest neighbors.
On the one hand, it's much harder to implement and collect useful information.
On the other hand, the method is more flexible.
We can fine tune things like the size of relevant neighborhood or metric.
The most common and natural example of
neighborhood analysis arises from purposive pricing.
Imagine that you need to predict rental prices.
You would probably have some characteristics like floor space,
number of rooms, presence of a bus stop.
But you need something more than that to create a really good model.
It could be the number of other houses in
different neighborhoods like in 500 meters, 1,000 meters,
or 1,500 meters, or average price per square meter in such neighborhoods,
or the number of schools,
supermarkets, and parking lots in such neighborhoods.
The distances to the closest objects of interest
like subway stations or gyms could also be of use.
I think you've got the idea.
In the example, we've used a very simple case,
where neighborhoods were calculated in geographical space.
But don't be afraid to apply this method to
some abstract or even anonymized feature space.
It still could be very useful.
My team and I used this method in Spring Leaf competition.
Furthermore, we did it in supervised fashion.
Here is how we have done it.
First of all, we applied mean encoding to all variables.
By doing so, we created homogeneous feature space so we
did not worry about scaling and importance of each particular feature.
After that, we calculated 2,000 nearest neighbors with Bray-Curtis metric.
Then we evaluated various features from
those neighbors like mean target of nearest 5, 10, 15, 500,
2,000 neighbors, mean distance to 10 closest neighbors,
mean distance to 10 closest neighbors with target 1,
and mean distance to 10 closest neighbors with target 0,
and, it worked great.
In conclusion, I hope you embrace the main ideas of
both groupby and nearest neighbor methods
and you would be able to apply them in practice.
Thank you for your attention.