Hi. In this video,
we will cover basic approach as to
feature preprocessing and feature generation for numeric features.
We will understand how model choice impacts feature preprocessing.
We will identify the preprocessing methods that are used most often,
and we will discuss feature generation and go through several examples.
Let's start with preprocessing.
First thing you need to know about handling numeric features is
that there are models which do and don't depend on feature scale.
For now, we will broadly divide
all models into tree-based models and non-tree-based models.
For example, decision trees classifier tries
to find the most useful split for each feature,
and it won't change its behavior and its predictions.
It can multiply the feature by a constant and to retrain the model.
On the other side, there are models which depend on these kind of transformations.
The model based on your nearest neighbors,
linear models, and neural network.
Let's consider the following example.
We have a binary classification test with two features.
The object in the picture belong to different classes.
The red circle to class zero,
and the blue cross to class one,
and finally, the class of the green object is unknown.
Here, we will use a one nearest neighbor's model
to predict the class of the green object.
We will measure distance using square distance,
which is also called altometric.
Now, if we calculate distances to the red circle and to the blue cross,
we will see that our model will predict class one for
the green object because the blue cross of class one is much closer than the red circle.
But if we multiply the first feature by 10,
the red circle will became the closest object,
and we will get an opposite prediction.
Let's now consider two extreme cases.
What will happen if we multiply the first feature by zero and by one million?
If the feature is multiplied by zero,
then every object will have feature relay of zero,
which results in KNN ignoring that feature.
On the opposite, if the feature is multiplied by one million,
slightest differences in that features values will impact prediction,
and this will result in KNN favoring that feature over all others.
Great, but what about other models?
Linear models are also experiencing difficulties with differently scaled features.
First, we want regularization to be applied
to linear models coefficients for features in equal amount.
But in fact, regularization impact turns out to be proportional to feature scale.
And second, gradient descent methods can go crazy without a proper scaling.
Due to the same reasons,
neural networks are similar to
linear models in the requirements for feature preprocessing.
It is important to understand that
different features scalings result in different models quality.
In this sense, it is just another hyper parameter you need to optimize.
The easiest way to do this is to rescale all features to the same scale.
For example, to make the minimum of a feature equal to zero and the maximum equal to one,
you can achieve this in two steps.
First, we sector at minimum value.
And second, we divide the difference base maximum.
It can be done with MinMaxScaler from sklearn.
Let's illustrate this with an example.
We apply the so-called MinMaxScaler to
two features from the detaining dataset, Age and SibSp.
Looking at histograms, we see that the features have different scale,
ages between zero and 80,
while SibSp is between zero and 8.
Let's apply MinMaxScaling and see what it will do.
Indeed, we see that after this transformation,
both age and SibSp features were successfully converted to the same value range of 0,1.
Note that distributions of values which we observe from the histograms didn't change.
To give you another example,
we can apply a scalar named StandardScaler in sklearn,
which basically first subtract mean value from the feature,
and then divides the result by feature standard deviation.
In this way, we'll get standardized distribution,
with a mean of zero and standard deviation of one.
After either of MinMaxScaling or StandardScaling transformations,
features impacts on non-tree-based models will be roughly similar.
Even more, if you want to use KNN,
we can go one step ahead and recall that the bigger feature is,
the more important it will be for KNN.
So, we can optimize scaling parameter to boost features which
seems to be more important for us and see if this helps.
When we work with linear models,
there is another important moment that influences model training results.
I'm talking about outiers.
For example, in this plot, we have one feature,
X, and a target variable, Y.
If you fit a simple linear model,
its predictions can look just like the red line.
But if you do have one outlier with X feature equal to some huge value,
predictions of the linear model will look more like the purple line.
The same holds, not only for features values,
but also for target values.
For example, let's imagine we have a model trained
on the data with target values between zero and one.
Let's think what happens if we add a new sample in
the training data with a target value of 1,000.
When we retrain the model,
the model will predict abnormally high values.
Obviously, we have to fix this somehow.
To protect linear models from outliers,
we can clip features values between two chosen values of lower bound and upper bound.
We can choose them as some percentiles of that feature.
For example, first and 99s percentiles.
This procedure of clipping is well-known in
financial data and it is called winsorization.
Let's take a look at this histogram for an example.
We see that the majority of feature values are between zero and 400.
But there is a number of outliers with values around -1,000.
They can make life a lot harder for our nice and simple linear model.
Let's clip this feature's value range and to do so, first,
we will calculate lower bound and upper bound values as
features values at first and 99s percentiles.
After we clip the features values,
we can see that features distribution looks fine,
and we hope now this feature will be more useful for our model.
Another effective preprocessing for numeric features is the rank transformation.
Basically, it sets spaces between proper assorted values to be equal.
This transformation, for example,
can be a better option than MinMaxScaler if we have outliers,
because rank transformation will move the outliers closer to other objects.
Let's understand rank using this example.
If we apply a rank to the source of array,
it will just change values to their indices.
Now, if we apply a rank to the not-sorted array,
it will sort this array,
define mapping between values and indices in this source of array,
and apply this mapping to the initial array.
Linear models, KNN, and neural networks can benefit from
this kind of transformation if we have no time to handle outliers manually.
Rank can be imported as a random data function from scipy.
One more important note about the rank transformation is that to apply to the test data,
you need to store the creative mapping from features values to their rank values.
Or alternatively, you can concatenate,
train, and test data before applying the rank transformation.
There is one more example of numeric features preprocessing which
often helps non-tree-based models and especially neural networks.
You can apply log transformation through your data,
or there's another possibility.
You can extract a square root of the data.
Both these transformations can be useful because they
drive too big values closer to the features' average value.
Along with this, the values near zero are becoming a bit more distinguishable.
Despite the simplicity, one of these transformations can
improve your neural network's results significantly.
Another important moment which holds true for all preprocessings is that sometimes,
it is beneficial to train a model on
concatenated data frames produced by different preprocessings,
or to mix models training differently-preprocessed data.
Again, linear models, KNN,
and neural networks can benefit hugely from this.
To this end, we have discussed numeric feature preprocessing,
how model choice impacts feature preprocessing,
and what are the most commonly used preprocessing methods.
Let's now move on to feature generation.
Feature generation is a process of creating
new features using knowledge about the features and the task.
It helps us by making model training more simple and effective.
Sometimes, we can engineer these features using prior knowledge and logic.
Sometimes we have to dig into the data,
create and check hypothesis,
and use this derived knowledge and our intuition to derive new features.
Here, we will discuss feature generation with prior knowledge,
but as it turns out,
an ability to dig into the data and
derive insights is what makes a good competitor a great one.
We will thoroughly analyze and illustrate this skill in
the next lessons on exploratory data analysis.
For now, let's discuss examples of feature generation for numeric features.
First, let's start with a simple one.
If you have columns,
Real Estate price and Real Estate squared area in the dataset,
we can quickly add one more feature,
price per meter square.
Easy, and this seems quite reasonable.
Or, let me give you another quick example from the Forest Cover Type Prediction dataset.
If we have a horizontal distance to a water source and
the vertical difference in heights within the point and the water source,
we as well may add combined feature indicating
the direct distance to the water from this point.
Among other things, it is useful to know that adding, multiplications,
divisions, and other features interactions can be of help not only for linear models.
For example, although gradient within decision tree is a very powerful model,
it still experiences difficulties with approximation of multiplications and divisions.
And adding size features explicitly can
lead to a more robust model with less amount of trees.
The third example of feature generation for numeric features is also very interesting.
Sometimes, if we have prices of products as a feature,
we can add new feature indicating fractional part of these prices.
For example, if some product costs 2.49,
the fractional part of its price is 0.49.
This feature can help the model utilize
the differences in people's perception of these prices.
Also, we can find similar patterns in tasks
which require distinguishing between a human and a robot.
For example, if we will have some kind of financial data like auctions,
we could observe that people tend to set round numbers as prices,
and there are something like 0.935,
blah, blah,, blah, very long number here.
Or, if we are trying to find spambots on social networks,
we can be sure that no human ever read messages with an exact interval of one second.
Great, these three examples should have provided you an idea that
creativity and data understanding are the keys to productive feature generation.
All right, let's summarize this up.
In this video, we have discussed numeric features.
First, the impact of feature preprocessing is different for different models.
Tree-based models don't depend on scaling,
while non-tree-based models usually depend on them.
Second, we can treat scaling as
an important hyper parameter in cases
when the choice of scaling impacts predictions quality.
And at last, we should remember
that feature generation is powered by an understanding of the data.
Remember this lesson and this knowledge will surely help you in your next competition.