In this video, we will review
the most common ranking metrics and establish an intuition about them.
Although in a competition,
the metric is fixed for us,
it is still useful to understand in what cases one metric could be preferred to another.
In this course, we concentrate on regression and classification,
so we will only discuss related metrics.
For a better understanding, for each metric,
we will also build the most simple baseline we could imagine, the constant model.
That is, if we are only allowed to predict the same value for every object,
what value is optimal to predict according to the chosen metric?
Let's start with regression task and related metrics.
In the following videos,
we'll talk about metrics for classification.
First, let us clarify the notation we're going to use throughout the lesson.
N will be the number of samples in our training data set,
y is that the target,
and y-hat is our model's predictions.
And y-hat and y with index i are the predictions,
and target value respectively for i-th object.
The first metric we will discuss is Mean Square Error.
It is for sure the most common metric for regression type of problems.
In data science, people use it when they don't have
any specific preferences for the solution to their problem,
or when they don't know other metric.
MSE basically measures average squared error of our predictions.
For each point, we calculate square difference between
the predictions of the target and then average those values over the objects.
Let's introduce a simple data set now.
Say, we have five objects,
and each object has some features,
X, and the target is shown in the column Y.
Let's ask ourselves a question.
How will the error change if we fix all the predictions but want to be perfect,
and we'll derive the value of the remaining one?
To answer this question,
take a look at this plot.
On the horizontal line,
we will first put points to the positions of the target values.
The points are colored according to the corresponding rows in our data table.
And on the Y-axis,
we will show the mean square error.
So, let's now assume that our predictions for the first four objects are perfect,
and let's draw a curve.
How the metric value will change if we change the prediction for the last object?
For MSE metric, it looks like that.
In fact, if we predict 25,
the error is zero,
and if we predict something else,
then it is greater than zero.
And the error curve looks like parabola.
Let's now draw analogous curves for other objects.
Well, right now it's hard to make any conclusions but we will build the same kind
of plot for every metric and we will note the difference between them.
Now, let's build the simplest baseline model.
We'll not use the features X at all and we will always predict a constant value Alpha.
But, what is the optimal constant?
What constant minimizes the mean square error for our data set?
In fact, it is easier to set the derivative
of our total error with respect to that constant to zero,
and find it from this equation.
What we'll find is that the best constant is the mean value of the target column.
If you think you don't know how to derive it,
take a look at the reading materials.
There is a fine explanation and links to related books.
But let us constructively check it.
Once again, on the horizontal axis,
let's denote our target values with dot and draw a function.
How the error changes is if we change the value of that constant Alpha?
We can do it with a simple grid search over a given range
by changing Alpha intuitively and recomputing an error.
Now, the green square shows a minimum value for our metric.
The constant we found is 10.99,
and it's quite close to the true mean of the target which is 11.
In fact, the value we got deviates from
the true mean value only because with the grid search,
we get only approximate answer.
Also note that the red curve on the second plot is
uniformly same and average of the curves from the first plot.
We are finished discussing MSE metric itself,
but there are two more related metrics used frequently, RMSE and R_squared.
And we will briefly study them now.
RMSE, Root Mean Square Error,
is a very similar metric to MSE.
In fact, it is calculated in two steps.
First, we calculate regular mean square error and then,
we take a square root of it.
The square root is introduced to make scale
of the errors to be the same as the scale of the targets.
For MSE, the error is squared,
so taking a root out of it makes total error
a little bit easier to comprehend because it is linear now.
Now, it is very important to understand in what sense RMSE is similar to MSE,
and what is the difference.
First, they are similar in terms of their minimizers.
Every minimizer of MSE is a minimizer of RMSE and vice versa.
But generally, if we have two sets of predictions, A and B,
and say MSE of A is greater than MSE of B,
then we can be sure that RMSE of A is greater RMSE of B.
And it also works in the opposite direction.
This is actually true only because square root function is non-decreasing.
What does it mean for us?
It means that, if our target the metric is RMSE,
we still can compare our models using MSE,
since MSE will order the models in the same way as RMSE.
And we can optimize MSE instead of RMSE.
In fact, MSE is a little bit easier to work with,
so everybody uses MSE instead of RMSE.
But there is a little bit of difference between the two for gradient-based models.
Take a look at the gradient of RMSE with respect to i-th prediction.
It is basically equal to gradient of MSE multiplied by some value.
The value doesn't depend on the index I.
It means that travelling along MSE gradient is equivalent to traveling along
RMSE gradient but with
a different flowing rate and the flowing rate depends on MSE score itself.
So, it is kind of dynamic.
So even though RMSE and MSE are really similar in terms of models scoring,
they can be not immediately interchangeable for gradient based methods.
We will probably need to adjust some parameters like the learning rate.
Now, what if I told you that MSE for my models predictions is 32?
Should I improve my model or is it good enough?
Or what if my MSE was 0.4?
Actually, it's hard to realize if our model is good or
not by looking at the absolute values of MSE or RMSE.
It really depends on the properties of the dataset and their target vector.
How much variation is there in the target vector.
We would probably want to measure how much
our model is better than the constant baseline.
And say, the desired metrics should give us zero if we
are no better than the baseline and one if the predictions are perfect.
For that purpose, R_squared metric is usually used. Take a look.
When MSE of our predictions is zero,
the R_squared is 1,
and when our MSE is equal to MSE over constant model,
then R_squared is zero.
Well, because the values in numerator and denominator are the same.
And all reasonable models will score between 0 and 1.
The most important thing for us is that to optimize R_squared,
we can optimize MSE.
It will be absolutely equivalent since R_squared is basically
MSE score divided by a constant and subtracted from another constant.
These constants doesn't matter for optimization.
Lets move on and discuss another metric called Mean Absolute Error,
or MAE in short.
The error is calculated as an average of
absolute differences between the target values and the predictions.
What is important about this metric is that it penalizes
huge errors that not as that badly as MSE does.
Thus, it's not that sensitive to outliers as mean square error.
It also has a little bit different applications than MSE.
MAE is widely used in finance,
where $10 error is usually exactly two times worse than $5 error.
On the other hand, MSE metric thinks that $10 error is four times worse than $5 error.
MAE is easier to justify.
And if you used RMSE,
it would become really hard to explain to your boss how you evaluated your model.
What constant is optimal for MAE?
It's quite easy to find that its a median of the target values.
In this case, it is eight.
See reading materials for a proof.
Just to verify that everything is correct,
we again can try to Greek search for an optimal value with a simple loop.
And in fact, the value we found is 7.98,
which indicates we were right.
Here, we see that MAE is more robust than MSE,
that is, it is not that influenced by the outliers.
In fact, recall that the optimal constant for MSE was about 11 while for MAE it is eight.
And eight looks like a much better prediction for the points on the left side.
If we assume that point with a target 27 is
an outlier and we should not care about the prediction for it.
Another important thing about MAE is its gradients with respect to the predictions.
The grid end is a step function and it takes
-1 when Y_hat is smaller than the target and +1 when it is larger.
Now, the gradient is not defined when the prediction is perfect,
because when Y_hat is equal to Y,
we can not evaluate gradient.
It is not defined. So formally,
MAE is not differentiable,
but in fact, how often your predictions perfectly measure the target.
Even if they do, we can write a simple IF condition and return
zero when it is the case and through gradient otherwise.
Also know that second derivative is zero everywhere and not defined in the point zero.
I want to end the discussion with the last note.
Well, it has nothing to do with competitions
but every data scientists should understand this.
We said that MAE is more robust than MSE.
That is, it is less sensitive to outliers,
but it doesnt mean it is always better to use MAE.
No, it does not.
It is basically a question.
Are there any real outliers in the dataset or there are just,
let's say, unexpectedly high values that we should treat just as others?
Outliers have usually mistakes,
measurement errors, and so on,
but at the same time,
similarly looking objects can be of natural kind.
So, if you think these unusual objects are normal in the sense that they're just rare,
you should not use a metric which will ignore them.
And it is better to use MSE.
Otherwise, if you think that they are really outliers,
like mistakes, you should use MAE.
So in this video,
we have discussed several important metrics.
We first discussed, mean square error and realized
that the best constant for it is the mean targeted value.
Root Mean Square Error, RMSE,
and R_squared are very similar to MSE from optimization perspective.
We then discussed Mean Absolute Error and when people prefer to use MAE over MSE.
In the next video, we will continue to study
regression metrics and then we'll get to classification ones.