This isn't the rare case in competitions when you see
people jumping down on leaderboard after revealing private results.
So, we ask ourselves,
what is happening out there?
There are two main reasons for these jumps.
First, competitors could ignore the validation and
select the submission which scored best against the public leaderboard.
Second, is that sometimes competitions have
no consistent public/private data split or they
have too little data in either public or private leaderboard.
Well, we as participants,
can't influence competitions organization.
We can certainly make sure that we select
our most appropriate submission to be evaluated by private leaderboard.
So, the broad goal of next videos is
to provide you a systematic way to set up validation in a competition,
and tackle most common validation problems.
Let's quickly overview of the content of the next videos.
First, in this video,
we will understand the concept of validation and overfitting.
In the second video,
we will identify the number of splits that should be done to establish stable validation.
In the third video, we will go through
most frequent methods which are used to make train/test split in competitions.
In the last video,
we will discuss most often validation problems.
Now, let me start to explain
the concept for validation for those who may never heard of it.
In the nutshell, we want to check if the model gives expected results on the unseen data.
For example, if you've worked in
a healthcare company which goal is to improve life of patients,
we could be given the task of predicting if a patient will
be diagnosed a particular disease in the near future.
Here, we need to be sure that the model we train will be applicable in the future.
And not just applicable,
we need to be sure about what quality this model will
have depending on the number of mistakes the model make.
And on the predictive probability of a patient having this particular disease,
we may want to decide to run
special medical tests for the patient to clarify the diagnosis.
So, we need to correctly understand the quality of our model.
But, this quality can differ on train data from
the past and on the unseen test data from the future.
The model could just memorize all patients from the train data and be
completely useless on the test data because we don't want this to happen.
We need to check the quality of the model with the data we
have and these checks are the validation.
So, usually, we divide data we have into two parts,
train part and validation part.
We fit our model on the train part and check its quality on the validation part.
Beside that, in the last example,
our model will be checked against the unseen data in
the future and actually these data can differ from the data we have.
So we should be ready for this.
In competitions, we usually have the similar situation.
The organizers of a competition give us the data in two chunks.
First, train data with all target values.
And second, test data without target values.
As in the previous example,
we should split the data with labels into train and validation parts.
Furthermore, to ensure the competition spirit,
the organizers split the test data into the public test set and the private test set.
When we sent our submissions to the platform,
we see the scores for the public test set while the scores for
the private test set are released only after the end of the competition.
This also ensures that we don't need the test set or in terms of a model do not overfit.
Let me draw you an analogy with the disease projection,
if we already divided our data into train and validation parts.
And now, we are repeatedly checking our model against the validation set,
some models, just by chance,
will have better scores than the others.
If we continue to select best models, modify them,
and again select the best from them,
we will see constant improvements in the score.
But that doesn't mean we will see these improvements on the test data from the future.
By repeating this over and over,
we could just achieve the validation set or in terms of a competition,
we could just cheat the public leaderboard.
But again, if it overfit,
the private leaderboard will let us down.
This is what we call overfitting in a competition.
Get an unrealistically good scores on
the public leaderboard that later result in jumping down the private leaderboard.
So, we want our model to be able to capture patterns in
the data but only those patterns that generalize well between both train and test data.
Let me show you this process in terms of underfitting and overfitting.
So, to choose the best model,
we basically want to avoid underfitting on the one side and overfitting on the other.
Let's understand this concept on a very simple example of a binary classification test.
We will be using simple models defined by formulas under
the pictures and visualize the results of model's predictions.
Here on the left picture,
we can see that if the model is too simple,
it can't capture underlined relationship and we will get poor results.
This is called underfitting.
Then, if we want our results to improve,
we can increase the complexity of the model and
that will probably find that quality on the training data is going down.
But on the other hand, if we make too complicated model like on the right picture,
it will start describing noise in the train data that doesn't generalize the test data.
And this will lead to a decrease of model's quality.
This is called overfitting.
So, we want something in between underfitting and overfitting here.
And for the purpose of choosing the most suitable model,
we want to be able to evaluate our results.
Here, we need to make a remark,
that the meaning of overfitting in machine learning in general and
the meaning of overfitting competitions in particular are slightly different.
In general, we say that the model is overfitted if
its quality on the train set is better than on the test set.
But in competitions, we often say,
that the models are overfitted only in case when
quality on the test set will be worse than we have expected.
For example, if you train gradient boosting decision tree in
the competition is our area under a curve metric.
We sometimes can observe that the quality on
the training data is close to one while on the test data,
it could be less for example, near 0.9.
In general sense, the models overfitted here but while we get area
under curve was 0.9 on both validation and public/private test sets,
we will not say that it is overfitted in the context of a competition.
Let me illustrate this concept again in a bit different way.
So, lets say for the purpose of model evaluation,
we divided our data into two parts.
Train and validation parts.
Like we already did,
we will derive model's complexity from low to high and look at the models here.
Note, that usually, we understand
error or loss is something which is opposite to model's quality or score.
In the figure, the dependency looks pretty reasonable.
For two simple models,
we have underfitting which means higher on both train and validation.
For two complex models,
we have overfitting which means low error on train but again high error on validation.
In the middle, between them,
if the perfect model's complexity,
it has the lowest train on the validation data and
thus we expect it to have the lowest error on the unseen test data.
Note, that here the training error is always
better than the test error which implies overfitting in general sense,
but doesn't apply in the context of competitions.
Well done.
In this video, we define validation,
demonstrated its purpose, and interpreted validation
in terms of underfitting and overfitting.
So, once again, in general,
the validation helps us answer the question,
what will be the quality of our model on the unseeing data and help
us select the model which will be expected to get the best quality on that test data.
Usually, we are trying to avoid underfitting on the one side that is
we want our model to be expressive enough to capture the patterns in the data.
And we are trying to avoid overfitting on the other side,
and don't make too complex model,
because in that case,
we will start to capture noise or patterns that doesn't generalize to the test data.