Since we already know the main strategies for validation,
we can move to more concrete examples.
Let's imagine, we're solving a competition with a time series prediction, namely,
we are to predict a number of customers for a shop for which they're due in next month.
How should we divide the data into train and validation here?
Basically, we have two possibilities.
Having data frame first, we can take random rows in validation and second,
we can make a time-based split,
take everything before some date as a train and everything out there as a validation.
Let's plan these two options next.
Now, when you think about features you need to generate and the model you need to train,
how complicated these two cases are?
In the first block,
we can just interpret between the previous and the next value to get our predictions.
Very easy, but wait.
Do we really have future information about the number of customers in the real world?
Well, probably not. But does this mean that this validation is useless?
Again, it doesn't.
What it does mean really that if we make
train validation split different from train/test split,
then we are going to create a useless model.
And here, we get to the main rule of making a reliable validation.
We should, if possible,
set up validation to mimic train/test split,
but that's a little later.
Let's go back to our example.
On the second picture,
for most of test point,
we have neither the next value nor the previous one.
Now, let's imagine we have a pool of different models trained on different features,
and we selected the best model for each type of validation.
Now, the question, will these models differ?
And if they will, how significantly?
Well, it is certain that if you want to predict what will happen a few points later,
then the model which favor features like previous
and next target values will perform poorly.
It happens because in this case,
we just don't have such observations for the test data.
But we have to give the model something in the feature value,
and it probably will be not numbers or missing values.
How much experience that model have with these type of situations?
Not much. The model just won't expect that and quality will suffer.
Now, let's remember the second case.
Actually, here we need to rely more on the time trend.
And so, the features,
which is the model really we need here,
are more like what was the trend in the last couple of months or weeks?
So, that shows that the model selected as the best model for
the first type of validation will perform poorly for the second type of validation.
On the opposite, the best model for the second type of
validation was trained to predict many points ahead,
and it will not use adjacent target values.
So, to conclude this comparison,
these models indeed differ significantly,
including the fact that most useful features for one model are useless for another.
But, the generated features are not the only problem here.
Consider that actual train/test split is time-based,
here is the question.
If we carefully generate features that are drawing attention to time-based patterns,
we'll get a reliable validation with a random-based split.
Let me say this again in another words.
If we'll create features which are useful for
a time-based split and are useless for a random split,
will be correct to use a random split to select the model?
It's a tough question.
Let's take a moment and think about it.
Okay, now let's answer this.
Consider the case when target falls a linear train.
In the first block,
we see the exact case of randomly chosen validation.
In the second, we see the same time-based split as we consider before.
first, let's notice that in general,
model predictions will be close to targets mean value calculated using train data.
So in the first block,
if the validation points will be closer to this mean value compared to test points,
we'll get a better score in validation than on test.
But in the second case,
the validation points are roughly as far as the test points from target mean value.
And so, in the second case,
validation score will be more similar to the test score.
Great, as we just found out,
in the case of incorrect validation,
not only features, but the value target can lead to unrealistic estimation of the score.
Now, that example was quite similar to what you may
encounter while solving real competitions.
Numerous competitions use time-based split namely: the Rossmann Store Sales competition,
the Grupo Bimbo Inventory Demand competition and others.
So, to quickly summarize this valuable example we just have discussed,
different splitting strategies can differ significantly,
namely: in generated features,
in the way the model will rely on that features,
and in some kind of target leak.
That means, to be able to find smart ideas for
feature generation and to consistently improve our model,
we absolutely want to identify train/test split made by organizers,
including the competition, and reproduce it.
Let's now categorize most of these splitting strategies and competitions,
and discuss examples for them.
Most splits can be united into three categories: a random split,
a time-based split and the id-based split.
Let's start with the most basic one, the random split.
Let's start, the most common way of making
a train/test split is to split data randomly by rows.
This usually means that the rows are independent of each other.
For example, we have a test of predicting if a client will pay off alone.
Each row represents a person,
and these rows are fairly independent of each other.
Now, let's consider that there is some dependency, for example,
within family members or people which work in the same company.
If a husband can pay a credit probably,
his wife can do it too.
That means if by some misfortune,
a husband will will present in the train data and his wife will present in the test data.
We probably can explore this and devise a special feature for that case.
For in such possibilities,
and realizing that kind of features is really interesting.
More in this case and others I will mention here,
comes in the next lesson of our course.
So again, that was a random split.
The second method is a time-based split.
We already discussed the unit example of the split in the beginning of this video.
In that case, we generally have everything before a particular date as a training data,
and the rating after date as a test data.
This can be a signal to use special approach to feature generation,
especially to make useful features based on the target.
For example, if we are to predict a number of
customers for the shop for each day in the next week,
we can came up with something like
the number of customers for the same day in the previous week,
or the average number of customers for the past month.
As I mentioned before,
this split is widespread enough.
It was used in
a Rossmann store sales competition and in the Grupo Bimbo inventory demand competition,
and in other's competitions.
A special case of validation for the time-based split is a moving window validation.
In the previous example,
we can move the date which divides train and validation.
Successively using week after week as a validation set,
just like on this picture.
Now, having dealt with the random and the time-based splits,
let's discuss the ID-based split.
ID can be a unique identifier of user,
shop, or any other entity.
For example, let's imagine we have to solve a task
of music recommendations for completely new users.
That means, we have different sets of users in train and test.
If so, we probably can make a conclusion that features based on user's history,
for example, how many songs user listened in the last week,
will not help for completely new users.
As an example of ID-based split,
I want to tell you a bit about the Caterpillar to pricing competition.
In that competition, train/test split was done on some category ID, namely, tube ID.
There is an interesting case when we should employ the ID-based split,
but IDs are hidden from us.
Here, I want to mention two examples of competitions with hidden ID-based split.
These include Intel and MumbaiODT Cervical Cancer Screening competition,
and The Nature Conservancy fisheries monitoring competition.
In the first competition,
we had to classify patients into three classes,
and for each patient,
we had several photos.
Indeed, photos of one patient belong to the same class.
Again, sets of patients from train and test did not overlap.
And we should also ensure these in the training regulations split.
As another example, in The Nature Conservancy fisheries monitoring competition,
there were photos of fish from several different fishing boats.
Again, fishing boats and train and test did not overlap.
So one could easily overfit if you would ignore risk and make a random-based split.
Because the IDs were not given,
competitors had to derive these IDs by themselves.
In both these competitions,
it could be done by clustering pictures.
The easiest case was when pictures were taken just one after another,
so the images were quite similar.
You can find more details of such clustering in the kernels of these competitions.
Now, having in these two main standalone methods,
we also need to know that they sometimes may be combined.
For example, if we have a task of predicting sales in a shop,
we can choose a split in date for each shop independently,
instead of using one date for every shop in the data.
Or another example, if we have search queries from multiple users,
is using several search engines,
we can split the data by a combination of user ID and search engine ID.
Examples of competitions with combined splits include
the Western Australia Rental Prices competition by Deloitte
and their qualification phase of data science game 2017.
In the first competition,
train/test was split by a single date,
but the public/private split was made by different dates for different geographic areas.
In the second competition,
participants had to predict whether
a user of online music service will listen to the song.
The train/test split was made in the following way.
For each user, the last song he listened to was placed in the test set,
while all other songs were placed in the train set.
Fine. These were the main splitting strategies employed in the competitions.
Again, the main idea I want you to take away from this lesson is that
your validation should always mimic train/test split made by organizers.
It could be something non-trivial.
For example, in the Home Depot Product Search Relevance competition,
participants were asked to estimate search relevancy.
In general, data consisted of search terms and search results for those terms,
but test set contained completely new search terms.
So, we couldn't use either a random split or a search term-based split for validation.
First split favored more complicated models,
which led to overfitting while second split, conversely, to underfitting.
So, in order to select optimal models,
it was crucial to mimic the ratio of new search terms from train/test split.
Great. This is it.
We just demonstrated major data splitting strategies employed in competitions.
Random split, time-based split,
ID-based split, and their combinations.
This will help us build reliable validation,
make a useful decisions about feature generation,
and in the end, select models which will perform best on the test data.
As the main point of this video,
remember the general rule of making a reliable validation.
Set up your validation to mimic the train/test split of the competition.