Hi everyone. In this section,
we will talk about a very sensitive topic data leakage or more simply, leaks.
We'll define leakage in a very general sense as
an unexpected information in the data that
allows us to make unrealistically good predictions.
For the time being,
you may have think of it as of directly or
indirectly adding ground truths into the test data.
Data leaks are very, very bad.
They are completely unusable in real world.
They usually provide way too much signal and thus make competitions lose its main point,
and quickly turn them into a leak hunt increase.
People often are very sensitive about this matter.
They tend to overreact.
That's completely understandable.
After spending a lot of time on solving the problem,
a sudden data leak may render all of that useless.
It is not a pleasant position to be in.
I cannot force you to turn the blind eye but keep in mind,
there is no ill intent whatsoever.
Data leaks are the result of unintentional errors, accidents.
Even if you find yourself in
a competition with an unexpected data leak close to the deadline,
please be more tolerant.
The question of whether to exploit the data leak
or not is exclusive to machine learning competitions.
In real world, the answer is obviously a no, nothing to discuss.
But in a competition,
the ultimate goal is to get a higher leaderboard position.
And if you truly pursue that goal,
then exploit the leak in every way possible.
Further in this section,
I will show you the main types of data leaks
that could appear during solving a machine learning problem.
Also focus on a competition specific leak exploitation technique leaderboard probing.
Finally, you will find special videos
dedicated to the most interesting and non-trivial data leaks.
I will start with the most typical data leaks that may occur in almost every problem.
Time series is our first target.
Typically, future picking.
It is common sense not to pick into the future like,
can we use stock market's price from day after tomorrow to predict price for tomorrow?
Of course not. However, direct usage of
future information in incorrect time splits still exist.
When you enter a time serious competition at first,
check train, public, and private splits.
If even one of them is not on time,
then you found a data leak.
In such case, unrealistic features like prices next week will be the most important.
But even when split by time,
data still contains information about future.
We still can access the rows from the test set.
We can have future user history in CTR task,
some fundamental indicators in stock market predictions tasks, and so on.
There are only two ways to eliminate the possibility of data leakage.
It's called competitions, where one can not access
rows from future or a test set with no features at all, only IDs.
For example, just the number and instrument ID in stock market prediction,
so participants create features based on past and join them themselves.
Now, let's discuss something more unusual.
Those types of data leaks are much harder to find.
We often have more than just train and test files.
For example, a lot of images or text in archive.
In such case, we can't access some meta information,
file creation date, image resolution etcetera.
It turns out that this meta information may be connected to target variable.
Imagine classic cats versus dogs classification.
What if cat pictures were taken before dog?
Or taken with a different camera?
Because of that, a good practice from organizers is to erase the meta data,
resize the pictures, and change creation date.
Unfortunately, sometimes we will forget about it.
A good example is Truly Native competition,
where one could get nearly perfect scores using just the dates from zip archives.
Another type of leakage could be found in IDs.
IDs are unique identifiers of every row usually used for convenience.
It makes no sense to include them into the model.
It is assumed that they are automatically generated.
In reality, that's not always true.
ID may be a hash of something,
probably not intended for disclosure.
It may contain traces of information connected to target variable.
It was a case in Caterpillar competition.
A link ID as a feature slightly improve the result.
So I advise you to pay close attention to IDs and
always check whether they are useful or not.
Next is row order.
In trivial case, data may be shuffled by target variable.
Sometimes simply adding row number or relative number,
suddenly improves this course.
Like, in Telstra Network Disruptions competition.
It's also possible to find something way more interesting
like in TalkingData Mobile User Demographics competition.
There was some kind of row duplication,
rows next to each other usually have the same label.
This is it with a regular type of leaks.
To sum things up, in this video,
we embrace the concept of data leak and cover data leaks from future picking,
meta data, IDs, and row order.