Hello everyone. This is Marios.
Today I would like to show you the Pipeline or like the approach I have used to
tackle more than 100 machine learning competitions
in cargo and obviously has helped me to do quite well.
Before I start, let me state that I'm not claiming this is the best pipeline out there,
is just the one I use.
You might find some parts of it useful.
So roughly, the Pipeline is,
as you see it on the screen,
here this is a summary and we will go through it in more detail later on.
But briefly, I spend the first day in order to understand
the problem and make the necessary preparations in order to deal with this.
Then, maybe one, two days in order to understand a bit about the data,
what are my features,
what I have available,
trying to understand other dynamics about the data,
which will lead me to define
a good cross validation strategy and we will see later why this is important.
And then, once I have specified the cross validation strategy,
I will spend all the days until
3-4 days before the end of the competition and I will keep iterating,
doing different feature engineering and applying different machine bearing models.
Now, something that I need to to highlight is that,
when I start this process I do it myself,
shut from the outside world.
So, I close my ears,
and I just focus on how I would tackle this problem.
That's because I don't want to get affected by what the others are doing.
Because I might be able to find something that others will not.
I mean, I might take a completely different approach and this always leads me to gain,
when I then combine with the rest of the people.
For example, through merges or when I use other people's kernels.
So, I think this is important,
because it gives you the chance to create an intuitive approach about the data,
and then also leverage the fact that
other people have different approaches and you will get more diverse results.
And in the last 3 to 4 days,
I would start exploring different ways to combine all the models of field,
in order to get the best results.
Now, if people have seen me in competitions,
you should know that you might have noticed that in the last 3-2 days I do
a rapid jump in the little box and that's exactly because I leave assembling at the end.
I normally don't do it.
I have confidence that it will work
and I spend more time in feature engineering and modeling,
up until this point.
So, let's take all these steps one by one.
Initially I try to understand the problem.
First of all, what type of problem it is.
Is it image classification,
so try to find what object is presented on an image.
This is sound classification,
like which type of bird appears in a sound file.
Is it text classification?
Like who has written the specific text,
or what this text is about.
Is it an optimization problem,
like giving some constraints how can I get from point A to point B etc.
Is it a tabular dataset,
so that's like data which you can represent in Excel for example,
with rows and columns,
with various types of features, categorical or numerical.
Is it time series problem?
Is time important?
All these questions are very very
important and that's why I look at the dataset and I try to understand,
because it defines in many ways what resources I would need,
where do I need to look and what kind of hardware and software I will need.
Also, I do this sort of preparation along with controlling the volume of the data.
How much is it.
Because again, this will define how I need to,
what preparations I need to do in order to solve this problem.
Once I understand what type of problem it is,
then I need to reserve hardware to solve this.
So, in many cases I can escape without using GPUs,
so just a few CPUs would do the trick.
But in problems like image classification of sound,
then generally anywhere you would need to use deep learning.
You definitely need to invest a lot in CPU,
RAM and disk space.
So, that's why this screening is important.
It will make me understand what type of machine I will need in order to
solve this and whether I have this processing power at this point in order to solve this.
Once this has been specified and I know how many CPUs,
GPUs, RAM and disk space I'm going to need,
then I need to prepare the software.
So, different software is suited for different types of problems.
Keras and TensorFlow is obviously really good for when solving
an image classification or sound classification and text problems
that you can pretty much use it in any other problem as well.
Then you most probably if you use Python,
you need scikit learn and XGBoost, Lighgbm.
This is the pinnacle of machine learning right now.
And how do I set this up?
Normally I create either an anaconda environment or a virtual environment in general,
and how a different one for its competition,
because it's easy to set this up.
So, you just set this up,
you download the necessary packages you need,
and then you're good to go.
This is a good way,
a clean way to keep everything tidy and
to really know what you used and what you find useful in the particular competitions.
It's also a good validation for later on,
when we will have to do this again,
to find an environment that has worked well for this type of
problem and possibly reuse it.
Another question I ask at this point is what the metric I'm being tested on.
Again, is it a regression program,
is it a classification program,
it is root mean square error,
it is mean absolute error.
I ask these questions because I try to find if there's any similar competition with
similar type of data that I may have dealt with in the past,
because this will make this preparation much much better,
because I'll go backwards,
find what I had used in the past and capitalize on it.
So, reuse it, improve it,
or even if I don't have something myself,
I can just find other similar competitions or explanations of these type of
problem from the web and try to see what people
had used in order to integrate it to my approaches.
So, this is what it means to understand the problem at this point.
It's more about doing the screen search,
this screening in order to understand what type of preparation I need to do,
and actually do this preparation,
in order to be able to solve this problem competitively,
in terms of hardware,
software and other resources,
past resources in dealing with these types of problems.
Then I spent the next one or two days to do some exploratory data analysis.
The first thing that I do is I see all my features,
assuming a tabular data set,
in the training and the test data and to see how consistent they are.
I tend to plot distributions and to try to find if there are any discrepancies.
So is this variable in the training data set very
different than the same variable in the task set?
Because if there are discrepancies or differences,
this is something I have to deal with.
Maybe I need to remove these variables or scale them in a specific way.
In any case, big discrepancies can cause problems to the model,
so that's why I spend some time here and do
some plotting in order to detect these differences.
The other thing that I do is I tend to
plot features versus the target variable and possibly versus time,
if time is available.
And again, this tells me to understand the effect of time,
how important is time or date in this data set.
And at the same time it helps me to understand which are like the most predictive inputs,
the most predictive variables.
This is important because it generally gives me intuition about the problem.
How exactly this helps me is not always clear.
Sometimes it may help me define a gross validation strategy or
help me create some really good features but in general,
this kind of knowledge really helps to understand the problem.
I tend to create cross tabs for example with
the categorical variables and the target variable and also creates
unpredictability metrics like information value and you see chi square for example,
in order to see what's useful and whether I can make hypothesis about the data,
whether I understand the data and how they relate with the target variable.
The more understanding I create at this point,
most probably will lead to better features for better models applied on this data.
Also while I do this,
I like to bin numerical features into
bands in order to understand if there nonlinear R.A.T's.
When I say nonlinear R.A.T's,
whether the value of a feature is low,
target variable is high,
then as the value increases the target variable decreases as well.
So whether there are strange relationships trends,
patterns, or correlations between features and the target variable,
in order to see how best to handle this later on and get
an intuition about which type of problems or which type of models would work better.
Once I have understood the data,
to some extent, then it's time for me to define a cross validation strategy.
I think this is a really important step
and there have been competitions where people were able to
win just because they were able to find
the best way to validate or to create a good cross validation strategy.
And by cross validation strategy,
I mean to create a validation approach that best resembles what you're being tested on.
If you manage to create this internally then you can create
many different models and create many different features and anything you do,
you can have the confidence that is working or it's not working,
if you've managed to build the cross validation strategy in a consistent way
with what you're being tested on so consistency is the key word here.
The first thing I ask is,
"Is time important in this data?"
So do I have a feature which is called date or time?
If this is important then I need to switch to a time-based validation.
Always have past data predicting future data,
and even the intervals,
they need to be similar with the test data.
So if the test data is three months in the future,
I need to build my training and validation to account for this time interval.
So my validation data always need to be
three months in the future and compared to the training data.
You need to be consistent in order to have
the most consistent results with what you are been tested on.
The other thing that I ask is,
"Are there different entities between the train and the test data?"
Imagine if you have different customers in
the training data and different in the test data.
Ideally, you need to formulate
your cross validation strategy so that in the validation data,
you always have different customers running in
training data otherwise you are not really testing in a fair way.
Your validation method would not be consistent with the test data.
Obviously, if you know a customer and you try to predict it, him or her,
why you have that customer in your training data,
this is a biased prediction when compared to the test data,
that you don't have this information available.
And this is the type of questions you need to ask yourself when you are at this point,
"Am I making a validation which is really consistent with what am I being tested on?"
The other thing that is often the case is
that the training and the test data are completely random.
I'm sorry, I just shortened my data and I took a random part, put it on training,
the other for test so in that case,
is any random type of cross validation could help for example,
just do a random K-fold.
There are cases where you may have to use a combination of all the above so you
have strong temporal elements at the same time you have different entities,
so different customers to predict for past and future and at the same time,
there is a random element too.
You might need to incorporate all of them do make a good strategy.
What I do is I often start with
a random validation and just see how it fares with the test leader board,
and see how consistent the result is with what they have internally,
and see if improvements in my validation lead to improvements to the leader board.
If that doesn't happen,
I make a deeper investigation and try to understand why.
It may be that the time element is very strong and I need to
take it into account or there are different entities between the train and test data.
These kinds of questions in order to formulate a better validation strategy.
Once the validation strategy has been defined,
now I start creating many different features.
I'm sorry for bombarding you with loads of
information in one slide but I wanted this to be standalone.
It says give you the different type of
future engineering you can use in different types of problems,
and also suggestions for the competition to look
up which was quite representative of this time.
But you can ignore these for now. Look at it later.
The main point is different problem requires
different feature engineering and I put everything when I say feature engineering.
I put the day data cleaning and preparation as well,
how you handle missing values,
and the features you generate out of this.
The thing is, every problem has its own corpus of
different techniques you use to derive or create new features.
It's not easy to know everything because sometimes it's too much,
I don't remember it myself so what I tend to do
is go back to similar competitions and see what
people are using or what people have used in the past and I incorporate into my code.
If I have dealt with this or a similar problem
in the past then I look at my code to see what I had done in the past,
but still looking for ways to improve this.
I think that's the best way to be able to handle any problem.
The good thing is that a lot of the feature engineering can be automated.
You probably have already seen that but,
as long as your cross validation strategy is consistent with the test data and reliable,
then you can potentially try all sorts of
transformations and see how they work in your validation environment.
If they work well, you can be confident that this type of
feature engineering is useful and use it for further modeling.
If not, you discard and try something else.
Also the combinations of what you can do in terms of
feature engineering can be quite vast in different types of
problems so obviously time is a factor here, and scalability too.
You need to be able to use your resources well in order to be able to search
as much as you can in order to get the best outcome.
This is what I do. Normally if I have more time
to do this feature engineering in a competition,
I tend to do better because I explore more things.