Hi everyone. We are starting course about machine learning competitions.
In this course, you will learn a lot of tricks
and best practices about data science competitions.
Before we start to learn advanced techniques,
we need to understand the basics.
In this video, I will explain the main concept of
competitions and you will become familiar with competition mechanics.
A variety of machinery competition is very high.
In some, participants are asked to process texts.
In others, to classify picture or select the best advertising.
Despite the variety, all of these competitions are very similar in structure.
Usually, they consist of the same elements or
concepts which we will discuss in this video.
Let's start with a data.
Data is what the organizers give us as training material.
We will use it in order to produce our solution.
Data can be represented in a variety of formats.
SSV file with several columns , a text file,
an archive with pictures,
a database dump, a disabled code or even all together.
With the data, usually there is a description.
It's useful to read it in order to understand
what we'll work with and which feature can be extracted.
Here is an example from Kaggle.
From the top, we see several files with data,
and below, is their description.
Sometimes in addition to data issued by organizers,
we can use other data.
For example, in order to improve image classification model,
one may use a publicly available data set of images.
But this depends on a particular competition and you need to check the rules.
The next concept is a model.
This is exactly what we will build during the competition.
It's better to think about model not as one specific algorithm,
but something that transforms data into answers.
The model should have two main properties.
It should produce best possible prediction and be reproducible.
In fact, it can be very complicated and contain a lot of algorithms,
handcrafted features, use a variety of libraries as
this model of the winners of the Homesite competition shown on this slide.
It's large and includes many components.
But in the course, we will learn how to build such models.
To compare our model with the model of other participants,
we will send our predictions to the server or in other words, make the submission.
Usually, you're asked about predictions only.
Sources or models are not required.
And also there are some exceptions,
cool competitions, where participants submit their code.
In this course, we'll focus on
traditional challenges where a competitor submit only prediction outputs.
Often, I can not just provide a so-called sample submission.
An example of how the submission file should look like,
look at the sample submission from the Zillow competition.
In it is the first column.
We must specify the ID of the object and then specify our prediction for it.
This is typical format that is used in many competitions.
Now, we move to the next concept, evaluation function.
When you submit predictions,
you need to know how good is your model.
The quality of the model is defined by evaluation function.
In essence and simply the function,
the text prediction and correct answers and
returns a score characterizes the performance of the solution.
The simplest example of such a function is the accurate score.
This is just a rate of correct answers.
In general, there are a lot of such functions.
In our course, we will carefully consider some of them.
The description of the competition always indicates which evaluation function is used.
I strongly suggest you to pay attention to
this function because it is what we will try to optimize.
But often, we are not interested in the score itself.
We should only care about our relative performance in comparison to other competitors.
So we move to the last point we are considering, the leaderboard.
The leaderboard is the rate which provides you with
information about performance of all participating teams.
Most machine learning competition platforms keep your submission history,
but the leaderboard usually shows only your best score and position.
They cannot as that submission score,
reveal some information about data set.
And, in extreme cases,
one can obtain ground truth targets after sending a lot of submissions.
In order to handle this,
the set is divided into two parts, public and private.
This split is hidden from users and during the competition,
we see the score calculated only on public subset of the data.
The second part of data set is used for
private leaderboard which is revealed after the end of the competition.
Only this second part is used for final rating.
Therefore, a standard competition routine looks like that.
You as the competition, you analyze the data, improve model,
prepare submission, send it, see leaderboard score.
You repeat this action several times.
All this time, only public leaderboard is available.
By the end of the competition,
you should select submissions which will be used for final scoring.
Usually, you are allowed to select two final submissions.
Choose wisely. Sometimes public leaderboard scores might be misleading.
After the competition deadline,
public leaderboard is revealed,
and its used for the final rating and defining the winners.
That was a brief overview of competition mechanics.
Keep in mind that many concepts can be slightly different in a particular competition.
All details, for example,
where they can join into teams or use external data,
you will find in the rules.
Strongly suggest you to read the rules carefully before joining the competition.
Now, I want to say a few words about competition platforms.
Although Kaggle is the biggest and most famous one,
there is a number of smaller platforms or
even single-competition sites like KDD and VizDooM.
Although this list will change over time,
I believe you will find the competition which is most relevant and interesting for you.
Finally, I want to tell you about the reasons to participate in data science competition.
The main reason is that competition is a great opportunity for learning.
You communicate with other participants,
try new approaches and get a lot of experience.
Second reason is that competition often offer you
non-trivial problems and state-of-the-art approaches.
It allows you to broaden the horizons and
look at some everyday task from a different point of view.
It's also a great way to become recognizable,
get some kind of frame inside data science community and receive a nice job offer.
The last reason to participate is that you have a chance for winning some money.
It shouldn't be the main goal,
just a pleasant bonus.
In this video, we analyzed the basic concept of the competition,
talked about platforms and reasons for participation.
In the next video, we will talk about the difference between real life and competitions.