[MUSIC]
Hi everyone, this week is about sequence to sequence tasks.
We have a lot of them in NLP, but
one obvious example would be machine translation.
So you have a sequence of words in one language as an input, and
you want to produce a sequence of words in some other language as an output.
Now, you can think about some other examples.
For example, summarization is also a sequence to sequence task and
you can think about it as machine translation but for
one language, monolingual machine translation.
Well we will cover these examples in the end of the week but now let us start with
statistical machine translation, and neural machine translation.
We will see that actually there are some techniques,
that are super similar in both these approaches.
For example, we will see alignments,
word alignments that we need in statistical machine translation.
And then, we will see that we have attention mechanism in neural networks
that kind of has similar meaning in these tasks.
Okay, so let us begin, and I think there is no need to tell you that
machine translation is important, we just know that.
So I would better start with two other questions.
Two questions that actually we skip a lot in our course and
in some other courses but these are two very important questions to speak about.
So one question is data and another question is evaluation.
When you get some real task in your life, some NLP task usually
this is not a model that is plane, this is usually data and evaluation.
So you can have a fancy neuro-architecture, but
if you do not have good data and if you haven't settled
down how to do evaluation procedure, you're not going to have good results.
So first data, well what kind of data do we need for machine translation?
We need some parallel corpora, so we need some text in one language and
we need its translation to another language.
Where does that come from, so what sources can you think of?
Well, one of your source well maybe not so obvious but
one very good source, is European Parliament proceedings.
So you have there some texts in several languages, maybe 20 languages and
very exact translations of one in the same statements.
And this is nice, so you can use that, some other domain would be movies.
So you have subtitles that are translated in many languages this is nice.
Something which is not that useful, but still useful,
would be books translations or Wikipedia articles.
So for example, for Wikipedia you can not guarantee that you
have the same text for two languages.
But you can have something similar, for example some vague translations or
which are to the same topic at least.
So we call this corpora comparable but not parallel.
The OPUS website has the nice overview of many sources so please check it out.
But I want to discuss something which is not nice, some problems with the data.
Actually, we have lots of problems for any data that we have, and
what kind of problems happen for machine translation?
Well, first, usually the data comes from some specific domain.
So imagine you have movie subtitles and you want to train a system for
scientific papers translations.
It's not going to work, right, so you need to have some close domain.
Or you need to know how to transfer your knowledge from one domain
to another domain, this is something to think about.
Now, you can have some decent amount of data for some language pairs like English
and French, or English and German, but probably for some rare language pairs,
you have really not a lot of data, and that's a huge problem.
Also you can have noisy and not enough data, and it can be not aligned well.
By alignment I mean, you need to know the correspondence between the sentences,
or even better the correspondence between the words and the sentences.
And this is luxury, so usually you do not have that, at least for
a huge amount of data.
Okay, now I think it's clear about the data, so the second thing, evaluation.
Well you can say that we have some parallel data.
So why don't we just split it to train and test and have our test set
to compare correct translations and those that are produced by our system.
But well, how do we know that the translation
is wrong just because it doesn't occur in your reference?
You know that the language is so
relative so every translator would do some different translations.
It means that if your system produce something different it doesn't mean yet
that it is wrong.
So well there is no nice answer for this question, I mean this is a problem, yes.
One thing that you can do is to have multiple references so you can have,
let's say five references and compare your system output to all of them.
And the other thing is you should be very careful how do you compare it.
So definitely you shouldn't do just exact match,
right you should do something more intelligent.
And I'm going to show you BLUE score which is known to be very popular measure
in machine translation that try somehow to softly measure whether your
system output is somehow similar to the reference translation.
Okay, let me show you an example.
So you have some reference translation and you have the output of your system and
you try to compare them.
Well, you remember that we have this nice tool which is called engrams.
So you can compute some unigrams and bigrams and trigrams.
Do you have any idea how to use that here?
Well, first we can try to compute some precision, what does it mean?
You look into your system output, and here you have six words,
six unigrams and compute how many of them actually occur in the reference.
So the unigram precision core will be 4 out of 6.
Now, tell me what would be bigram score here.
Well, the bigram score will be 3 out of 5 because you have
5 bigrams in your system output and only 3 of them was sent sent on and
on Tuesday occurred in the reference.
Now you can proceed and you can compute 3-grams score and
4-grams score, so that's good.
Maybe we can just average them and have some measure.
Well we could, but there is one problem here,
well imagine that the system tries to be super precise.
Then it is good for system to output super short sentences, right?
So if I'm sure that this union gram should occur,
I will just output this and I will not output more.
So just to punish into penalty the model, we can have some brevity score.
This brevity penalty says that we
divide the length of the output by the length of the reference.
And then the system outputs two short sentences, we will get to know that.
Now how do we compute the BLEU score out of these values?
Like this so we have some average so this root is the average
of our union gram, bigram, 3-gram, and 4-gram's course.
And then we multiply this average by the brevity.
Okay, now let us speak about how the system actually works.
So this is kind of a mandatory slide on machine translation,
because kind of any tutorial on machine translation has this.
So I decided not to be an exception and show you that.
So the idea is like that, we have some source sentence and
we want to translate it to get some target sentence.
Now the first thing that we can do is just direct transfer.
So we can translate this source sentence word by word and get the target sentence.
But well, maybe it's not super good, right?
So if you have ever studied some foreign language, you know that just by dictionary
translations of every word, you usually do not get nice coherent translation.
So probably we would better go into some synthetic level.
So we do syntax analysis, and then we do the transfer and
then we generate the target sentence by knowing how it
should look like on on the syntactic level.
Even better, we could try to go to semantic layer, so that first we analyze
the source sentence and understand some meanings of some parts of the sentence.
We somehow transfer these meanings to in our language and
then we generate some good syntactic structures with good meaning.
And our dream, like the best things as we could ever think of,
would be having some interlingual.
So by interlingual, we mean some n ice representation of the whole
source sentence that is enough to generate the whole target sentence.
Actually it is still a dream, so it is still a dream of the translators
to have that kind of system because it sounds so appealing.
But neural translation systems somehow have mechanisms that
resembles that and I will show you that in a couple of slides.
Okay, so for now I want to show you some brief history of the area.
And like any other area,
machine translation has some bright and dark periods.
So in 1954 there were great expectations, so there was IBM experiments where
they translated 60 sentences from Russian to English.
And they said, that's easy we can solve the machine translation
task completely in just three or five years.
So they tried to work on that and they worked a lot, and
after many years they concluded that actually it's not that easy.
And they said, well, machine translation is too expensive and
we should not do automatic machine translation system.
We should better focus on just some tools that help human
translators to provide good quality translations.
So you know these great expectations and
then the disappointment made the area silent for
a while, but then in 1988 IBM researchers
proposed word-based machine translation systems.
These machine translation systems were rather simple, so
we will cover them, kind of in this video and in the next video, but
these systems were kind of the first working system for machine translation.
So this was nice and then the next important step was phrase based machine
translations system that were proposed by Philip Koehn in 2003.
And this is what probably people mean by statistical machine translation now.
You definitely know Google Translate, right?
But maybe you haven't heard about Moses.
So Moses is the system that allows a researchers to build their own
machine translation systems.
So it allows to train your models and to compare them, so
this is a very nice tool for researchers and it was made available in 2007.
Now, with an extent,
obviously very important step here is neural machine translation.
It is amazing how fast the neural machine translation systems
could go from research papers to production.
Usually we have such a big gap between these two things.
But in this case there were just two or three years so
it is amazing that those ideas that were proposed could be implemented and
just launched in many companies in 2016 so we have neutral machine translations now.
You might be wondering what is WMT there, it is the workshop on machine translation,
which is kind of the annual competition, the annual event and shared tasks.
Which means that you can compare your systems there, and it is a very nice
venue to compare different systems by different researchers and companies.
And to see what are the traits of machine translations.
And it happens every year, so usually people who do research in this
area keep eye on this and this is very nice thing.
This is the slide about intralingual that I promised to show you.
So this is how Google neural machine translation works, and
there was actually lots of hype around it maybe even too much.
But still, so the idea is that you train some system or some pair of languages.
For example on English to Japanese and Japanese to English and
English to Korean and some other pair, you train some encoder, decoder architecture.
It means that you have some encoder that encodes your sentence to
some hidden representation.
And then you have decoder that takes that hidden representation and
decodes it to the target sentence.
Now the nice thing is, that if you just take your encoder, let's say for
Japanese and decoder for Korean and you just take them.
Somehow it works nicely even though the system has never seen
Japanese to Korean translations.
You see so this is zero-shot translation you have never seen Japanese to Korean,
but just by building nice encoder and
nice decoder, you can stack them and get this path.
So it seems like this hidden representation that you have,
is kind of universal for any language pair.
Well, it is not completely true but at least it is very promising result.
[MUSIC]