Welcome to week three.
As promised, we'll now begin our coverage
of big data technology starting this week
with Map Reduce and a programming
assignment based on this new way of doing
parallel computing.
And next week, we'll cover the big data
platforms.
Such as distributed file systems, new
database technologies, and most
importantly, where all this is headed with
a glimpse of some research topics
currently being explored.
Obviously, the first question that comes
to mind is, with 30 to 40 years of
database work behind us, why did we
suddenly need to invent a new database
technology?
All this was done in the web and they
clearly found some reasons why the
traditional technology didn't work for
them.
Basically, there are four reasons which
we'll basically summarize right now, but
we'll get into the details next week.
First of all, the traditional technologies
were not fault tolerant at scale.
They would handle maybe dozens or hundreds
of processing machines, but not thousands
or tens of thousands, or even millions,
which are common in web platforms like
Google and Facebook.
Next, traditional technologies were not
really good at handling text, and
certainly not handling video and images.
Third, and this is a technical point.
Large data volumes needed to be kept
online and available all the time, which
was simply not possible in the traditional
way of doing things where old data would
usually be archived onto tape or some
other storage which didn't eat up the
online space.
And most importantly, parallel computing
was really an add on feature bolted onto
traditional database technologies only in
the'90s.
Whereas, the scale of parallelism that the
web required simply didn't work with such
add-on technology.
As a result, traditional relational
database technology simply could not
scale.
And it was also not suited for the deep
analytics tasks which were computing
intensive that all the web companies
needed to perform, primarily, in order to
target advertisements better as we have
seen the past week.
What is resulted is a bunch of new
technologies coming out of the web world,
which not only perform at a higher scale
than traditional ones.
But because they were built using
commodity hardware and many of them are
now open source, they present a price
performance challenge as compared to the
new technologies.
So, it's not that big data technology
should be used only if, if you have big
data or huge computational requirements.
The fact is, that some of these new
technology is just cheaper and faster than
the old technology.
Now, the main innovations in big data
technology are the Map Reduce programming
paradigm which we'll study this week, and
the distributed file systems in data bases
which we'll come to next week.
However, the main message is a little
different.
It's not just the technology.
As we shall see soon, a different approach
to data processing problems is required.
If one tries to use new technology with an
old mindset, one can still get into the
same kinds of difficulties.
There are some important caveats, though.
The new technology is still maturing.
Many database innovations which have come
up over the past 40 years remain unique to
the traditional relational database stack.
Varieties of indexing, very complex query
optimizations, storage optimizations.
All of these are been rediscovered and
reinvented for big data.
We'll come to some of these things next
week.
But for the moment, let's plunge into
hollow computing.