Since Google invented MapReduce, BigTable, distributed file systems.
It has moved on and is now using something called Dremel.
Recall that, in the early days of databases, the relational database was
used both for transaction processing. That is, inserting new records, as well
for answering complex queries. Storage was expensive.
It was too expensive to, for example, create a fresh copy of the entire data.
To, in a, in a better form, more suited for efficient query processing.
So, one was happy with the compromise of having a sim-, the one size fits all model
of the relational database. Over the years storage became very cheap,
and one started to move data into specific column oriented databases as well as have
analytical queries that touched all the data performed using MapReduce, where
large volumes data would be read and then equally large volumes freshly written
again and again as one performed more and more processing on them.
This was fine as long as you had terabytes, hundreds of terabytes of data,
even Google. But, once one started dealing with
petabytes of data, and wanted queries on such volumes.
One could not afford to produce a new petabyte of data every time when processed
the old petabyte. So, the challenge of storage being a
constraint once again, enters into the arena when one is dealing with very large
volumes. At the same time, writing extremely large
volumes is itself costly, and so by avoiding writing again and again, one does
introduce more efficiencies. So this is, is essentially what Dremel
does. Dremel today powers Google's BigQuery,
which is a service, which one can access over the web.
One can define extremely large tables. One can populate them through computations
or importing data from various sources and execute extremely fast queries that
process large volumes of data using the Dremel structure underneath.
There are two important innovations in Dremel, which was published only in 2010.
First, it uses a column-oriented storage. Just like a column-oriented database in,
in, in some sense. But for nested and possibly non-unique
fields. For example, you could have a document
which had a field A. Within that field A, one could have
another field B, which itself would have say, three, two different fields C and D
which actually contain values. So, the nested field A.B.C or A.B.D is
actually how you would access this data. Further in a particular record, there
could be multiple values for A.B.C. For example.
So, you could have multiple names, multiple IP addresses or whatever for this
particular nested field name. This is very common in web-oriented text,
text to unstructured data, not that common in structured relational data, for
example. But this is the kind of large petabyte
volume data that Google needs to process. So, the column where in to storage is
fairly unique in that each nested field is stored contiguously.
So, all the values for record one and record two for this nested field are
stored in, close together on disk and processed by leaf servers.
Similarly these, this nested field A.B.D is stored contiguously and so on.
So, first innovation that it's column orientated for nested and possibly
non-unique fields. The second innovation is that instead of
reading and writing data repeatedly like in MatReduce.
One assumes that the intermediate data that one produces is always much, much
less than the original data, which was quite obvious if you're dealing with
petabytes of data or you won't be producing more petabytes.
You'll be summarizing it in some form or selecting it or querying it, exactly as
you had in the traditional relational databases where you would query and get
small results from large results, from large data.
So, the second innovation is that there, there is a tree of query servers that pass
intermediate results from the root to the leaves and back.
And the intermediate servers essentially execute a complex query plan.
Very similar, in some respects to traditional sequel engines.
However, these operate at a different scale.
Sequel engines predominantly operated in memory.
These are operating in a distributed fashion across a tree of query servers,
and passing results back and forth across a network.
As a result, Google is able to demonstrate orders of magnitude better performance
than MapReduce when performing queries on petabytes of data.
Not only does it give more speed, but it also clearly saves storage as compared to
MapReduce. The underlying storage layer remains the
distributed GFS system. But the Dremel is now widely used in
Google and is available publicly using the BigQuery service.
There is some effort at creating an open source equivalent of Dremel.
It's in its infancy right now. It's under Apache.
It's called Drill. But beyond the name, I don't think they've
made too much progress so far. So, we can now summarize our picture of
how database technology has evolved over the years.
We started out with the relational row store, which was essentially a one size
fits all and still works fine for gigabytes of data.
Then we moved onto column-oriented data warehouse technologies, specifically
designed for all that queries. Which scaled up to terabytes of data, but
required us to move off of the relational row store into a data warehouse.
In parallel, the web side just created, distributed NoSQL databases which were a
mix of row and column stores which also allowed MapReduce pro-, processing for
bulk analysis and this scaled to tens of terabytes or sometimes even more volumes
of data. In parallel with this, we had in memory
databases emerging in the past few years, which now can do what the
one-size-fits-all relational row stores did.
Again, on gigabytes of data, but with an order of magnitude more performance.
And on the large scale processing for petabytes of data, Google has evolved
Dremel, which again, is a one-size-fits-all model for petabytes of
data. So we have three models today.
We have Dremel which only Google uses. We have In-Memory which is fine for doing
OLAP on reasonably small databases. And for intermediate processing to do
things like computing classifiers on large terabytes of data.
Distributed noSQL's here's the preferred choice.
At the same time, when you have terabytes of data and you want to do all that
queries very fast using SQL, then there still remains a place for the column store
data warehouses, typically the special purpose appliances like Netezza which use
panel computing and column storage also have a place.
This place where the column store warehouse is might evolve to a Dremel-like
architecture in the future once we actually have.
A publicly-available version of Dremel. This is a space to really watch carefully
and that's what big data technology is looking forward to in the next three