One of the first no sequel databases was Google's Big Table.
And its, Hadoop equivalent, which is called HBase.
Big Table is, in many ways, probably the origin of the term big data.
This is one of the earliest, papers describing this was almost three or four
years ago and it is Google's implementation on top of the distributed
GFS file system, which led to the development of many no sequel databases in
the first place. A big table, table is distributed across
many different servers by row first. So that table is broken up into many
tablets, each containing multiple rows. The tablets are called regions by the way
in H base tablet is the GFS term. Each tablet in turn, is broken up into
column families, each containing a set of columns from among those in that row.
Each column family which contains. A particular set of columns spanning
multiple rows, is stored in a chunk of the distributed file system.
The GFS or HGFS. Each, such chunk is served by, a tablet
server which takes care of three separate replicas as they are maintained in GFS as
we have seen before, and the tablet servers together, form the entire big
table. Of course in order to access any
particular record, we need to know which tablet server that particular record and
column that we are looking for falls in. And for that purpose there is a metadata
table which keeps track of which tablet server a particular row, lies in.
The metadata table itself is another big table and is therefore cons-, comprises of
tablets or regions which are maintained on, a separate set of tablets servers.
One particular tablet is the root tablet. So all our searches start from there.
So to look for a particular row, and column.
One looks up the tablet, root tablet. Which will tell us which child's tablet,
among the metadata tablets contain the information about that particular row
column. The child tablet will in turn tell us
which tablet server to look for, and there we can pick up that chunk where the
required row and column family is actually stored.
Let's take a look at date, what data in a big table looks like, to get a feel for
how it works. The row is indexed by some key, for
example a transaction ID, when one is storing sales transactions.
Different column families can have multiple columns within them.
For example, we can have the location column family which contains a whole bunch
of columns with to deal with location, and all those are stored together in single
chunks. The sale column family might have other
set of columns and similarly for products and many other column families would be
there. Each of them is stored separately in
different chunks servers. An important point about big table is
that, the num, while the number of column families for a big table is fixed when you
create it. The number of columns within the column
family can vary. So you could, dynamically add new columns
to a particular big table column family. Further, each parti, each column family
for a particular row can have multiple entries.
For example, you could have the region being the U.S.
East Coast as well as the U.S. Northeast, and that's perfectly fine
unlike a relational database where a particular row-column combination can have
only one value. Additionally, each of these different
values for a particular row. Column combination can be time stamped.
So that for example, the, the location for this transaction might be the US East
Coast today, but tomorrow one is free to change it by making it U.S.
Northeast. But that's not done by updating the value
US East Coast, but by inserting a new value with a new time stamp.
So that one can always look up whichever version of the region, one wants to look
for depending on, which time stamps one are, one is searching for.
Someone can keep essential multiple snapshots of one's categorization of data
together in the same big table, which is a, a tredmendous advantage compared to a
traditional relational database. Because Big Table and HBase rely on the
underlying distributed file system GFS or HDFS respectively.
They also inherit some of the properties of these systems.
In particular, large parallel reads and inserts are efficiently supported even
simultaneously on the same table, unlike a traditional relational database.
Similarly, reading all the rows for a small number of column families in a large
table. Such as for an aggrega, aggregation query,
like summing them all up. Is efficient in the manner similar to the
column oriented databases. Of course one of the down sides of the Big
Table or H Base architecture is that, there is really only one key, which is the
primary key with which, data is distributed across different processors or
different channels. If one wants to access data by any other
column family. One can't really rely on any other index
and the only way to do that is by reading all the data.
So for queries, Big Table is not that efficient, and one needs to add additional
structure to it, so as to enable efficient queries.
In fact this is true for, any mechanism that one used to store data in sharded
form. By sharding, one means storing different
rows on different pieces of disk or different servers like, like tablet
servers and distributed file systems. So, if you have sharded data, where data
is distributed across machines by some key, and you want to access it using
another key. You need to do something smarter.
Which is essentially to create an index of some kind.
For example let's take our, Big Table of records which indicate invoice
transactions. For example which are to do with billing
of some kind, or some products. So that your, our main table is the
invoice table which has keys which is the transaction.
Which might have different values for different time-stamps as we have discussed
earlier. But now if you want to search this table
by some other column such as by product, one would need to create index tables
which would also be big tables but, their keys would be things like, product CD
Achieve, product DBME, etcetera, which would tell us which key in the original
table, this particular combination of product actually lies.
Similarly one could create index tables for amounts, as well as for combinations
between, say city and the status of the index, of the, of the transaction.
It's also useful to create this index in sorted form.
So that when you insert the records in a index table they actually get inserted in
sorted order. As a result, making a query which asks us
to find all transactions with amounts between two ranges, say between 50 and 90
becomes easier since all the values between such a range lie in the same
contiguous piece of the index table for same amount.
Once you've retrieved all these index values, one knows which keys to access and
one then can directly access them using the original Big Table or H Base Table.
One example where exactly such a structure is likely used is Google App Engine's Data
Store which is, probably based on big table as most have speculated and uses
indexes in exactly this way to query the big table efficiantely