Of course as we have seen parallel
computing is not new and has been around
essentially since the early 80s and
databases have also, traditionally evolved
to exploit parallel computing over the
years.
In the beginning we had shared memory
databases.
Which still persists to date where you
have a large.
Multi processor system, with multiple CPUs
sharing memory.
A single operating system scheduling jobs
or processes across different CPUs and a
common disk or storage area network, where
all the data is stored.
Examples of such systems abound.
Almost all servers today are
multiprocessing shared memory systems.
But they, almost a few dozen processors,
and even with each of them having multiple
cores, we find shared memory systems
supporting at most a few hundred
processing units.
The shared memory model simply doesn't
scale beyond this level.
Databases have exploited other parallel
architectures, such as the shared disk
architecture, and the shared nothing
architecture, which scale to greater
number of processors as compared to shared
memory.
In a shared disk architecture you may have
multiple processors which communicate over
a network.
Accessing, however, a common disk system
which could be a storage area network or a
network attached system of storage using
two different networks, one for
communication between processors, and the
other one for accessing storage.
The shared nothing architecture on the
other hand.
Relies on local disks with each processor.
So that the only communication that takes
place over the network is between
processors.
In parallele database in both the shared
nothing architectures as well as the
others.
Sequel queries are executed in parallele
by multiple processors.
In the shared nothing architecture in
addition.
Data itself is distributed across
different discs using varieties of
partitioning schemes.
Such as different sets of rows on
different discs or in the case of column
oriented, specialized engines for
analytical processing, different sets of
columns on different discs.
All this has happened, in the database
community and parallel databases are now
almost given.
They all support sequel.
Some of them support transaction
processing, where of course there is the
additional overhead of managing
transaction isolation and consistency of
cross multiple processors.
But we won't get into that right now.
The thing that is not handled by parallel
databases properly is fault tolerance.
They didn't have to handle fault tolerance
because with just a few dozen processors,
you don't need to worry about processors
failing in while executing a sequel query
which in any way will take few seconds or
few minutes utmost.
When you are executing a large batch job
which touches virtually all the data.
The chances of a processor failing are
very high, especially a large number of
processors.
And some situations, the pile of database
architecture simply are not full tolerant.
Full tolerance in the parallel database
world relies on having a hot standby
architecture or deployment which is
identical to the primary and essentially
replicating data over a high speed network
between the primary and the hot standby.
A very costly and still not completely
fault tolerant architecture as compared to
say, the highly distributed,
fault-tolerant, map-produced system on a
distributed GFS or HDFS.
So here is where the dichotomy comes in
when you're doing large volume analytical
processing which touches all the data the
parallel database architecture simply
doesn't work.
So, where databases have been evolving
over the past few years.
Because of the big data technology that
has emerged from the web, is in two
directions.
On the one hand we have the no sequel
databases.
Which are based on big data technology.
They're called no sequel because firstly
don't, they don't support full asset
transactions or rather fully isolated
serializable transactions in the
traditional sense.
Secondly, instead of having complex
indexing, they have chartered indexing,
which is essentially having partitioning
of the data between different chunks or
different blocks of disk.
We'll come to sharded indexing shortly.
Third, they don't support full joints.
They, for a variety of reasons, they are
restricted in the kind of joints that they
can perform efficiently.
And lastly they support column-oriented
stories if needed, so that for very long,
wide columns, different parts of a record
can be stored on different service.
The other side of database evolution is in
memory databases, which is, has been
driven by the, increasing.
Volumes of main memory available on
today's servers.
And the falling cost of memories.
So today servers have, you know, 64
gigabytes or even 124 gigabytes of main
memory.
Which allows for most practical purposes
many ordinary enterprise transaction
systems to actually reside in main memory.
So, real time transactions become
possible.
At much higher rates than before.
Varieties of indexes can be supported just
as before.
In a, in traditional databases, but now in
memory.
So, different kinds of indexing structures
are possible.
And of course complex joints are possible.
Of course the data has to be still small
compared to web scale petabytes of data or
many hundreds of terabytes.
But if you are talking about gigabytes of
data which are traditional enterprise
transaction system, rarely exceeds in
memory database is a happy compromise
where all lab queries and all kinds of
bulk data processing as well can be fairly
efficiently performed.
However this is not the big data world, in
the web.
That doesn't quite work and, therefore, no
Sequel databases have become quite
popular.
We will study some of the non sequel table
databases and the concepts very shortly.
As, as far in memory databases are
concerned, many of the techniques that
worked in traditional databases, simply
carry over, except for the fact that
because everything is in memory, one
doesn't have to deal with additional
complexity of some parts of the index
being cached versus being on disc and
things like that.