Let's recap now our unified formal
framework for dealing with classification
clustering, rule mining, all under the
same umbrella and see what it has to do
with big data.
And, whether it gives any additional
insight as we have alluded to a couple of
times in the past few minutes.
Recall that we defined this function, f of
x, where x is the set of features which
can take various values in a large space.
As the expected value of the output, which
could be zero and one for classification,
etc.,, for appropriate data sets.
So, we define this, not on the original
data but on other types of data.
For example, for the classification exam
case, we had the output variable to zero,
one, and the problem becomes estimating
this function or estimating where the
values of this function will be less than
a half or more than a half to decide the
decision boundary.
Then, we added on random data, which means
we added more data to the data set to deal
with clustering.
In the case of rule mining, we added
independent data not random data, but data
where all the features were independent of
each other.
And then, our problem became that of
finding regions where this function f is
large on this new data set.
Now, suppose we really have big data that
means, lots and lots of examples.
It's long data, in the sense that there
are lots of examples.
But, the number of features isn't too
large, so it's not wide data.
This is typical of real world data outside
of the web world where you have words, and
even images and videos, where the numbers
of features can be huge.
But in, in things like transactions or
small number of features of which are
typically found in traditional data sets.
This is a typical situation where the data
is very long, and if you start storing
lots and lots of transactions, you can get
big data into sets.
So, the problem A now reduces to just
querying the data and figuring out, for a
particular combination of x,
What's the expected value of y?
And if you remember, there was a question
in our earlier.
Homework or problem set where I asked
whether, if we had lots of data we could
simply estimate the joint probability
directly.
And indeed, if you really have enough data
that for every possible combination of x,
We can compute this expected value with
some degree of accuracy because you have
enough data.
Then, finding out which class an instance
belongs to is simply a matter of querying
and figuring out for that particular
combination how many positive instances
have you seen and how many negative,
And deciding based on the expected value.
The problem B on the other hand, reduces
to finding regions of high support.
Let's see how that happens.
Suppose we have added new data to our data
set in this matter, and computed the value
of f for every data point.
Now, data points which have high values of
f, we keep, and those that don't, we
discard.
Let's, let's look at that, that way.
Once we have only points which have high
values of f,
Now the question is, what regions
characterize these high values of f?
Are there particular combinations of
features which we can say are typical of
high regions and these would be our
interesting rules or interesting clusters?
And again, all we need to do is find high
support region, in the sense that which
combination of features have high support
out of those which have high value of f.
Of course, we still have the problem of
dealing with negative rules even in this
high support technique.
So that, that doesn't solve the entire
problem.
It certainly is a route to solving the
problem.
Finally, remember that we're, even though,
in principle we added random data for
clustering and for rule mining, it was
start experiment so we don't actually add
data.
We sort of, can compute the value of f by
dividing by the probability of that
particular combination occurring, assuming
uniform density of data or the particular
probability assuming independent data by
direct calculation rather than by adding
random sets of data.
I hope that's clear, that you don't
actually have to add data.
It's, it's, it's, it's a way of imagining
what's really going on but we actually
simply count the actual real data, divided
by the appropriate quantity.
In the case of, say, independent,
The P0 being independent assumption,
you're actually computing the information
gain between features when you do this.
And using that value for f,,
Filtering out that data based on the value
of f, and then finding reason of high
support.
Note here that we're talking about finding
high support regions and not just high
support combinations of features as in the
association rule mining case.
Finding regions of high support is a
little bit more difficult and is often
referred to as bump hunting.
We won't go into this in more detail here,
But the unified formulation that we have
used is particularly useful in tackling
bump hunting problems.
The important point is, such techniques
where we simply query the data or find
high support regions, not maybe in the
original data, but in some slightly
modified data where we filter it by high
f, is just in the end counting.
So, techniques like MapReduce or on very
large data sets dremel for querying, work
by brute force.
And so, big data where there's lot of
data,
Can actually be solved with fairly simple
techniques of just counting.
And, this is something which we need to
understand that big data can change the
way one does statistical analysis because
counting now seems to work much better
than it did if one didn't have enough
data.
Of course, why data?
That means, when the number of features is
very large or are the high dimensional
data is still a problem as we shall see
very soon.