So what we're, trying to discern. Is intent.
What meaning is trying to be conveyed by the writer, the speaker or the actor?
Well, we'll return to, the business of trying to figure out what, meaning is
trying to be conveyed. Next week, when we look at extracting
this, information from documents. But for the moment, let's turn to trying
to find out other predictors of intent. Which might not have anything to do with
the content of a document, which is being read.
But, by the actions taken to retrieve the document by searching for it.
For example, suppose we are searching for flower, red, gift, cheap, I mean you are
searching using some combination of these keyword, and the web property, which is
Google for example, needs to decide whether or not to show you some ads.
In other words, are you a surfer or a shopper?
Are you, interested in buying something or are you just browsing?
Let's see what I get. When I search for cheap flowers.
Clearly, it thinks that I'm trying to find some flowers to buy.
On the other hand, if I just search for flowers and red, it shows me a whole bunch
of red flowers, and no hats. Somehow, Google has figured out that if
I'm searching for red flowers, I'm most likely trying to find out some information
about flowers, rather than buy some flowers.
How might it have figured this out? One way is by looking at past history.
What the people searching about red flowers normally do?
Do they buy stuff or do they not? On the other hand, people searching for
cheap flowers or gift using flowers, do they buy or don't they?
Learning from past experiences is something that we do all our lives.
The field of machine learning is all about teaching computers, how to learn form
their past experience or from past data and as you all know this is the heart of
web intelligence and is what big data analytics is really all about.
We'll introduce machine learning very simply, in a very basic manner.
By looking at the simple example of how one might guess whether or not to put an
ad, based on the past behavior of many, many searchers using just these four
keywords. So let's suppose we have all the
historical data. For example, somebody who used the
keywords red, flower, gift, and not cheap, and did not buy something.
On the other hand, somebody used the terms red and cheap but did buy something.
And at the same time there may be people using exactly the same combination such as
red flower gift. But these, they bought something where as
in this case they didn't buy anything. So you have all this data from which you
want to learn whether or not you should put an ad given the fact that somebody is
searching with some combination of these four keywords.
In the language of probability theory, what we really want to figure out is the
probability of a by-action, given values yes or no, for whether or not the words,
red, flowers, gift and cheap are present or absent in the query.
In other words, we're trying to find the conditional probability of a buy given the
values of these four random variables. Let's see what we really need to do.
For each combination of keywords being present or absent.
For example, all yes, three yes and a no. Two no and two yes.
We have the probability that there is a buy and we would also like to figure out
the probability for the same combination that there is not a buy.
Let's see how we might do this, using the data that we have from past history.
Notice that this is a summary table with one entry, for each combination, whereas
our historical data probably has millions of entries for each combination.
So this probability is computed by adding up, appropriately, the data from our
historical transactions. Let's look at this pictorially.
Suppose we have N instances or N historical queries.
In, our cases they had, the keyword red. In F cases, they had the keyword flower.
Similarly G for gift and C for cheap. And for K cases there was actually a buy
action. And for N - K cases there was not a buy
action. Well, let's try to find out what the
conditional probability of a buy action is given that query had all the keywords
present. So, the query lies in this piece of this
diagram, because all these ovals are overlapping for I cases.
The denominator is the set of all transactions which actually had a yes for
R, F, G, and C. So, it's R + F + G + C.
So, this becomes the conditional probability of a buy given R, F, G, and C
are all yes. Similarly, this part of the diagram J
indicates those transactions that the number of transactions which had a no for
exactly the same combination R, F, G, and C being yes.
So J / R + F + G + C is the conditional probability that buy = no given that
combination all yes. So it appears all we need to figure out is
all these values for every possible combination and we should be able to
decide whether or not to put an add in front of a particular query sequence.
The trouble is the number of such combinations can be quite large.
How many do you think there are for just four keywords?
Obviously, just sixteen. But suppose we had 1000 key words that
suddenly becomes a very, very large number.
Even for a few 100 key words it becomes extremely difficult to compute all these
conditional probabilities. More importantly, when the number of
keywords gets really large, there will be many combinations for which, you never
actually have any history. For example, you're quite unlikely to find
a query, which has red flowers and say map produce where intelligence all in one
query you'll just not have such history. Nobody would be searching for this
combination. So even if you had infinite computing
power, you simply don't have enough history to compute all entries in this
conditional probability table.