So in order to compute these conditional probabilities when there are a large
number of possible words or features, we need to do a little more work.
Let's look at the simple case of just one word, say red.
There are two possibilities. The red could be present or absent.
So, for our cases red is present. Out of these I have a Bi and the rest
don't. The total number of cases which are Y
situations are obviously K as before. So lets see what the simple condition
probabilities are in this case. The probability of a Bi given that there
is an R is just I over R. On the other hand, the probability of an R
over all is just R over the total number N.
And the probability that both R occurs and there is a Bi is I over N.
Now this is the joint probability, so you'll have to divide by the total number
of instances N rather than any of the conditional ones K or R.
Baye's rule is actually just simple arithmetic.
In particular if you write I over N as I over R times R over N, lets see what you
get. Well we already know what I over R is its
just the conditional probability of a Bi given R.
And R over N is the probability of R itself, so simple arithmetic tells us that
the joint probability of R and B is the product of the conditional probability and
the a prior I probability. This is just Bayes rule.
The probability of B and R is the conditional probability of B given R times
P of R which is also the same thing as the probability of R given B times the
probability of B. We can see that by just rewriting I over
N, not by introducing an R, but by introducing a K.
In which case you'll get I over K, which is just the probability of R out of all
those that are B and the probability of B which is just K over N, right?
Bayes' Rule which some of you may or may not remember turns out to be just simple
arithmetic. As we shall soon see, Bayes' Rule is
critical to machine learning because it allows us to compute any of those many,
many joint probabilities even if there is no data for a particular combination.
Before we can do machine learning using Bayesian techniques we need one more
important concept that is independence. Think about two words like red and cheap.
As before, we have R equal to yes for R queries, cheap equal to yes for C cases
and in I cases both key words are present. The probability of an R occurring is
clearly R over N. And the probability of cheap occurring is
C over N, as before. And similarly, the conditional
probabilities of an R given that C occurs can be computed.
Independence says that the probability of R does not depend on whether or not the
word cheap is already present in the query.
In other words the probability of R should be the same as the probability of R given
C. Similarly the probability that cheap
occurs should not depend on whether or not red occurs.
In such situations these two features are independent.
Of course, that might not necessarily always be the case.
For example, somebody searching for big data might actually search for map reduce
at the same time rather than something else, like red or flower.