Well as you might have guessed we are going to return to information theory.
And look at machine learning from that perspective.
We're transmitting signals consisting of features - words or other features for
example and we are using those to predict the values of some behavior - browsers
versus buyers etc. And our goal is to improve the mutual
information between these two signals via a machine learning algorithm.
Now, what you all are probably waiting for is the actual definition of mutual
information. So the mutual information between a
feature, a word for example, and a behavior, browser or buyer for example is
defined formally as the probability of that feature and behavior occurring for
some particular value of CF and B, multiplied by this quantity.
And this sum is a double summation over all values of feature and all possible
behavior. So, in our case, it would be four values
in the sum, feature present, absent, behavior browser, buyer.
And in each case you would compute the joint probability of a feature and a
behavior multiplied by this ratio. Now, to understand this ratio better.
Imagine if the feature and behavior are independent.
If that's the case then the probability of the feature and the behavior together is
nothing but the product of the probability of the feature times the probability of
the behavior. So this ratio becomes one the logarithm
become zero and so does the mutual information.
But if the feature and the behavior are independent then obviously it's hopeless
to try to predict the values of the behavior from the values of the feature.
We'll do an example in a minute to actually compute the mutual information.
But before that a little history about mutual information.
What Shannon was really trying to do is measure the information content of various
signals. So, there's a formal definition of
information content called the entropy, which we again haven't defined.
And we're not going to do that. But still, believe me, there is one and
similarly there'll be an information content for the behavior signal.
There also be an information content for the signal consisting of both observations
- the feature and the behavior combined and the mutual information is nothing but
the difference between the total information in f and b.
And information of f and b when observed together.
We won't go into the intuition behind this too much, except to note that the
information content of two variables together.
Then, the total information in both the variables put together.
As a result the mutual information has to be positive.
Just remember this, if during any of your calculations you get a negative value of
mutual information, you've made a mistake. Now let's do an example.
We'll take the same set of comments that we used earlier and compute the mutual
information between a word such as hate and the sentiment positive or negative.
The probability that a comment is positive is just a fraction of positive comments,
which is 6000/8000 and similarly 2000/8000 for negative comments.
The probability that hate occurs in a comment is just the ratio of hate out of
the total number of 8000 comments and similarly probably that hate does not
occur. The joint probabilities are a little more
tricky. The're a little different from the
conditional probabilities that we had earlier.
We, for example there will probably be the hate does not occur in a positive comment
is well it's all the positive comments that is six thousand of them but we divide
by the total number of comments. Because this is the joint probability
rather than the conditional one. In this case for probability of hate it
doesn't occur in the positive comment so we have to smooth it by just making the
value one over 8000. And similarly we can compute the
probability of not hate. In a negative comment in the probability
of hate in a negative comment. The mutual information between the word
hate and the sentiment, + or -, is obtained by plugging into the formula.
And you have four terms in the formula, one for hate +, hate not hate +, hate -,
and not hate -. And the result is that we get the mutual
information between hate and sentiment is .22.
Well, is this good or bad? Let's check another word, the work course.
It occurs in all the comments, so the probability of course is just one, and the
probability of not course is actually zero but we smooth it by making it one over a
thousand. The joint probability, of course, for
positive comments, is just the fraction of positive comments, because it occurs, all,
all positive comments. In fact, it occurs in all comments.
And similarly, the probability, of course, for negative comments, is just the
probability of negative comments, because it occurs in all comments.
And not course doesn't occur, so again we smooth this.
The resulting mutual information is.003. So what this is saying is that a word like
course which occurs everywhere is not able to tell me anything about whether a
sentiment is positive/negative. Quite obvious.
But let's change the problem slightly. Let's now look at the case where these two
comments don't actually have the word, course.
So, we have just reworded them a bit. For course, now, we have different values,
because course occurs only in some of the comments.
And not course occurs in these 1,400 comments.
The joint probability, of course, given the positive, again changes.
Not all the positive comments, it's just 5,000 out of 6,000 positive comments have
course, because you're removing these 1,000 comments that are positive and have,
don't have course. And the pro-, probability that not course
occurs in a positive comment is exactly this 1,000.
So, you get these values. Now the value of mutual information
between course and sentiment is a bit bigger than before.
But still much, much smaller than .22. What this tells us is that course is still
a poor determiner of whether or not a comment is positive or negative.
Something which is intuitively obvious to us.
What's interesting is that using mutual information a computer can determine such
facts from examining vast volumes of data.