Now let's look at language itself, in terms of information theory.
Clearly language is a channel where we try to convey our meaning via spoken or
written words, just as I'm doing right now.
And we try to ensure that the mutual information between what you receive in
terms of what you hear or what you see is text is close to what I intend to convey
as the meaning. Many magicians, philosophers, linguists
and computer scientists have studied this idea in great detail and it's far from
being a resolved issue. For example, Richard Montague viewed this
from the perspective of truth versus falsehood.
Assuming that I'm conveying something true, are you able to discern that truth
from the spoken or written words that you receive?
So Montauge's view was a logical one where the purpose of language is to convey a
truth and the issue is whether or not that truth can be discerned.
Chomsky, on the other hand viewed the problem from the perspective of grammar,
whether or not a sentence is grammatically correct.
And, deeper grammar in terms of the different roles played by the actors and
the verbs in the sentence. These constituted the meaning for Chomsky,
regardless of whether there's actually some real truth being conveyed.
Some of you might think this is too philosophical for a discussion on web
intelligence, but consider this. Sentences which are grammatical need not
be conveying any meaning; they could be completely nonsensical.
At the same time tweets or SMS's are hardly ever grammatical, but they are
conveying some real meaning. So, the distinctions are not necessarily
purely philosophical, but actually have some practical value.
We'll return to this much later when we talk about extracting information from
spoken or written words. But, for the moment let's return to
information theory and language with the point that language is actually highly
redundant. In particular, Shannon figured out that
English is 75 percent redundant. He came to this conclusion by conducting
experiments, such as asking somebody to guess the next letter in this sentence.
For example, the lamp was on the D. Most of you would guess desk.
Many such examples show that context, history, experiences allow us to
essentially predict the next word, or the next letter.
His conclusion was that language is highly redundant and for exactly similar reasons
as we thought earlier, efficiency. Communicating information is more
efficient if we use more bits or more words to transmit concepts which are
rarer. And therefore have more information
content. As opposed to cases where we're
transmitting something fairly obvious. Turns out that actual experiments with
human subjects have confirmed that language actually tries to maintain a
uniform information density. So we use more words, therefore more bits
when trying to convey something which is deeper or has more information or is a
rarer event that the listener might not be expecting.
So is language all about statistics? Redundancy, TFIDF, counts.
Well, imagine yourself at a party. You hear snippets of conversation, which
ones catch your interest. Similarly imagine a web intelligence
program which is tapping Twitter, Facebook or even mails.
It needs to figure out what people are talking about, which ones have similar
interests. How might it do so?
Well, as we have seen in our discussion about keyword extraction, similar
documents probably have similar have Tf-idf keywords.
So maybe we just need to compare documents by looking at the keywords that we might
be getting using Tf-idf. Is this enough?
But think about words like, river, bank, account, boat, sand, deposit.
Well. River bank versus a bank account are two
different contexts for the word bank. Similarly.
Sand, bank, and river occurring together versus deposit, sand deposits.
There are again two different concepts for the word sand.
So the semantics of the use of a word depend on the context on which it is in
which it is being used. Is this context itself computable?
It requires a little more work than merely TFIDF.
To figure out the semantics of a word, that is the context in which it is used,
we need to investigate which documents, keywords, co-occurring very often.
The idea behind many techniques that try to compute the semantics is to view
documents and words in a sense as a Bipartite Graph, where you have document
on the one hand, keywords on the other, and you figure out which words are
contained in a document and figure out which other documents contain that word
and then iterate further to figure out which documents are closer because they
contain the same words. As well as which keywords are closer
because they occur in the same document. Techniques that exploit such iterations
probabilistically in fact. Try to uncover the latent semantics, so
they are called latent models. They try to discover the topics, which are
collection of documents are talking about. And they are also used in diverse areas
such as computer vision to figure out which objects are similar, which
collection or sequences are moving objects represent the same kinds of activity and
variety of other. Kinds of meaning that we almost
intuitively and unconsciously extract from words, spoken language as well as the
video that we continously see around us when we look around at the world.
All these techniques, whether they are simple counts words and in the various
document frequencies. Or more complicated co-occurrences across
large collections of documents are nevertheless statistical models.
So the question we also need to ask is, is meaning or semantics just statistics or is
there more?