When we normally use the word 'information' our meaning is colloquial.
We use this term very informally to convey its informal meaning.
But it has a very formal mathematical definition as well.
For a really enjoyable explanation of the formal meaning of its information and its
history, a recent book by James Clyke is highly recommended.
In the context of advertising and news information has a very important role to
play. Many of you will remember the scandal that
broke last year in the British media world.
And you may have wondered, like I did, why did these people do these obviously
illegal things? Well, simply so that you and I read their
story and they become more and more popular.
To put it simply, a story about a dog biting a man is not newsworthy at all.
It happens very often. But if you reverse the words and a story
is about a man biting a dog, well it becomes interesting because this doesn't
really happen that often. Why is news so much about the scandal of
such events, the rare events, the unexpected?
As it turns out, Lord Shannon in his famous 1948 theory about communications
defined the term 'information' formally as being related to surprise.
In particular, a message informing us that some event has occurred, an event which
normally has probability p of occurring, well such a message conveys a precise
information content which Shannon argued was exactly minus log of p.
Lets look at this for a moment. If P is one then log of P is zero.
So a message that informs us that an event which is guaranteed to occur has actually
occurred is not really telling us anything new.
So, the information content is zero. On the other hand, if p is close to zero,
that means the event is very rare, then minus log p is a very large number telling
us that a rare event conveys a large amount of information.
Equally importantly, Shannon defined the term bit, saying that a message informing
us of an event of probability P conveys minus log P bits of information.
Now one might wonder whether this bit has anything to do with the bits we all used
to? Well, it's exactly the same bit and here
is why. Think about the Morse code where letters
are encoded by dots and dashes which is used for communication by telegraph.
Outdated now, but very common at the time of Shannon.
Notice that letters E, A, I, which are the most common letters in the language are
encoded with a small number of symbols. Whereas less common letters are encoded
with longer sequences. Similarly notice that the more common
words in language itself have the shorter spellings.
And, the rarer words have longer spellings.
Quite simply words, or even artificial codes, naturally encode those items which
are more frequent with smaller numbers of bits.
Reserving the longer sequences, or the longer bit sequences for the rarer items.
Just for efficiency purposes. And this is the relationship between bits
and surprise. Another way to understand this is to see
what happens when he is point five or when you are tossing the coin which can be
either heads or tails, one or zero. The information works out to be exactly
one. So the news of event which tells us that
one of two equally probable things have occurred Conveys a bit of information.
In fact, the concept of information is so deep, not only for our digital world but
for our entire universe itself, that some physicists, such as John Wheeler have
suggested that the entire laws of physics can be derived directly form information,
yin and yang It from bit. Well all that sounds very interesting, but
lets now return to our discussion about news, news papers and what it takes to
make news sell.