Bit by Bit Measuring Information. Brad Osgood Information Systems Lab Stanford

Transcription

1 Bit by Bit Measuring Information Brad Osgood Information Systems Lab Stanford

2 Material ages The Stone Age From - to about 4000BC The Bronze Age From 2300 BC to 500 BC The Iron Age From 800 BC to 100 AD

3 The Information Age Begins in 1948 with the work of Claude Shannon at Bell Labs

4 What do the codes used for sending messages back from spacecraft have in common with genes on a molecule of DNA? How is it that the second law of thermodynamics, a physicist s discovery, is related to communication? Why are the knotty problems in the mathematical theory of probability connected with the way we express ourselves in speech and communication? The answer to all of these questions is information Jeremy Campbell, Grammatical Man,1982

5 I shall argue that this information flow, not energy per se, is the prime mover of life that molecular information flowing in circles brings forth the organization we call organism and maintains it against the ever present disorganizing pressures in the physics universe. So viewed, the information circle becomes the unit of life. Werner Lowenstein, The Touchstone of Life, 2000

6 And for today s college students? Information wants to be free. Stewart Brand, founder of the Whole Earth Catalog I want my tunes. Anon.

7 In April of 1999 the term MP3 surpassed sex as the most-searched-on term at some of the Internet s top search engines - -- a phenomenal achievement for a complicated digital music encoding algorithm devised over the course of a decade by a few scientists and audiophiles in an obscure German laboratory. Scot Hacker, MP3: The Definitive Guide

8 Aspects of Information?

9 Practical Perceptual Physical All have something to do with communication

10 Aspects of information the theory The fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at another point. Frequently the messages have meaning; that is they refer to or are correlated to some system with certain physical or contextual entities. These semantic aspects of communication are irrelevant to the engineering problem. The significant aspect is that the actual message is one selected from a set of possible messages. Claude Shannon, The Mathematical Theory of Communication, 1948

11 (Also in 1948) Mahatma Gandhi is assassinated The Marshall Plan is enacted Israel is created Truman defeats Dewey T.S. Elliot wins the Nobel Prize for Literature Columbia records introduces the LP CBS begins network programming

12 How do we measure information? Remember Shannon s quote: The significant aspect is that the actual message is one selected from a set of possible messages.

13 Prior condition for communication to be possible: The sender and receiver both have to have the same set of all possible messages, or be able to construct it. They need the same codebook

14 The most famous codebook in history?

15

16 Let s play 20 questions! I m thinking of a famous person.

17 1. The person is Brad Osgood

18 1. The person is Brad Osgood 2. The person is Rebecca Osgood

19 1. The person is Brad Osgood 2. The person is Rebecca Osgood 3. The person is Miles Osgood 4. The person is Madeleine Osgood

20 1. The person is Brad Osgood 2. The person is Rebecca Osgood 3. The person is Miles Osgood 4. The person is Madeleine Osgood 5. The person is Ruth Osgood 6. The person is Herbert Osgood 7. The person is Lynn Osgood 8. The person is Alex Beasley 9. The person is Thomas Faxon 10. The person is Virginia Faxon 11. The person is Thomas Faxon, Jr. 12. The person is Meer Deiters 13. The person is Francisca Faxon 14. The person is Pia Faxon 15. The person is George W. Bush 16. The person is Saddam Hussein

21 Brad says: Who needs 20 questions. I bet I can pick out any object (in English) by asking 18 questions. OK, maybe 19. Hah! What is the basis for this bold claim? Is it justified? In the real version of 20 questions the sender says that object is animal, mineral or vegetable to allow the receiver to narrow down their questions. Just how many things can you determine by asking 20 questions?

22 2 18 = 262, = 524,288 The number of entries in the 1989 edition of the Oxford English Dictionary is 291, = 1,048,576

23 The unit of information is the bit How many bits how many yes-no questions are needed to select one particular message from a set of possible messages? The possible messages are encoded into sequences of bits. In practice, 0 s and 1 s (off, on; no, yes). Many coding schemes are possible, some more efficient or reliable than others. There are many ways to play 20 questions

24 General definition of amount of information Suppose there are N possible messages. The amount of information in any particular message is I = log 2 N (unit is bits) (Same thing as saying 2 I =N) What does it mean to say that the amount of information in a message is, e.g., 3.45 bits?

25 I m more famous than you are In any practical application not all messages are equally probable. How can we measure information taking probabilities into account?

26 1. The person is Brad Osgood 2. The person is Brad Osgood 3. The person is Brad Osgood 4. The person is Brad Osgood 5. The person is George W. Bush 6. The person is Saddam Hussein 7. The person is Colin Powell 8. The person is Condoleezza Rice Playing the game many times, how many questions do you think you d need, on average to pick out a particular message?

27 Is the person in the group 1 through 4? Yes. No One question resolves the uncertainty. Need two more questions, for a total of three. Brad Osgood occurs 4 out of 8 times: Probability 4/8=1/2 I( Brad Osgood ) = 1 Everybody else occurs 1 out of 8 times: Probability 1/8 I( George W. ) = 3

28 In general, if a message S occurs with probability p then I(S) = log 2 (1/p) If we have N messages (the source ) S 1, S 2,,S N occurring with probabilities p 1,p 2,,p N then the average information of the source as a whole (the entropy of the source ) is the weighted average of the information of the individual messages: H=p 1 log 2 (1/p 1 )+p 2 log 2 (1/p 2 )+ + p N log 2 (1/p N )

29 Shannon defined entropy as a measure of average information in a source (the collection of possible messages), taking probabilities into account. He defined the capacity of a channel as a measure of how much information it could transmit. And he proved:

30 Noiseless Source Coding Theorem: For any coding scheme the average length of a codeword is at least the entropy. Channel Coding Theorem: A channel with capacity C is capable, with suitable coding, of transmitting at any rate less than C bits per symbol with vanishingly small probability of error. For rates greater than C the probability of error cannot be made arbitrarily small.

31 Most great physical and mathematical discoveries seem trivial after you understand them. You say to yourself: I could have done that. But as I hold the tattered journal containing Claude Shannon s classic 1948 paper A Mathematical Theory of Communication I see yellowed pages filled with vacuum tubes and mechanisms of yesteryear, and I know I could never have conceived the insightful theory of information shining through these glossy pages of archaic font. I know of no greater work of genius in the annals of technological thought. Robert W. Lucky, Silicon Dreams, 1989

32 It took awhile for the technology to catch up with Shannon s theory

33 The news from Troy In Agamemnon by Aeschylus The fall of Troy was signaled by a beacon. The play opens with a watchman who waited for 12 years for a single piece of news: the promised sign, the beacon flare to speak from Troy and utter one word, `Victory!'."

34 The news from Paris A message was spelled out, symbol by symbol, and relayed from one station to the next. Operators at intermediate stations were allowed to know only portions of the codebook. The full codebook, which had over 25,000 entries, was given only to the inspectors.

35

36 High Tech of the mid 19 th Century 1824 Samuel F.B. Morse, an art instructor, learns about electromagnetism 1831 Joseph Henry demonstrates an electromagnetic telegraph with a one mile run in Albany, New York Morse demonstrates his electric telegraph in New York 1837 Wheatstone and Cooke set up British electric telegraph. Transatlantic cables around 1904

37 The first shot in the second William Thomson (later Lord Kelvin, ) On the theory of the electric telegraph, Proceedings of the Royal Society, 1855 industrial revolution Answered the question of why signals smear out over a long cable.

38 Communication became mathematical! Surely this must have been hailed as a breakthrough!

39 I believe nature knows no such application of this law and I can only regard it as a fiction of the schools; a forced and violent application of a principle in Physics, good and true under other circumstances, but misapplied here. Edward Whitehouse, chief electrician for the Atlantic Telegraph Company, speaking in 1856.

40 Right. The first transatlantic cable used Whitehouse s specifications, not Thomson s The continents were joined August 5, 1858 (after four previous failed attempts). The first successful message was sent August 16. The cable failed three weeks later. Whitehouse insisted on using high voltage, disregarding Thomson s analysis

41 The rise of telegraph, telephone and networks Brought to you by AT&T

42 Broadway & John Street, New York 1890

43 Gerard Exchange, London, 1926

44 The last of the great data networks?

45

46 The full impact of Shannon s theory was felt only with the development of digital signal processing and communication

47 Analog Signal (e.g. Music, Speech, Images) A to D converte Digitized signal (0s and 1 s) Compression (e.g. MP3) Add error correction (e.g fixes scratches in CDs) Noise! The Channel (e.g. Fiber optics, the Internet, Computer memory) Correct errors (Remove redundancy) Uncompress D to A converte

48 One example: Data compression We try to use the fewest number of bits possible to send digital messages. How efficient we are depends on the coding scheme. But Shannon s theorem on source coding and entropy limits how efficient we can be! Different coding schemes are used for different types of messages (text, sound or images), but there are common ideas in the methods.

49 Imagine sitting at a table talking about someone you all know who isn t there. At one point, I look up at my wife on the other side of the table and I wink. After the dinner, you come to me and ask what the wink meant. I say that three weeks ago had dinner with this person and she said she was going to do 100,000 bits later I ve told you what I told to my wife with one bit through the ether. The reason that works, and indulge me that a wink is a bit, is that the transmitter has a body of knowledge that is shared with the receiver. And I know she knows I know and she knows I know she knows. When you don t have that intelligence, then I have to give you all 100,000 bits. Nicolas Negroponte, Director MIT Media Lab

50 The briefest correspondence in history (according to the Guiness Book of World Records) In 1862 Victor Hugo had published Les Miserables and then went on vacation. He could not keep himself from wondering about the reception and sales of his latest book. He wrote his publisher

51 ?

52 His publisher wrote back

53 !

54 General categories of compression techniques Dictionary methods These methods work by replacing symbols, words or phrases that occur in a message by pointers to a dictionary (a codebook ) Lempel-Ziv (UNIX compress command) Symbolwise methods Use estimates for frequency of occurrence, or probability, of symbols or blocks of symbols. Shorter codewords are used for frequently occurring symbols, longer codewords are used for less frequently occurring symbols. Huffman coding (JPEG, MPEG, MP3)

55

56

57 More to the story Other compression techniques, especially for images and video. It s a big world out there. Error correction It s a noisy world out there.

58 And one more thing. It s a risky world out there.

59 Analog Signal (e.g. Music, Speech, Images) A to D converte Digitized signal (0s and 1 s) Compression (e.g. MP3) Add error correction (e.g fixes scratches in CDs) Noise! The Channel (e.g. Fiber optics, the Internet, Computer memory) Correct errors (Remove redundancy) Uncompress D to A converte