Claude E. Shannon. Tina Jayroe. University of Denver

Transcription

1 Claude E. Shannon Tina Jayroe University of Denver Shimelis G. Assefa, PhD Information Science October 18, 2008

2 2 Abstract Claude Shannon s impact on digital technology, computer science, and information science is undisputed. Yet increasingly, Shannon s Mathematical Theory of Communication is being used to further research in areas such as psychology, semiotics, and logistics. This essay discusses Shannon s principal influences; provides an explanation of the quantitative theory and its effect on communication systems and designs; and emphasizes the importance of Warren Weaver s theoretical interpretation of the formula and how it relates to human cognition.

3 3 Claude Elwood Shannon ( ) was a brilliant mathematician and electrical engineer. His genius was publicly recognized in 1940 when he won the Alfred Nobel prize for his master s thesis A symbolic analysis of relay and switching circuits (Gallager, 2001, p. 2681). Today he is considered the founder of information theory, a title predominately due to his seminal paper A mathematical theory of communication which was published in He worked on the theory between the years 1940 and 1948 while employed at Bell Telephone Laboratories (Gallager, 2001, p. 2682), and in it he used the term information to mean a logarithm of the number of available choices and regarded it as a physical unit that is, a binary digit or bit (Shannon & Weaver, 1949 [1962], p. 100). Shannon compiled a mathematical formula that addressed the probability that certain amounts of information (signals or symbols) are initiated from what he called sets of events or states of affairs during transmission from source to recipient. The formula suggests optimizing channel capacities in communication systems by bringing forth a heightened notion of entropy, the measurement of randomness or disorganization in the amount of data as it moves through a transmission channel, and channel capacity where the bandwidth together with the signal-tonoise ratio determines the quality of a transmission channel (Gappmair, 1999, p. 103). Shannon in no way meant for his theory to encompass the semantic meaning (i.e., the message) of the information being transmitted; for him, only the engineering aspects applied (Shannon & Weaver, 1949 [1962], p. 100). Fortunately, another scientist and mathematician named Warren Weaver would read his publication, analyze his theory, and relate it to other forms of communication. Weaver stated that while semantics may not affect engineering aspects in the transmission of a message, the message s meaning could definitely be affected by its physical transmission (Shannon & Weaver, 1949 [1962], p. 100).

4 4 Weaver wrote an essay entitled Recent contributions to the mathematical theory of communication, and in it he declares Shannon s work The mathematical theory of communication. He was also able to explain the theory in less formidably mathematical terms (Weaver, 1970, p. 111). Weaver effectively proclaimed the work a theory of meaning in addition to validating its engineering concepts. Both Shannon and Weaver s papers were published together as a monograph in 1949, and upon its release, made a huge impact in the fields of science and technology. In fact, the theory was so widely received that it transformed communication systems immediately (Solana-Ortega, 2002, p. 462). Years later, the MITRE Corporation and the Electronic Systems Division (MITRE/ESD) congresses of 1962, 1964, and 1966 convened to explore the nature of information as it pertained to military and human-machine systems. More specifically, the first and second congresses fully relied upon Shannon s theory of communication along with Norbert Wiener s cybernetic research in order to assess the quantitative measurements of signal detection technologies of that time (Debons & Horne, 1998, pp ). The three meetings had a significant impact on computer systems design and the evolving discipline of information science. They also had an effect on universities whose programs contained computer and information science studies at the graduate and doctoral level leading to more scholarly focus on human-machine interaction, also known as systems theory (Debons & Horne, 1998, p. 209). The Mathematical Theory of Communication Shannon s theory was influenced by and built upon the work of the many people. John Tukey was a chemist and mathematician who coined the term bits. Shannon was the first to use the term and provided this definition: A device with two stable positions, such as a relay or flipflop circuit can store one bit of information (Shannon & Weaver, 1949 [1962], p. 4). Norbert

5 5 Wiener was a mathematician and the founder of cybernetics whose work greatly benefited from the recognized importance of Shannon s mathematical theory. Ralph Hartley and Harry Nyquist had written papers in the 1920s on pulse-code modulation (PCM) and pulse-position modulation (PPM) which Shannon studied prior to developing the theory. And his employment at Bell Labs had him working alongside other notable mathematicians and scientists such as J.R. Pierce, an expert in satellite communications; J. Bardeen, W. Brattain, and W.B. Shockley, the inventors of the transistor; and G. Stibitz, the builder of one of the first computers based on the binary system (Solana-Ortega, 2002, p. 460). Shannon also received guidance from MIT professor Vannevar Bush, an engineer and visionary of the Internet who pioneered many automatic processes. Bush recruited Shannon to work on his Differential Analyzer a machine that was used to solve differential equations (Griffin, 2000, 9). According to both Shannon and Weaver, the theory states that information consists of uncertainties, in that there are multiple choices in selecting a message to be conveyed (Shannon & Weaver, 1949 [1962], p. 96). In Shannon s theory, entropy is that freedom of choice, and by calculating the relative entropy the ratio of the actual to the maximum entropy of the source and the redundancy the fraction of the message which is determined not by the free choice of the sender, but rather by the accepted statistical rules governing the use of symbols in question it is possible to maximize the value of an information source (Shannon & Weaver, 1949 [1962], pp ). While Shannon attributes this concept to binary digits, Weaver uses the English language as an example: [T]he redundancy of English is just about 50 per cent, so that about half of the letters or words we choose in writing or speaking are under our free choice, and about half

6 6 (although we are not aware of it) are really controlled by the statistical structure of the language (1949 [1962], p. 104). In other words, it is possible to predict which words/letters/characters will follow others. All a system needs to do is account for and accommodate all possible choices via statistical logarithmic functions. If communication systems can be made to estimate and optimize the length and amount of symbols moving through a transmission channel, then the symbols are organized, which lessens the entropy; meaning the chances that there will be missing information when the source signal gets to the receiver is significantly lowered (Shannon & Weaver, 1949 [1962], p. 103). The opposite is often called inverse probability, where chances of something unexpected occurring produces a lot more information (more freedom of choice) and therefore, more chance for information loss, or noise (Perez-Montoro, 2007, p. 29). In his information theory Shannon factored noise into how it affects the fidelity (distortion), and consequently, the content of a signal. This aspect of communication relies on his coding theorem. Shannon suggested developing efficient codes that would maximize information transmission (entropy) and lessen noise as it moves through a given channel. In reference to Shannon s theorem, Belzer states: Since transmission is measured by the number of bits per second, efficient codes give short codes to messages which occur frequently and longer codes for messages which are transmitted less frequently (1973, p. 301). One could relate this to Zipf s Law (the principle of least effort) in that by coding the symbols in this way, the result is less tax on the system (Zunde, 1981, p. 344; Lynch, 1977, p. 19), and to relevance theory by calculating the contextual effect vs. processing effort, it is possible to predict which information would be selected by an individual. Goatley elaborates using Sperber and Wilson:

7 7 You wake up thinking, (1) If it s raining I won t go to the lecture this morning. You look out the window and discover, (2) It s raining. From existing assumption (1) and the new information (2) you can deduce further information (3): (3) I won t go to the lecture this morning. (2) is relevant because, in the context of (1), it produces new information or contextually implies (3).... You wake up thinking: (4) If it s raining I won t go to the lecture this morning. Then either you look out of the window and see: (5) It s raining. or you look out of the window and see: (6) It s raining and the refuse collectors are emptying the bins. In the context of (4), (5) and (6) have the same Contextual Effects. But (5) is more relevant than (6), because (6) requires more Processing Effort (Wilson and Sperber 1986, pp ). The notion of Relevance, then, which is comparative rather than absolute, can be summed up in the following formulae: (7) Other things being equal, the greater the Contextual Effects, the greater the relevance. (8) Other things being equal, the smaller the Processing Effort the greater the relevance. Or, alternatively, expressed as a fraction: (9) Relevance = Contextual Effects Processing Effort This equation makes it clear that if there is no Contextual Effect there will be no relevance, no matter how little the Processing Effort involved (Goatley, 1997, p ).

8 8 Researchers are finding more applications for Shannon s theory in the realm of information science. Many are trying to link linguistics and semantics to information theory for such purposes as automatic indexing and translation (Warner, 2007, p. 313). The benefit would lie in information retrieval systems where the syntax of a language (patterns of signs, signals, and symbols) is statistically calculated in order to deliver highly relevant information: This already works for systems in which the receiver has both human and electronic components, such as the telegraph (Warner, 2007, p. 317). Shannon Style Research Shannon was a very curious and instinctual person who had a passion for electronics, games, toys, problem solving, and artificial intelligence. Because he was considered so smart and so successful at such a young age, Bell Labs let him work on whatever he wanted; a privilege that was earned because of what he had already accomplished (Gallager, 2001, p. 2682). He later became a very popular and profound professor at MIT. His teaching methods were somewhat unorthodox in that he always lectured about new topics rather than going over the same subject again and again. But most importantly, he enabled his students to find simpler and more fundamental ways of looking at a problem before they got caught up in the details (Gallager, 2001, p. 2683). As a result of this methodology, Claude Shannon made another significant contribution to the world: a concept known as Shannon Style Research. The premise entails giving students and researchers the time to reflect upon and simplify complex problems in order to gain better insight on how a problem can be approached, and subsequently solved: The real legacy of Shannon's research, beyond all the neat results, is the existence proof that systems can be made understandable if we take the time to understand them. This

9 9 takes genius, but might be possible if we let students and young researchers develop these talents (Gallager, 1998, 16). This style of research is counterproductive in today s rapid pace of technological advances which often come with strict deadlines. The current environment thrives on tracking procedures and producing specific results. However, the reciprocity between Claude Shannon and Bell Telephone Laboratories should stand to remind educators and employers of a unique research model; one that, given the right circumstances, allots individuals a generous amount of time and intellectual freedom in order to expose the seemingly limitless boundaries of human knowledge. Conclusion Messages need to be transmitted and received in order for information to reduce uncertainty. The probability or indication of a message being selected, sent, and received is a good way to determine how much information it contains. In the library science discipline, this means that when a user poses a query into an online system, that system should be made to deliver relevant items based on the probability that it will resolve the user s uncertainty (the information need) where the entropy of the system is maximized (Belzer, 1973, p. 300). Ideally, this happens when little friction or noise (irrelevant documents) are present although until the user reads the document it is not known whether or not the content is actually relevant (Belzer, 1973, p. 30; a bzillion other people ). Shannon created these concepts and applied them to signal flow using engineering formulas combined with statistical terms and probabilistic processes. His theory was a huge milestone immediately applied to areas such as radio, television, and telecommunications. In the age of digital communications systems and the advent of the World Wide Web, Shannon s

10 10 encoding, decoding, and cryptography ideas come into play every day resolving computer design issues and enabling digital transmissions over the Internet. Warren Weaver was instrumental in interpreting and promoting the philosophical aspects of the theory by relating it to any information source and destination. He implied that Shannon s work was to be considered a theory of meaning and context applicable to speech, dance, art, and more. His belief was that all human behavior facilitates communication and is therefore related to Shannon s theory on one of the three levels that accompany the transmission of signals: technical problems; semantic problems; and effectiveness problems (Shannon & Weaver, 1949 [1962], pp ). Although there are still those who criticize applying its theoretical value to areas such as semiotics, logistics, or relevance theory (Solana-Ortega, 2002, p. 465; Herron, 2006, pp ), there is no denying its influence on computer science, information science, and systems design. By quantifying information as a thing in 1948 Claude Shannon enabled computers to think (Shannon & Weaver, 1949 [1962], p. 115). Now his concise computations enable scientists and researchers to think of new ways to foster knowledge in many non-mathematical realms.

11 11 References Belzer, J. (1973). Information theory as a measure of information content. Journal of the American Society for Information Science, 24(4), doi: /asi Debons, A. & Horne, E. E. (1998). NATO advanced study institutes of information science and foundations of information science. In Hahn, T. B. & Buckland, M. (Eds.), Historical studies in information science (pp ). Medford, NJ: Information Today, Inc. Gallager, R. G. (1998). Lessons learned from Claude Shannon. Retrieved October 12, 2008 from Gallager, R. G. (2001). Claude E. Shannon: A retrospective on his life, work, and impact. IEEE Transactions on Information Theory, 47(7), doi: / Gappmair, W. (1999). Claude E. Shannon: the 50 th anniversary of information theory. IEEE Communications Magazine, 37(4), doi: / Goatley, A. (1997). The language of metaphors. London, UK: Routledge. Griffin, S. (2000). Internet Pioneers. Retrieved October 15, 2008 from /pioneers/bush.html. Herron, P. J. (2006). Text mining adoption for pharmacogenomics-based drug discovery in a large pharmaceutical company: A case study. (Unpublished master s thesis) University of North Carolina, Chapel Hill, NC. Lynch, M. F. (1977). Variety generation a reinterpretation of Shannon s mathematical theory of communication, and its implications for information science. Journal of the American Society for Information Science, 28(1), doi: /asi Perez-Montoro, M. (2007). The phenomenon of information: A conceptual approach to information flow. Lanham, MD: The Scarecrow Press, Inc.

12 12 Shannon, C. E. & Weaver, W. (1962). The mathematical theory of communication (9 th ed.). Urbana, IL: The University of Illinois Press. Solana-Ortega, A. (2002). The information revolution is yet to come (an homage to Claude E. Shannon). AIP Conference Proceedings, No. 617, pp doi: / Sperber, D. & Wilson, D. (1986). Relevance: Communication and cognition. Cambridge, MA: Harvard University Press. Warner, J. (2007). Analogies between linguistics and information theory. Journal of the American Society for Information Science and Technology, 58(3), doi: /asi Weaver, W. (1970). Scene of change. New York, NY: Charles Scribner s Sons. Zunde, P. (1981). Information theory and information science. Information Processing & Management 17(6), doi: / (81)