Information Theory: the Day after Yesterday

: the Day after Yesterday Department of Electrical Engineering and Computer Science Chicago s Shannon Centennial Event September 23, 2016

: the Day after Yesterday IT today

Outline The birth of information theory; Applications; The theory today; An outlook.

Communication systems existing in mid 20th century telegraph (1830s) Morse code: A. _ B _... C _. _ telephone (Bell 1876) wireless telegraph (Marconi, 1887) AM radio (early 1900s) television (1925 1927) frequency modulation (FM) (Armstrong, 1936) pulse-coded modulation (PCM) (Reeves, 1937 1939) vocoder (Dudley, 1939) spread spectrum (1940s) Known techniques: efficient encoding of text, understanding of bandwidth, digital vs. continuous-time signaling, tradeoff between fidelity and bandwidth.

Fundamental questions Nyquist, Certain factors affecting telegraph speed, 1924. W log(the number of signal levels) How much improvement in telegraphy trasmission rate could be achieved by replacing the Morse code by an optimum code? Hartley, Transmission of information, 1928. The capacity of a channel is proportional to its bandwith. What is the maximum telegraph signaling speed sustainable by bandlimited linear systems? Answered by the sampling theorem (Küpfmüller 1924, Nyquist 1928, Kotelnikov 1933, J. Whittaker 1915)

Inception of a unifying theory Excerpt of a letter from Claude Shannon to Vannevar Bush on Feb. 16, 1939 [Library of Congress]:

The Bell System Technical Journal Vol. XXVII July, 194S No. 3 THE A Mathematical Theory of Communication By C. E. SHANNON Introduction recent development of various methods of modulation such as PCM and PPM which exchange bandwidth for signal-to-noise ratio has intensified the interest in a general theory of communication. A basis for such a theory is contained in the important papers of Nyquist 1 and Hartley 2 on this subject. In the present paper we will extend the theory to include a number of new factors, in particular the effect of noise in the channel, and the savings possible due to the statistical structure of the original message and due to the nature of the final destination of the information. The fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at another point. Frequently the messages have meaning; that is they refer to or are correlated according to some system with certain physical or conceptual entities. These semantic aspects of communication are irrelevant to the engineering problem. The significant aspect is that the actual message is one selected from a set of possible messages. The system must be designed to operate for each possible selection, not just the one which will actually be chosen since this is unknown at the time of design. If the number of messages in the set is finite then this number or any monotonic function of this number can be regarded as a measure of the inis chosen from the set, all choices

2. A transmitter which operates on the message in some way to produce a Outline Yesterday Today: applications Today: theory Outlook signal suitable for transmission over the channel. In telephony this operation consists merely of changing sound pressure into a proportional electrical current. Shannon s abstraction In telegraphy we have an encoding operation which produces a sequence of dots, dashes and spaces on the channel corresponding to the message. In a multiplex PCM system the different speech functions must be sampled, compressed, quantized and encoded, and finally interleaved INFORMATION SOURCE TRANSMITTER RECEIVER DESTINATION MESSAGE SIGNAL O RECEIVED SIGNAL NOISE SOURCE Fig. 1 Schematic diagram of a general communication system. properly to construct the signal. Vocoder systems, television, and frequency modulation are other examples of complex operations applied Source as random process (Shannon worked on cryptography); the message to obtain the signal. Channel 3. Themodeled channel is merely as a the random medium transformation; used to transmit the signal from Related transmitter work: to receiver. It may be a pair of wires, a coaxial cable, a band of radio frequencies, a beam of light, etc. N. Wiener, Extrapolation, interpolation, and smoothing of 4. The receiver ordinarily performs the inverse operation of that done by thestationary transmitter, reconstructing time series, the1949; message from the signal. 5. The destination is person (or thing) for whom the message in- S. O. Rice, Mathematical analysis of random noise, 1952. tended. We wish to consider certain general problems involving communication systems. To do this it is first necessary to represent the various elements

Shannon s theorems Theorem (Lossless source coding) H < R Theorem (Channel coding) R < C Theorem (Source channel separation) H < C

Roles played by IT since 1948 IT predicts the fundamental limits lossless data compression lossy data compression channel coding network coding signal processing statistical inference complexity theory portfolio theory IT as a design driver IT as a foundation of sciences and engineering

Lossless data compression Huffman coding (a component in JPEG, MP3, MPEG-4)

Universal lossless data compression Softwares based on the Lempel-Ziv algorithm:

Lossy data compression Examples: mp3, JPEG, MPEG-4.

Channel coding: deep-space communication Over 100 million km away.

Channel coding: modem Example: trellis codes Also, CRC for the internet

Channel coding: Compact Disk (CD) Reed-Solomon codes

Channel capacity: WiFi and multiple antennas Also, space-time codes.

Source and channel coding: cellular networks Vocoder (speech bit stream); modem (bit stream waveform); Also, MIMO, OFDM, CDMA, multiuser detection.

What is IT? theorems about the fundamental limits algorithms for achieving/approaching those limits people who call themselves information theorists and develop those theorems and algorithms

Editorial areas: coding techniques coding theory communications communication networks complexity cryptography detection and estimation machine learning probability and statistics quantum information theory sequences Shannon theory signal processing source coding statistical learning

Single-user information theory Efficient lossless/lossy codes for ergodic sources; Efficient capacity-achieving codes for ergodic channels; Capacity of Gaussian MIMO channels generally known; Capacity of certain quantum channels known;...

Finite-blocklength channel coding rate 2328 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 56, NO. 5, MAY 2010 Fig. 7. Bounds for the AWGN channel, SNR =20dB, =10. R(n, ɛ) = C n 1 V Q 1 (ɛ) + O(n 1 log n). asymptotic expansion of. In Figs. 6 and 7 we can maximal probability of error, whereas the RCU bound requires see that the bound is also quite competitive for finite. further manipulation (e.g., Appendix A). Comparing the bound and the classical bounds of Feinstein and Gallager, we see that, as expected, the bound is uniformly better than Feinstein s bound. In the setup of Fig. 6, the We turn to the asymptotic analysis of the maximum achiev- IV. NORMAL APPROXIMATION bound is a significant improvement over Gallager s bound, able rate for a given blocklength. In this section, our goal is coming very close to the Shannon bound as well as the con- to show a normal-approximation refinement of the coding the-

Almost lossless analog compression I.i.d. (analog) source X 1,..., X n, with Renyi information dimension d(x). Encoder Decoder Minimum ɛ-achievable rate linear Borel R (ɛ) = d(x) continuous continuous R 0 (ɛ) = 0 Borel Lipschitz R(ɛ) = d(x)

Multi-user information theory extension of Shannon s basic theorems to networks with multiple sources and receivers; new ingredients: interference, cooperation, side information; multiaccess channel capacity known; degraded broadcast channel capacity known; capacity of Gaussian MIMO broadcast channel known; capacity of Gaussian interference channel well approximated; lossy compression of correlated sources partially solved; degrees of freedom of MIMO interference channels known; fundamental limits of caching;...

The new many-user regime k = 1, n classical single-user IT k fixed, n multiuser IT k after n Large-system analysis k, n many-user IT (with application to the IoT)

Foundation for thermal physics?

Foundation for evolution, neuroscience? Examples: evolution and information acquisition; understanding neural spikes.

Outline Yesterday Today: applications Today: theory Outlook Foundation for economics?

Use of entropy, mutual information, relative entropy probability and statistics complexity theory computational theory biostatistics machine learning physics chemistry economics neuroscience...

Open problems: single-user exact throughput-delay-reliability trade-off (non-asymptotics); the channel reliability function; capacity of channels with memory; deletions, insertions, synchronization; joint source-channel coding;...

Open problems: multiuser problems multiple description of sources; broadcast channel capacity; interference channel capacity; two-way channel capacity; relay channel capacity; capacity of multiuser channels with feedback;...

Match Overview Open problem: mobile ad hoc networks One node per network serves as a gateway Collaboration takes place over internet-like infrastructure connected to the gateway (models realistic internets) DARPA Spectrum Collaboration Challenge (2016 2019) Team 1 Team 2 Team 3 Team 4 Team 5 IP Traffic Ensemble of up to 5 teams placed in arena Incumbent Each node is given IP traffic Sources and destinations are contained in the same network Traffic will emulate multiple canonical types Arena may also contain other Non-Collaborative Radios (NCR): Incumbents Jammers Radio environment emulated in real-time: Large-scale path loss Multipath & Doppler Channel correlation Motion Email questions to darpa-baa-16-48@darpa.mil Distribution A. Approved for public release 27

IT is a way of thinking Like Shannon, we should always question what are the fundamental limits; We then invent schemes to achieve those limits; New progress is often made when we challenge existing constraints and assumptions (network coding, full duplex,...).

What about Shannon s premises? The fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at another point. Frequently the messages have meaning; that is they refer to or are correlated according to some system with certain physical or conceptual entities. These semantic aspects of communication are irrelevant to the engineering problem. The significant aspect is that the actual message is one selected from a set of possible messages. The system must be designed to operate for each possible selection, not just the one which will actually be chosen since this is unknown at the time of design.