Network Information Theory

Transcription

1 1 / 191 Network Information Theory Young-Han Kim University of California, San Diego Joint work with Abbas El Gamal (Stanford) IEEE VTS San Diego 2009

2 2 / 191 Network Information Flow Consider a general networked information processing system: Source Network Node Sources: data, speech, music, images, video, sensor data Nodes: handsets, base stations, servers, sensor nodes Network: wired, wireless, or hybrid

3 3 / 191 Network Information Flow Each node observes some sources, wishes to obtain descriptions of other sources, or to compute function/make decision based on them

4 4 / 191 Network Information Flow Each node observes some sources, wishes to obtain descriptions of other sources, or to compute function/make decision based on them To achieve goal, the nodes communicate and perform local computing

5 5 / 191 Network Information Flow Each node observes some sources, wishes to obtain descriptions of other sources, or to compute function/make decision based on them To achieve goal, the nodes communicate and perform local computing Information flow questions: What are the necessary and sufficient conditions on information flow? What are the optimal coding schemes/techniques to achieve them?

6 6 / 191 Network Information Flow Each node observes some sources, wishes to obtain descriptions of other sources, or to compute function/make decision based on them To achieve goal, the nodes communicate and perform local computing Information flow questions: What are the necessary and sufficient conditions on information flow? What are the optimal coding schemes/techniques to achieve them? Shannon answered these questions for point-point-communication Ford Fulkerson and Elias Feinstein Shannon answered them for noiseless unicast network

7 7 / 191 Noiseless Unicast Network Consider a noiseless network modeled by directed graph (N, E) with link capacities C jk bits/transmission 2 j C jk k M 1 C 12 C 14 4 N M C 13 3 Source node 1 wishes to send message M to destination node N What is the capacity (highest transmission rate)?

8 Max-Flow Min-Cut Theorem 2 S j S c C jk k C 12 M 1 C 14 4 N M C 13 3 Max-flow min-cut theorem (Ford Fulkerson 1956, Elias Feinstein Shannon 1956) where C(S) is capacity of cut S C = min S N:1 S,N S c C(S) bits/transmission, 8 / 191

9 Max-Flow Min-Cut Theorem 2 S j S c C jk k C 12 M 1 C 14 4 N M C 13 3 Max-flow min-cut theorem (Ford Fulkerson 1956, Elias Feinstein Shannon 1956) where C(S) is capacity of cut S C = min S N:1 S,N S c C(S) bits/transmission, Capacity is achieved error-free using only forwarding 9 / 191

10 10 / 191 Noisy Communication Channel Consider a point-to-point communication channel modeled by a discrete memoryless channel (X, p(y x), Y) M X p(y x) Y M Sender X wishes to send a message M to receiver Y

11 11 / 191 Noisy Communication Channel Consider a point-to-point communication channel modeled by a discrete memoryless channel (X, p(y x), Y) M X p(y x) Y M Sender X wishes to send a message M to receiver Y Cannot in general achieve error-free transmission Need sophisticated coding to achieve nonzero rate with arbitrarily small probability of error

12 12 / 191 Channel Capacity Setup (Shannon 1948): M X n Y n Encoder p(y x) Decoder M (2 nr,n) block code consists of: Encoder: Assigns codeword x n (m) to each message m [1 : 2 nr ] Decoder: Assigns estimate m [1 : 2 nr ] to each received sequences y n

13 13 / 191 Channel Capacity Setup (Shannon 1948): M X n Y n Encoder p(y x) Decoder M (2 nr,n) block code consists of: Encoder: Assigns codeword x n (m) to each message m [1 : 2 nr ] Decoder: Assigns estimate m [1 : 2 nr ] to each received sequences y n Assume M picked at random from [1 : 2 nr ]

14 14 / 191 Channel Capacity Setup (Shannon 1948): M X n Y n Encoder p(y x) Decoder M (2 nr,n) block code consists of: Encoder: Assigns codeword x n (m) to each message m [1 : 2 nr ] Decoder: Assigns estimate m [1 : 2 nr ] to each received sequences y n Assume M picked at random from [1 : 2 nr ] Average probability of error: P (n) e = P{ M M}

15 15 / 191 Channel Capacity Setup (Shannon 1948): M X n Y n Encoder p(y x) Decoder M (2 nr,n) block code consists of: Encoder: Assigns codeword x n (m) to each message m [1 : 2 nr ] Decoder: Assigns estimate m [1 : 2 nr ] to each received sequences y n Assume M picked at random from [1 : 2 nr ] Average probability of error: P (n) e = P{ M M} Rate R achievable if there exists sequence of codes with P (n) e 0

16 16 / 191 Channel Capacity Setup (Shannon 1948): M X n Y n Encoder p(y x) Decoder M (2 nr,n) block code consists of: Encoder: Assigns codeword x n (m) to each message m [1 : 2 nr ] Decoder: Assigns estimate m [1 : 2 nr ] to each received sequences y n Assume M picked at random from [1 : 2 nr ] Average probability of error: P (n) e = P{ M M} Rate R achievable if there exists sequence of codes with P (n) e 0 What is the capacity (highest transmission rate)?

17 17 / 191 Channel Capacity Setup (Shannon 1948): M X n Y n Encoder p(y x) Decoder M Channel coding theorem (Shannon 1948) where C = max p(x) I(X;Y) bits/transmission, I(X;Y) = E X,Y log p(x,y) p(x)p(y) is the mutual information between X and Y

18 18 / 191 Shannon s Point-to-Point Information Theory Channel coding theorem Capacity C = maxi(x;y): maximum rate of reliable data transmission

19 19 / 191 Shannon s Point-to-Point Information Theory Channel coding theorem Capacity C = maxi(x;y): maximum rate of reliable data transmission Lossless source coding theorem Entropy H(X): minimum rate of lossless compression

20 20 / 191 Shannon s Point-to-Point Information Theory Channel coding theorem Capacity C = maxi(x;y): maximum rate of reliable data transmission Lossless source coding theorem Entropy H(X): minimum rate of lossless compression Lossy source coding theorem Rate-distortion function R(D) = min I(X; p( x x): E(d(X, X)) D X): minimum rate of lossy compression at distortion D

21 21 / 191 Shannon s Point-to-Point Information Theory Channel coding theorem Capacity C = maxi(x;y): maximum rate of reliable data transmission Lossless source coding theorem Entropy H(X): minimum rate of lossless compression Lossy source coding theorem Rate-distortion function R(D) = min p( x x): E(d(X, X)) D I(X; X): minimum rate of lossy compression at distortion D Separation theorem It suffices to perform source coding and channel coding separately (a standard digital interface between the source and the channel)

22 22 / 191 From Wired to Wireless Networks These results have been at foundation of wired, static information age

23 23 / 191 From Wired to Wireless Networks These results have been at foundation of wired, static information age Model of network as point-to-point links, dumb nodes fails to capture important aspects of emerging wireless, dynamic information age

24 24 / 191 From Wired to Wireless Networks These results have been at foundation of wired, static information age Model of network as point-to-point links, dumb nodes fails to capture important aspects of emerging wireless, dynamic information age The wireless channel is inherently a shared, broadcast medium, naturally allowing for multicasting, but creating complex tradeoffs between competition and cooperation

25 25 / 191 From Wired to Wireless Networks These results have been at foundation of wired, static information age Model of network as point-to-point links, dumb nodes fails to capture important aspects of emerging wireless, dynamic information age The wireless channel is inherently a shared, broadcast medium, naturally allowing for multicasting, but creating complex tradeoffs between competition and cooperation Emerging applications demand very high data rates, are highly interactive, and distributed

26 26 / 191 Network Information Theory Network information theory extends Shannon s point-to-point communication results and max-flow min-cut theorem to noisy network models with: Multiple sources and destinations Broadcasting Interference Relaying

27 27 / 191 Network Information Theory Network information theory extends Shannon s point-to-point communication results and max-flow min-cut theorem to noisy network models with: Multiple sources and destinations Broadcasting Interference Relaying Goal of the theory Establish limits and tradeoffs on network information flow Find optimal coding techniques, protocols, and network architecture

28 28 / 191 Network Information Theory Network information theory extends Shannon s point-to-point communication results and max-flow min-cut theorem to noisy network models with: Multiple sources and destinations Broadcasting Interference Relaying Goal of the theory Establish limits and tradeoffs on network information flow Find optimal coding techniques, protocols, and network architecture Although a general theory is yet to be developed, several beautiful and potentially useful results have been established

29 29 / 191 Outline: State of the Theory 1 Single-hop Networks 2 Multi-hop Networks 3 Extensions and Applications

30 30 / 191 Single-hop Networks Single-round one-way communication (each node is either a transmitter or a receiver but not both) Independent messages over noiseless channels

31 31 / 191 Single-hop Networks Single-round one-way communication (each node is either a transmitter or a receiver but not both) Independent messages over noiseless channels Independent messages over noisy channels (capacity region)

32 32 / 191 Single-hop Networks Single-round one-way communication (each node is either a transmitter or a receiver but not both) Independent messages over noiseless channels Independent messages over noisy channels (capacity region)

33 33 / 191 Multiple Access Channel M 1 M 2 X 1 X 2 p(y x 1,x 2 ) Y ( M 1, M 2 )

34 34 / 191 Multiple Access Channel M 1 M 2 X 1 X 2 p(y x 1,x 2 ) Y ( M 1, M 2 ) Capacity region (Ahlswede 1971, Liao 1972) Union of pentagons R 1 I(X 1 ;Y X 2,Q), R 2 I(X 2 ;Y X 1,Q), R 1 +R 2 I(X 1,X 2 ;Y Q) R 2 over all p(q)p(x 1 q)p(x 2 q) R 1

35 35 / 191 Multiple Access Channel M 1 M 2 X 1 X 2 p(y x 1,x 2 ) Y ( M 1, M 2 ) Capacity region (Ahlswede 1971, Liao 1972) Union of pentagons R 1 I(X 1 ;Y X 2,Q), R 2 I(X 2 ;Y X 1,Q), R 1 +R 2 I(X 1,X 2 ;Y Q) R 2 over all p(q)p(x 1 q)p(x 2 q) R 1 Can be generalized to any number of transmitters

36 36 / 191 Successive Cancellation AWGN-MAC: Y = g 1 X 1 +g 2 X 2 +Z Z: WGN(1) X j : power constraint P g j : channel gain (SNR S j = g 2 j P) X 1 X 2 g 1 g 2 Z Y

37 37 / 191 Successive Cancellation AWGN-MAC: Y = g 1 X 1 +g 2 X 2 +Z Z: WGN(1) X j : power constraint P g j : channel gain (SNR S j = g 2 j P) X 1 X 2 g 1 g 2 Z Y Capacity region (Cover, Wyner 1975) R 1 C(S 1 ), R 2 C(S 2 ), R 1 +R 2 C(S 1 +S 2 ), R 2 C(S 2 ) C S 2 S 1 +1 C(S): AWGN capacity with SNR S C S 1 S 2 +1 C(S 1 ) R 1

38 38 / 191 Successive Cancellation AWGN-MAC: Y = g 1 X 1 +g 2 X 2 +Z Z: WGN(1) X j : power constraint P g j : channel gain (SNR S j = g 2 j P) X 1 X 2 g 1 g 2 Z Y Capacity region (Cover, Wyner 1975) R 1 C(S 1 ), R 2 C(S 2 ), R 1 +R 2 C(S 1 +S 2 ), R 2 C(S 2 ) C S 2 S 1 +1 Treating other signal as noise C(S): AWGN capacity with SNR S C S 1 S 2 +1 C(S 1 ) R 1

39 39 / 191 Successive Cancellation AWGN-MAC: Y = g 1 X 1 +g 2 X 2 +Z Z: WGN(1) X j : power constraint P g j : channel gain (SNR S j = g 2 j P) X 1 X 2 g 1 g 2 Z Y Capacity region (Cover, Wyner 1975) R 1 C(S 1 ), R 2 C(S 2 ), R 1 +R 2 C(S 1 +S 2 ), R 2 C(S 2 ) C S 2 S 1 +1 TDMA C(S): AWGN capacity with SNR S C S 1 S 2 +1 C(S 1 ) R 1

40 40 / 191 Successive Cancellation AWGN-MAC: Y = g 1 X 1 +g 2 X 2 +Z Z: WGN(1) X j : power constraint P g j : channel gain (SNR S j = g 2 j P) X 1 X 2 g 1 g 2 Z Y Capacity region (Cover, Wyner 1975) R 1 C(S 1 ), R 2 C(S 2 ), R 1 +R 2 C(S 1 +S 2 ), R 2 C(S 2 ) C S 2 S 1 +1 TDMA with power control C(S): AWGN capacity with SNR S C S 1 S 2 +1 C(S 1 ) R 1

41 41 / 191 Successive Cancellation AWGN-MAC: Y = g 1 X 1 +g 2 X 2 +Z Z: WGN(1) X j : power constraint P g j : channel gain (SNR S j = g 2 j P) X 1 X 2 g 1 g 2 Z Y Capacity region (Cover, Wyner 1975) R 1 C(S 1 ), R 2 C(S 2 ), R 1 +R 2 C(S 1 +S 2 ), R 2 C(S 2 ) C S 2 S 1 +1 Successive cancellation C(S): AWGN capacity with SNR S Successive cancellation Decode X 2 from Y = g 2 X 2 +g 1 X 1 +Z Decode X 1 from Y = g 1 X 1 +Z C S 1 S 2 +1 C(S 1 ) R 1

42 42 / 191 Successive Cancellation AWGN-MAC: Y = g 1 X 1 +g 2 X 2 +Z Z: WGN(1) X j : power constraint P g j : channel gain (SNR S j = g 2 j P) X 1 X 2 g 1 g 2 Z Y Capacity region (Cover, Wyner 1975) R 1 C(S 1 ), R 2 C(S 2 ), R 1 +R 2 C(S 1 +S 2 ), R 2 C(S 2 ) C S 2 S 1 +1 Successive cancellation C(S): AWGN capacity with SNR S Successive cancellation Decode X 2 from Y = g 2 X 2 +g 1 X 1 +Z Decode X 1 from Y = g 1 X 1 +Z C S 1 S 2 +1 C(S 1 ) R 1

43 43 / 191 Successive Cancellation AWGN-MAC: Y = g 1 X 1 +g 2 X 2 +Z Z: WGN(1) X j : power constraint P g j : channel gain (SNR S j = g 2 j P) X 1 X 2 g 1 g 2 Z Y Capacity region (Cover, Wyner 1975) R 1 C(S 1 ), R 2 C(S 2 ), R 1 +R 2 C(S 1 +S 2 ), R 2 C(S 2 ) C S 2 S 1 +1 Time sharing C(S): AWGN capacity with SNR S Successive cancellation Decode X 2 from Y = g 2 X 2 +g 1 X 1 +Z Decode X 1 from Y = g 1 X 1 +Z C S 1 S 2 +1 C(S 1 ) R 1

44 44 / 191 Successive Cancellation AWGN-MAC: Y = g 1 X 1 +g 2 X 2 +Z Z: WGN(1) X j : power constraint P g j : channel gain (SNR S j = g 2 j P) X 1 X 2 g 1 g 2 Z Y Capacity region (Cover, Wyner 1975) R 1 C(S 1 ), R 2 C(S 2 ), R 1 +R 2 C(S 1 +S 2 ), R 2 C(S 2 ) C S 2 S 1 +1 Time sharing C(S): AWGN capacity with SNR S C S 1 S 2 +1 C(S 1 ) Successive cancellation Decode X 2 from Y = g 2 X 2 +g 1 X 1 +Z Decode X 1 from Y = g 1 X 1 +Z 3GPP HSPA, EV-DO rev A (Hou Smee Pfister Tomasin 2006) R 1

45 45 / 191 Degraded Broadcast Channel (M 1,M 2 ) X p(y 1,y 2 x) Y 1 Y 2 M 1 M 2 Suppose the channel is degraded (Y 1 has a better channel than Y 2 ) Capacity region (Cover 1972, Bergmans 1973, Gallager 1974) Union of rectangles R 1 I(X;Y 1 U), R 2 I(U;Y 2 ) R 2 over all p(u,x) R 1 Can be generalized to any number of (degraded) receivers The problem is open for a general nondegraded broadcast channel

46 46 / 191 Superposition Coding low rate X(m 1,m 2 ) X(m 1,m 2 ) high rate

47 47 / 191 Superposition Coding cloud center U(m 2 ) X(m 1,m 2 ) cloud

48 48 / 191 Superposition Coding cloud center U(m 2 ) X(m 1,m 2 ) cloud Z 1 N(0,N 1 ) AWGN-BC: Y 1 = X +Z 1, Y 2 = X +Z 2 Degraded: N 1 N 2 X Y 1 Y 2 Z 2 N(0,N 2 )

49 49 / 191 Superposition Coding cloud center U(m 2 ) X(m 1,m 2 ) cloud Z 1 N(0,N 1 ) AWGN-BC: Y 1 = X +Z 1, Y 2 = X +Z 2 Degraded: N 1 N 2 Transmit X(m 1,m 2 ) = V(m 1 )+U(m 2 ) X Y 1 Y 2 Z 2 N(0,N 2 )

50 50 / 191 Superposition Coding cloud center U(m 2 ) X(m 1,m 2 ) cloud Z 1 N(0,N 1 ) AWGN-BC: Y 1 = X +Z 1, Y 2 = X +Z 2 Degraded: N 1 N 2 Transmit X(m 1,m 2 ) = V(m 1 )+U(m 2 ) The worse receiver decodes U(m 2 ) from Y 2 = U +V +Z 2 X Y 1 Y 2 Z 2 N(0,N 2 )

51 51 / 191 Superposition Coding cloud center U(m 2 ) X(m 1,m 2 ) cloud Z 1 N(0,N 1 ) AWGN-BC: Y 1 = X +Z 1, Y 2 = X +Z 2 Degraded: N 1 N 2 Transmit X(m 1,m 2 ) = V(m 1 )+U(m 2 ) The worse receiver decodes U(m 2 ) from Y 2 = U +V +Z 2 The better receiver uses successive cancellation: First decode U(m 2 ) from Y 1 = U +V +Z 1 Then decode V(m 1 ) from Y 1 = V +Z 1 X Y 1 Y 2 Z 2 N(0,N 2 )

52 52 / 191 Superposition Coding cloud center U(m 2 ) X(m 1,m 2 ) cloud Z 1 N(0,N 1 ) AWGN-BC: Y 1 = X +Z 1, Y 2 = X +Z 2 Degraded: N 1 N 2 Transmit X(m 1,m 2 ) = V(m 1 )+U(m 2 ) The worse receiver decodes U(m 2 ) from Y 2 = U +V +Z 2 The better receiver uses successive cancellation: First decode U(m 2 ) from Y 1 = U +V +Z 1 Then decode V(m 1 ) from Y 1 = V +Z 1 Hierarchical modulation: DVB-T, 3GPP2 UMB X Y 1 Y 2 Z 2 N(0,N 2 )

53 53 / 191 Interference Channel M 1 M 2 X 1 X 2 p(y 1,y 2 x 1,x 2 ) Y 1 Y 2 M 1 M 2 Capacity region is not known in general

54 54 / 191 Interference Channel M 1 M 2 X 1 X 2 p(y 1,y 2 x 1,x 2 ) Y 1 Y 2 M 1 M 2 Capacity region is not known in general Basic coding techniques Interference as noise (ignore it) R 2 C(S 2 ) C S 2 I 1 +1? C S 1 I 2 +1 C(S 1) R 1

55 55 / 191 Interference Channel M 1 M 2 X 1 X 2 p(y 1,y 2 x 1,x 2 ) Y 1 Y 2 M 1 M 2 Capacity region is not known in general Basic coding techniques Interference as noise (ignore it) TDMA (avoid it) R 2 C(S 2 ) C S 2 I 1 +1? C S 1 I 2 +1 C(S 1) R 1

56 56 / 191 Interference Channel M 1 M 2 X 1 X 2 p(y 1,y 2 x 1,x 2 ) Y 1 Y 2 M 1 M 2 R 2 Capacity region is not known in general Basic coding techniques Interference as noise (ignore it) TDMA (avoid it) Interference as signal (decode it) C(S 2 ) C S 2 I 1 +1? C S 1 I 2 +1 C(S 1) R 1

57 57 / 191 Interference Channel M 1 M 2 X 1 X 2 p(y 1,y 2 x 1,x 2 ) Y 1 Y 2 M 1 M 2 R 2 Capacity region is not known in general Basic coding techniques Interference as noise (ignore it) TDMA (avoid it) Interference as signal (decode it) C(S 2 ) C S 2 I 1 +1 C S 1 I 2 +1 C(S 1) R 1

58 58 / 191 Interference Channel M 1 M 2 X 1 X 2 p(y 1,y 2 x 1,x 2 ) Y 1 Y 2 M 1 M 2 R 2 Capacity region is not known in general Basic coding techniques Interference as noise (ignore it) TDMA (avoid it) Interference as signal (decode it) Han Kobayashi rate splitting (1981) (M 10,M 11 ) (M 20,M 22 ) X 1 X 2 p(y 1,y 2 x 1,x 2 ) C(S 2 ) C S 2 I 1 +1 Y 1 Y 2 C S 1 I 2 +1 C(S 1) R 1

59 59 / 191 Interference Channel M 1 M 2 X 1 X 2 p(y 1,y 2 x 1,x 2 ) Y 1 Y 2 M 1 M 2 R 2 Capacity region is not known in general Basic coding techniques Interference as noise (ignore it) TDMA (avoid it) Interference as signal (decode it) Han Kobayashi rate splitting (1981) (M 10,M 11 ) (M 20,M 22 ) X 1 X 2 p(y 1,y 2 x 1,x 2 ) C(S 2 ) C S 2 I 1 +1 R 1 C S 1 C(S 1) I 2 +1 Y 1 ( M 10, M 11, M 20 ) Y 2 ( M 20, M 22, M 10 )

60 60 / 191 Interference Channel M 1 M 2 X 1 X 2 p(y 1,y 2 x 1,x 2 ) Y 1 Y 2 M 1 M 2 R 2 Capacity region is not known in general Basic coding techniques Interference as noise (ignore it) TDMA (avoid it) Interference as signal (decode it) Han Kobayashi rate splitting (1981) (M 10,M 11 ) (M 20,M 22 ) X 1 X 2 p(y 1,y 2 x 1,x 2 ) C(S 2 ) C S 2 I 1 +1 R 1 C S 1 C(S 1) I 2 +1 Y 1 ( M 10, M 11, M 20 ) Y 2 ( M 20, M 22, M 10 ) Best known coding scheme (includes all the above) Optimal within 1/2 bit from the capacity region (Etkin Tse Wang 2008)

61 61 / 191 Interference Channel M 1 M 2 X 1 X 2 p(y 1,y 2 x 1,x 2 ) Y 1 Y 2 M 1 M 2 R 2 Capacity region is not known in general Basic coding techniques Interference as noise (ignore it) TDMA (avoid it) Interference as signal (decode it) Han Kobayashi rate splitting (1981) (M 10,M 11 ) (M 20,M 22 ) X 1 X 2 p(y 1,y 2 x 1,x 2 ) C(S 2 ) C S 2 I 1 +1 R 1 C S 1 C(S 1) I 2 +1 Y 1 ( M 10, M 11, M 20 ) Y 2 ( M 20, M 22, M 10 ) Best known coding scheme (includes all the above) Optimal within 1/2 bit from the capacity region (Etkin Tse Wang 2008) Interference alignment (Cadambe Jafar 2008)

62 62 / 191 Channels with State p(s) M X p(y x, s) Y M

63 63 / 191 Channels with State p(s) M X p(y x, s) Y M Channel state models: Uncertainty about channel (compound channel)

64 64 / 191 Channels with State p(s) M X p(y x, s) Y M Channel state models: Uncertainty about channel (compound channel) Jamming (arbitrarily varying channel)

65 65 / 191 Channels with State p(s) M X p(y x, s) Y M Channel state models: Uncertainty about channel (compound channel) Jamming (arbitrarily varying channel) Channel fading

66 66 / 191 Channels with State p(s) M X p(y x, s) Y M Channel state models: Uncertainty about channel (compound channel) Jamming (arbitrarily varying channel) Channel fading Write-Once-Memory (WOM)

67 67 / 191 Channels with State p(s) M X p(y x, s) Y M Channel state models: Uncertainty about channel (compound channel) Jamming (arbitrarily varying channel) Channel fading Write-Once-Memory (WOM) Memory with defects

68 68 / 191 Channels with State p(s) M X p(y x, s) Y M Channel state models: Uncertainty about channel (compound channel) Jamming (arbitrarily varying channel) Channel fading Write-Once-Memory (WOM) Memory with defects Host image in digital watermarking

69 69 / 191 Channels with State p(s) M X p(y x, s) Y M Channel state models: Uncertainty about channel (compound channel) Jamming (arbitrarily varying channel) Channel fading Write-Once-Memory (WOM) Memory with defects Host image in digital watermarking Feedback from the receiver

70 70 / 191 Channels with State p(s) M X p(y x, s) Y M Channel state models: Uncertainty about channel (compound channel) Jamming (arbitrarily varying channel) Channel fading Write-Once-Memory (WOM) Memory with defects Host image in digital watermarking Feedback from the receiver Finite state channels (p(y i,s i x i,s i 1 ))

71 71 / 191 Channels with State p(s) M X p(y x, s) Y M Channel state models: Uncertainty about channel (compound channel) Jamming (arbitrarily varying channel) Channel fading Write-Once-Memory (WOM) Memory with defects Host image in digital watermarking Feedback from the receiver Finite state channels (p(y i,s i x i,s i 1 )) Known interference

72 72 / 191 Writing on Dirty Paper S N(0,Q) Z N(0,1) X Y

73 73 / 191 Writing on Dirty Paper S N(0,Q) Z N(0,1) X Y If nobody knows the state:

74 74 / 191 Writing on Dirty Paper S N(0,Q) Z N(0,1) X Y If nobody knows the state: C = C(P/(1 +Q))

75 75 / 191 Writing on Dirty Paper S N(0,Q) Z N(0,1) X Y If nobody knows the state: C = C(P/(1 +Q)) If the decoder knows the state:

76 76 / 191 Writing on Dirty Paper S N(0,Q) Z N(0,1) X Y If nobody knows the state: C = C(P/(1 +Q)) If the decoder knows the state: C SI D = C(P)

77 77 / 191 Writing on Dirty Paper S N(0,Q) Z N(0,1) X Y If nobody knows the state: C = C(P/(1 +Q)) If the decoder knows the state: C SI D = C(P) Writing on dirty paper (Costa 1983) If the encoder knows the state:

78 78 / 191 Writing on Dirty Paper S N(0,Q) Z N(0,1) X Y If nobody knows the state: C = C(P/(1 +Q)) If the decoder knows the state: C SI D = C(P) Writing on dirty paper (Costa 1983) If the encoder knows the state: C SI E = C(P)

79 79 / 191 Writing on Dirty Paper S N(0,Q) Z N(0,1) X Y If nobody knows the state: C = C(P/(1 +Q)) If the decoder knows the state: C SI D = C(P) Writing on dirty paper (Costa 1983) If the encoder knows the state: C SI E = C(P) Key idea: precoding (cf. Tomlinson Harashima precoding, lattice coding)

80 80 / 191 Writing on Dirty Paper S N(0,Q) Z N(0,1) X Y If nobody knows the state: C = C(P/(1 +Q)) If the decoder knows the state: C SI D = C(P) Writing on dirty paper (Costa 1983) If the encoder knows the state: C SI E = C(P) Key idea: precoding (cf. Tomlinson Harashima precoding, lattice coding) Digital watermarking (Moulin O Sullivan 2003)

81 81 / 191 Writing on Dirty Paper S N(0,Q) Z N(0,1) X Y If nobody knows the state: C = C(P/(1 +Q)) If the decoder knows the state: C SI D = C(P) Writing on dirty paper (Costa 1983) If the encoder knows the state: C SI E = C(P) Key idea: precoding (cf. Tomlinson Harashima precoding, lattice coding) Digital watermarking (Moulin O Sullivan 2003) MIMO broadcast channels

82 Dirty Paper Coding for MIMO Broadcast Channel Z 1 M 1 X 1 (M 1 ) G 1 Y 1 M 1 X M 2 X 2 (X 1,M 2 ) G 2 Y 2 M 2 Z 2 Encoding: X = X 1 +X 2 Regular encoding: X 1 (M 1 ) Dirty paper encoding: X 2 (X 1,M 2 ) 82 / 191

83 Dirty Paper Coding for MIMO Broadcast Channel Z 1 M 1 X 1 (M 1 ) G 1 Y 1 M 1 X M 2 X 2 (X 1,M 2 ) G 2 Y 2 M 2 Z 2 Encoding: X = X 1 +X 2 Regular encoding: X 1 (M 1 ) Dirty paper encoding: X 2 (X 1,M 2 ) Decoding at receiver 1: Y 1 = X 1 +X 2 +Z 1 83 / 191

84 84 / 191 Dirty Paper Coding for MIMO Broadcast Channel Z 1 M 1 X 1 (M 1 ) G 1 Y 1 M 1 X M 2 X 2 (X 1,M 2 ) G 2 Y 2 M 2 Z 2 Encoding: X = X 1 +X 2 Regular encoding: X 1 (M 1 ) Dirty paper encoding: X 2 (X 1,M 2 ) Decoding at receiver 1: Y 1 = X 1 +X 2 +Z 1 Decoding at receiver 2: Y 2 = X 2 +X 1 +Z 2

85 85 / 191 Dirty Paper Coding for MIMO Broadcast Channel Z 1 M 1 X 1 (M 1 ) G 1 Y 1 M 1 X M 2 X 2 (X 1,M 2 ) G 2 Y 2 M 2 Z 2 Encoding: X = X 1 +X 2 Regular encoding: X 1 (M 1 ) Dirty paper encoding: X 2 (X 1,M 2 ) Decoding at receiver 1: Y 1 = X 1 +X 2 +Z 1 Decoding at receiver 2: Y 2 = X 2 +X 1 +Z 2 State information (interference) X 1 at the transmitter Effective channel: Y 2 = X 2 +Z 2

86 86 / 191 Dirty Paper Coding for MIMO Broadcast Channel Z 1 M 1 X 1 (M 1 ) G 1 Y 1 M 1 X M 2 X 2 (X 1,M 2 ) G 2 Y 2 M 2 Z 2 Encoding: X = X 1 +X 2 Regular encoding: X 1 (M 1 ) Dirty paper encoding: X 2 (X 1,M 2 ) Decoding at receiver 1: Y 1 = X 1 +X 2 +Z 1 Decoding at receiver 2: Y 2 = X 2 +X 1 +Z 2 State information (interference) X 1 at the transmitter Effective channel: Y 2 = X 2 +Z 2 Optimal (Weingarten Steinberg Shamai 2006, Mohseni Cioffi 2006)

87 87 / 191 Dirty Paper Coding for MIMO Broadcast Channel Z 1 M 1 X 1 (M 1 ) G 1 Y 1 M 1 X M 2 X 2 (X 1,M 2 ) G 2 Y 2 M 2 Z 2 Encoding: X = X 1 +X 2 Regular encoding: X 1 (M 1 ) Dirty paper encoding: X 2 (X 1,M 2 ) Decoding at receiver 1: Y 1 = X 1 +X 2 +Z 1 Decoding at receiver 2: Y 2 = X 2 +X 1 +Z 2 State information (interference) X 1 at the transmitter Effective channel: Y 2 = X 2 +Z 2 Optimal (Weingarten Steinberg Shamai 2006, Mohseni Cioffi 2006) Special case of Marton s coding scheme (1979) for the general BC

88 88 / 191 Single-hop Networks Single-round one-way communication (each node is either a transmitter or a receiver but not both) Independent messages over noiseless channels Independent messages over noisy channels (capacity region) Correlated sources over noiseless channels (rate distortion region)

89 89 / 191 Single-hop Networks Single-round one-way communication (each node is either a transmitter or a receiver but not both) Independent messages over noiseless channels Independent messages over noisy channels (capacity region) Correlated sources over noiseless channels (rate distortion region)

90 90 / 191 Distributed Lossless Source Coding X 1 X 2 Encoder Encoder M 1 Decoder ( X 1, X 2 ) M 2 Sources are separately encoded and transmitted to a common decoder

91 91 / 191 Distributed Lossless Source Coding X 1 X 2 Encoder Encoder M 1 Decoder ( X 1, X 2 ) M 2 Sources are separately encoded and transmitted to a common decoder With cooperation, the sum rate R 1 +R 2 = H(X 1,X 2 ) is optimal

92 92 / 191 Distributed Lossless Source Coding X 1 X 2 Encoder Encoder M 1 Decoder ( X 1, X 2 ) M 2 Sources are separately encoded and transmitted to a common decoder With cooperation, the sum rate R 1 +R 2 = H(X 1,X 2 ) is optimal Can we achieve the same rate with distributed encoders?

93 93 / 191 Distributed Lossless Source Coding X 1 X 2 Encoder Encoder M 1 Decoder ( X 1, X 2 ) M 2 Sources are separately encoded and transmitted to a common decoder Slepian Wolf theorem (1973) The optimal rate region R 1 H(X 1 X 2 ), R 2 H(X 2 X 1 ), R 1 +R 2 H(X 1,X 2 ) R 2 H(X 2 ) H(X 2 X 1 ) H(X 1 X 2 ) H(X 1 ) R 1 Hashing-based compression: rsync

94 Distributed Lossless Source Coding X 1 X 2 Encoder Encoder M 1 Decoder ( X 1, X 2 ) M 2 Sources are separately encoded and transmitted to a common decoder Slepian Wolf theorem (1973) The optimal rate region R 1 H(X 1 X 2 ), R 2 H(X 2 X 1 ), R 1 +R 2 H(X 1,X 2 ) R 2 H(X 2 ) H(X 2 X 1 ) H(X 1 X 2 ) H(X 1 ) R 1 Hashing-based compression: rsync Distributed lossy source coding (Berger Tung 1978) 94 / 191

95 95 / 191 Multiple Description Coding Encoder M 1 Decoder ( X 1,D 1 ) X Decoder ( X 0,D 0 ) Encoder M 2 Decoder ( X 2,D 2 ) Streaming with packet loss or QoS

96 96 / 191 Multiple Description Coding Encoder M 1 Decoder ( X 1,D 1 ) X Decoder ( X 0,D 0 ) Encoder M 2 Decoder ( X 2,D 2 ) Streaming with packet loss or QoS

97 97 / 191 Multiple Description Coding Encoder M 1 Decoder ( X 1,D 1 ) X Decoder ( X 0,D 0 ) Encoder M 2 Decoder ( X 2,D 2 ) Streaming with packet loss or QoS Rate distortion region is not known in general

98 98 / 191 Multiple Description Coding Encoder M 1 Decoder ( X 1,D 1 ) X Decoder ( X 0,D 0 ) Encoder M 2 Streaming with packet loss or QoS Rate distortion region is not known in general Successive refinement Multi-layer video coding Interlaced GIF

99 99 / 191 Multiple Description Coding Encoder M 1 Decoder ( X 1,D 1 ) X Decoder ( X 0,D 0 ) Encoder M 2 Streaming with packet loss or QoS Rate distortion region is not known in general Successive refinement Multi-layer video coding Interlaced GIF

100 100 / 191 Multiple Description Coding Encoder M 1 Decoder ( X 1,D 1 ) X Decoder ( X 0,D 0 ) Encoder M 2 Streaming with packet loss or QoS Rate distortion region is not known in general Successive refinement Multi-layer video coding Interlaced GIF

101 101 / 191 Multiple Description Coding Encoder M 1 Decoder ( X 1,D 1 ) X Decoder ( X 0,D 0 ) Encoder M 2 Streaming with packet loss or QoS Rate distortion region is not known in general Successive refinement Multi-layer video coding Interlaced GIF

102 102 / 191 Multiple Description Coding Encoder M 1 Decoder ( X 1,D 1 ) X Decoder ( X 0,D 0 ) Encoder M 2 Streaming with packet loss or QoS Rate distortion region is not known in general Successive refinement Multi-layer video coding Interlaced GIF

103 103 / 191 Single-hop Networks Single-round one-way communication (each node is either a transmitter or a receiver but not both) Independent messages over noiseless channels Independent messages over noisy channels (capacity region) Correlated sources over noiseless channels (rate distortion region) Correlated sources over noisy channels

104 104 / 191 Joint Source Channel Coding Measurement collection over sensor networks U 1 U 2 X 1 X 2 p(y x 1,x 2 ) Y (Û 1,Û 2 )

105 105 / 191 Joint Source Channel Coding Measurement collection over sensor networks U 1 U 2 X 1 X 2 p(y x 1,x 2 ) Y (Û 1,Û 2 ) Broadcasting multi-layer video (U 1,U 2 ) X p(y 1,y 2 x) Y 1 Y 2 Û 1 Û 2

106 106 / 191 Joint Source Channel Coding Measurement collection over sensor networks U 1 U 2 X 1 X 2 p(y x 1,x 2 ) Y (Û 1,Û 2 ) Broadcasting multi-layer video (U 1,U 2 ) X p(y 1,y 2 x) Y 1 Y 2 Û 1 Û 2 Sufficient and necessary conditions for reliable transmission

107 107 / 191 Joint Source Channel Coding Measurement collection over sensor networks U 1 U 2 X 1 X 2 p(y x 1,x 2 ) Y (Û 1,Û 2 ) Broadcasting multi-layer video (U 1,U 2 ) X p(y 1,y 2 x) Y 1 Y 2 Û 1 Û 2 Sufficient and necessary conditions for reliable transmission Problem open in general

108 108 / 191 Joint Source Channel Coding Measurement collection over sensor networks U 1 U 2 X 1 X 2 p(y x 1,x 2 ) Y (Û 1,Û 2 ) Broadcasting multi-layer video (U 1,U 2 ) X p(y 1,y 2 x) Y 1 Y 2 Û 1 Û 2 Sufficient and necessary conditions for reliable transmission Problem open in general Source channel separation is suboptimal: joint source channel coding

109 109 / 191 Joint Source Channel Coding Measurement collection over sensor networks U 1 U 2 X 1 X 2 p(y x 1,x 2 ) Y (Û 1,Û 2 ) Broadcasting multi-layer video (U 1,U 2 ) X p(y 1,y 2 x) Y 1 Y 2 Û 1 Û 2 Sufficient and necessary conditions for reliable transmission Problem open in general Source channel separation is suboptimal: joint source channel coding Common randomness between distributed sources

110 110 / 191 Multi-hop Networks Multi-round two-way communication (relaying and interaction) (each node can be both a transmitter and a receiver)

111 111 / 191 Multi-hop Networks Multi-round two-way communication (relaying and interaction) (each node can be both a transmitter and a receiver) Independent messages over noiseless networks

112 112 / 191 Noiseless Single-Source Network Consider noiseless network modeled by graph (N, E) 2 j M j M 1 C 12 C 14 4 N M N C 13 3 M k Node 1 wishes to multicast M to set of destination nodes D k

113 113 / 191 Noiseless Single-Source Network Consider noiseless network modeled by graph (N, E) 2 j M j M 1 C 12 C 14 4 N M N C 13 3 M k Node 1 wishes to multicast M to set of destination nodes D Cutset upper bound C min j D min S N,1 S,j S c C(S) k

114 114 / 191 Noiseless Single-Source Network Consider noiseless network modeled by graph (N, E) 2 j M j M 1 C 12 C 14 4 N M N C 13 3 M k Node 1 wishes to multicast M to set of destination nodes D Cutset upper bound C min j D min S N,1 S,j S c C(S) Achieved via forwarding if D = {N} or D = [2 : N] k

115 115 / 191 Noiseless Single-Source Network Consider noiseless network modeled by graph (N, E) 2 j M j M 1 C 12 C 14 4 N M N C 13 3 M k Node 1 wishes to multicast M to set of destination nodes D Cutset upper bound C min j D min S N,1 S,j S c C(S) Achieved via forwarding if D = {N} or D = [2 : N] In general cutset bound is not achievable by forwarding k

116 116 / 191 Butterfly network Assume C jk = 1 for all links 2 6 M 6 M 1 M 1 M 1 (M 1,M 2 ) 4 5 (M 1,M 2 ) M 2 M 2 (M 1,M 2 ) 3 7 M 7

117 117 / 191 Butterfly network Assume C jk = 1 for all links 2 6 M 6 M 1 M 1 M 1 (M 1,M 2 ) 4 5 (M 1,M 2 ) M 2 M 2 (M 1,M 2 ) 3 7 M 7 Cutset bound C 2

118 118 / 191 Butterfly network Assume C jk = 1 for all links 2 6 M 6 M 1 M 1 M 1 (M 1,M 2 ) 4 5 (M 1,M 2 ) M 2 M 2 (M 1,M 2 ) 3 7 M 7 Cutset bound C 2 Forwarding rate R 1

119 119 / 191 Butterfly network Can achieve cutset bound, i.e., C = 2 2 M 1 6 M 6 M 1 M 1 M 1 M 1 M M 1 M 2 M 2 M 2 M 1 M 2 M M 7

120 120 / 191 Butterfly network Can achieve cutset bound, i.e., C = 2 2 M 1 6 M 6 M 1 M 1 M 1 M 1 M M 1 M 2 M 2 M 2 M 1 M 2 M M 7 Network coding theorem (Ahlswede Cai Li Yeung 2000) Cutset bound is always achievable for noiseless multicast network

121 121 / 191 Butterfly network Can achieve cutset bound, i.e., C = 2 2 M 1 6 M 6 M 1 M 1 M 1 M 1 M M 1 M 2 M 2 M 2 M 1 M 2 M M 7 Network coding theorem (Ahlswede Cai Li Yeung 2000) Cutset bound is always achievable for noiseless multicast network Linear network coding (Koetter Medard 2003)

122 122 / 191 Butterfly network Can achieve cutset bound, i.e., C = 2 2 M 1 6 M 6 M 1 M 1 M 1 M 1 M M 1 M 2 M 2 M 2 M 1 M 2 M M 7 Network coding theorem (Ahlswede Cai Li Yeung 2000) Cutset bound is always achievable for noiseless multicast network Linear network coding (Koetter Medard 2003) XORs in the air (Katti Rahul Hu Katabi Medard Crowcroft 2006)

123 Noiseless Multi-source Multicast Network Consider noiseless network modeled by graph (N, E) M 2 2 j ( M 1j, M 2j, M 3j ) C 12 M 1 1 C 14 4 N ( M 1N, M 2N, M 3N ) C 13 k M 3 3 ( M 1k, M 2k, M 3k ) Source nodes wish to send their messages to destination nodes D 123 / 191

124 Noiseless Multi-source Multicast Network Consider noiseless network modeled by graph (N, E) M 2 2 j ( M 1j, M 2j, M 3j ) C 12 M 1 1 C 14 4 N ( M 1N, M 2N, M 3N ) C 13 k M 3 3 ( M 1k, M 2k, M 3k ) Source nodes wish to send their messages to destination nodes D Again capacity coincides with cutset bound and achieved via network coding (Dana Gowaikar Palanki Hassibi Effros 2006) 124 / 191

125 Noiseless Multi-source Multicast Network Consider noiseless network modeled by graph (N, E) M 2 2 j ( M 1j, M 2j, M 3j ) C 12 M 1 1 C 14 4 N ( M 1N, M 2N, M 3N ) C 13 k M 3 3 ( M 1k, M 2k, M 3k ) Source nodes wish to send their messages to destination nodes D Again capacity coincides with cutset bound and achieved via network coding (Dana Gowaikar Palanki Hassibi Effros 2006) Capacity region of the general nonmulticast network is not known 125 / 191

126 126 / 191 Multi-hop Networks Multi-round two-way communication (relaying and interaction) (each node can be both a transmitter and a receiver) Independent messages over noiseless networks

127 127 / 191 Multi-hop Networks Multi-round two-way communication (relaying and interaction) (each node can be both a transmitter and a receiver) Independent messages over noiseless networks Independent messages over noisy networks

128 128 / 191 Relay Channel Y 2 X 2 M X 1 p(y 2,y 3 x 1,x 2 ) Y 3 M Sender X 1 wishes to send message M to receiver Y 3 with help of relay Capacity is not known in general

129 129 / 191 Relay Channel Y 2 X 2 M X 1 p(y 2,y 3 x 1,x 2 ) Y 3 M Sender X 1 wishes to send message M to receiver Y 3 with help of relay Capacity is not known in general Cutset upper bound (Cover El Gamal 1979) C max p(x,x1 )min I(X,X 1 ;Y),I(X;Y,Y 1 X 1 ) X 1 Y 1 : X 1 X Y X Y

130 130 / 191 Basic Relaying Techniques (Cover El Gamal 1979) Y 2 :X 2 M X 1 Y 3 M

131 131 / 191 Basic Relaying Techniques (Cover El Gamal 1979) Y 2 :X 2 M X 1 Y 3 M Direct transmission (no interface)

132 132 / 191 Basic Relaying Techniques (Cover El Gamal 1979) Y 2 :X 2 M X 1 Y 3 M Direct transmission (no interface) Decode forward (digital digital interface)

133 133 / 191 Basic Relaying Techniques (Cover El Gamal 1979) Y 2 :X 2 M X 1 Y 3 M Direct transmission (no interface) Decode forward (digital digital interface)

134 134 / 191 Basic Relaying Techniques (Cover El Gamal 1979) Y 2 :X 2 M X 1 Y 3 M Direct transmission (no interface) Decode forward (digital digital interface)

135 135 / 191 Basic Relaying Techniques (Cover El Gamal 1979) Y 2 :X 2 Ŷ 2 M X 1 Y 3 M Direct transmission (no interface) Decode forward (digital digital interface) Compress forward (analog digital interface)

136 136 / 191 Basic Relaying Techniques (Cover El Gamal 1979) αy 2 M X 1 Y 3 M Direct transmission (no interface) Decode forward (digital digital interface) Compress forward (analog digital interface) Amplify forward (analog analog interface)

137 137 / 191 Basic Relaying Techniques (Cover El Gamal 1979) Y 2 :X 2 M X 1 Y 3 M Direct transmission (no interface) Decode forward (digital digital interface) Compress forward (analog digital interface) Amplify forward (analog analog interface) Partial decode forward (direct transmission + decode forward)

138 138 / 191 Basic Relaying Techniques (Cover El Gamal 1979) Y 2 :X 2 M X 1 Y 3 M Direct transmission (no interface) Decode forward (digital digital interface) Compress forward (analog digital interface) Amplify forward (analog analog interface) Partial decode forward (direct transmission + decode forward) Partial decode forward + compress forward

139 139 / 191 Basic Relaying Techniques (Cover El Gamal 1979) Y 2 :X 2 M X 1 Y 3 M Direct transmission (no interface) Decode forward (digital digital interface) Compress forward (analog digital interface) Amplify forward (analog analog interface) Partial decode forward (direct transmission + decode forward) Partial decode forward + compress forward Linear relaying

140 140 / 191 Interactive Communication M 1 Y i 1 Encoder 1 X 1i p(y x 1,x 2 ) Y i Decoder ( M 1, M 2 ) M 2 Encoder 2 Y i 1 X 2i Free, noiseless, instantaneous output feedback

141 141 / 191 Interactive Communication M 1 Y i 1 Encoder 1 X 1i p(y x 1,x 2 ) Y i Decoder ( M 1, M 2 ) M 2 Encoder 2 Y i 1 X 2i Free, noiseless, instantaneous output feedback First step towards two-way communication

142 142 / 191 Role of Feedback in Communication Iterative refinement: Simpler coding (Schalkwijk Kailath 1966, Horstein 1963, Weldon 1963, Shayevitz Feder 2009)

143 143 / 191 Role of Feedback in Communication Iterative refinement: Simpler coding (Schalkwijk Kailath 1966, Horstein 1963, Weldon 1963, Shayevitz Feder 2009) Feedback can greatly improve the decoding error probability ARQ, hybrid ARQ

144 144 / 191 Role of Feedback in Communication Iterative refinement: Simpler coding (Schalkwijk Kailath 1966, Horstein 1963, Weldon 1963, Shayevitz Feder 2009) Feedback can greatly improve the decoding error probability ARQ, hybrid ARQ Feedback can increase the capacity of channels with memory (Butman 1969, Kim 2010) Channel state feedback (block fading)

145 145 / 191 Role of Feedback in Communication Iterative refinement: Simpler coding (Schalkwijk Kailath 1966, Horstein 1963, Weldon 1963, Shayevitz Feder 2009) Feedback can greatly improve the decoding error probability ARQ, hybrid ARQ Feedback can increase the capacity of channels with memory (Butman 1969, Kim 2010) Channel state feedback (block fading) Feedback can enlarge the capacity region of multi-user channels (Gaarder Wolf 1975, Cover Leung 1981, Ozarow 1984) Coherent cooperation among transmitters Common state information multicast

146 146 / 191 Role of Feedback in Communication Iterative refinement: Simpler coding (Schalkwijk Kailath 1966, Horstein 1963, Weldon 1963, Shayevitz Feder 2009) Feedback can greatly improve the decoding error probability ARQ, hybrid ARQ Feedback can increase the capacity of channels with memory (Butman 1969, Kim 2010) Channel state feedback (block fading) Feedback can enlarge the capacity region of multi-user channels (Gaarder Wolf 1975, Cover Leung 1981, Ozarow 1984) Coherent cooperation among transmitters Common state information multicast Noisy feedback (Kim Lapidoth Weissman 2008, 2009)

147 147 / 191 Two-Way Channel M 1 X 1 X M p(y 2 2 1,y 2 x 1,x 2 ) M 2 Y 1 Y 2 M 1 Each node wishes to send a message to the other node

148 148 / 191 Two-Way Channel M 1 X 1 X M p(y 2 2 1,y 2 x 1,x 2 ) M 2 Y 1 Y 2 M 1 Each node wishes to send a message to the other node First multi-user channel model (Shannon 1961)

149 149 / 191 Two-Way Channel M 1 X 1 X M p(y 2 2 1,y 2 x 1,x 2 ) M 2 Y 1 Y 2 M 1 Each node wishes to send a message to the other node First multi-user channel model (Shannon 1961) Capacity region of the general TWC is not known

150 150 / 191 Two-Way Channel M 1 X 1 X M p(y 2 2 1,y 2 x 1,x 2 ) M 2 Y 1 Y 2 M 1 Each node wishes to send a message to the other node First multi-user channel model (Shannon 1961) Capacity region of the general TWC is not known Binary multiplier channel (Shannon Blackwell 1961) X 1 = X 2 = {0,1} and Y 1 = Y 2 = Y = X 1 X 2 X 1 {0,1} X 2 {0,1} Y {0,1} Y {0,1}

151 151 / 191 Noisy Multicast Network M j (X j,y j ) M (X 1,Y 1 ) p(y 1,...,y N x 1,...,x N ) (X N,Y N ) M N Node 1 wishes to send message M to subset of receivers Y j, j D

152 152 / 191 Noisy Multicast Network M j (X j,y j ) M (X 1,Y 1 ) p(y 1,...,y N x 1,...,x N ) (X N,Y N ) M N Node 1 wishes to send message M to subset of receivers Y j, j D Capacity is not known in general

153 153 / 191 Cutset Upper Bound S S c (X j,y j ) M (X 1,Y 1 ) p(y 1,...,y N x 1,...,x N ) (X N,Y N ) Cutset upper bound (El Gamal 1981) C max p(x N ) min j D min S:1 S,j S ci(x(s);y(sc ) X(S c ))

154 154 / 191 Cutset Upper Bound S S c (X j,y j ) M (X 1,Y 1 ) p(y 1,...,y N x 1,...,x N ) (X N,Y N ) Cutset upper bound (El Gamal 1981) C max p(x N ) min j D min S:1 S,j S ci(x(s);y(sc ) X(S c )) Tight for noiseless multicast network (network coding theorem)

155 155 / 191 Network Decode Forward Decode forward for relay can be extended to noisy multicast networks Network decode forward lower bound (Aref 1980) C max min j [2:N] I(Xj 1 ;Y j X N j ) p(x N )

156 156 / 191 Network Decode Forward Decode forward for relay can be extended to noisy multicast networks Network decode forward lower bound (Aref 1980) C max min j [2:N] I(Xj 1 ;Y j X N j ) p(x N ) Compare to Cutset upper bound (simpler form) C max min j [2:N] I(Xj 1 ;Y N j X N j ) p(x N )

157 157 / 191 Network Decode Forward Decode forward for relay can be extended to noisy multicast networks Network decode forward lower bound (Aref 1980) Compare to C max min j [2:N] I(Xj 1 ;Y j X N j ) p(x N ) Cutset upper bound (simpler form) C max min j [2:N] I(Xj 1 ;Y N j X N j ) p(x N ) Both bounds coincide for broadcasting (D = [2 : N]) over acyclic networks

158 158 / 191 Network Decode Forward Decode forward for relay can be extended to noisy multicast networks Network decode forward lower bound (Aref 1980) C max min j [2:N] I(Xj 1 ;Y j X N j ) p(x N ) One Laptop per Child, wireless mesh networks (multi-hop)

159 159 / 191 Network Decode Forward Decode forward for relay can be extended to noisy multicast networks Network decode forward lower bound (Aref 1980) C max min j [2:N] I(Xj 1 ;Y j X N j ) p(x N ) One Laptop per Child, wireless mesh networks (multi-hop) Routing becomes the main issue

160 160 / 191 Network Compress Forward Compress forward for relay can be extended to noisy networks Network compress forward lower bound (El Gamal Kim 2009) C maxmin j D min S:1 S,j S c I(X(S);Ŷ(Sc ),Y j X(S c )) I(Y(S);Ŷ(S) X N,Ŷ(S c ),Y j ), where the maximum is over N k=1 p(x k)p(ŷ k y k,x k )

161 161 / 191 Network Compress Forward Compress forward for relay can be extended to noisy networks Network compress forward lower bound (El Gamal Kim 2009) C maxmin j D min S:1 S,j S c I(X(S);Ŷ(Sc ),Y j X(S c )) I(Y(S);Ŷ(S) X N,Ŷ(S c ),Y j ), where the maximum is over N k=1 p(x k)p(ŷ k y k,x k ) Compare to Cutset upper bound (El Gamal 1981) C max p(x N ) min j Dmin S:1 S,j S c I(X(S);Y(S c ) X(S c ))

162 162 / 191 Network Compress Forward Compress forward for relay can be extended to noisy networks Network compress forward lower bound (El Gamal Kim 2009) C maxmin j D min S:1 S,j S c I(X(S);Ŷ(Sc ),Y j X(S c )) I(Y(S);Ŷ(S) X N,Ŷ(S c ),Y j ), where the maximum is over N k=1 p(x k)p(ŷ k y k,x k ) Both bounds coincide for Noiseless network (Ahlswede Cai Li Yeung 2000) Deterministic network with broadcast but no interference (Aref 1980, Ratnakar Kramer 2006) Linear finite-field deterministic network (Avestimehr Diggavi Tse 2009) Wireless erasure network (Dana Gowaikar Palanki Hassibi Effros 2006)

163 163 / 191 Noisy Multi-source Multicast Network M j (X j,y j ) M 1 (X 1,Y 1 ) p(y 1,...,y N x 1,...,x N ) (X 2,Y 2 ) M 2

164 164 / 191 Noisy Multi-source Multicast Network M j (X j,y j ) M 1 (X 1,Y 1 ) p(y 1,...,y N x 1,...,x N ) (X 2,Y 2 ) M 2 Network compress forward generalizes to this case (Lim Kim El Gamal Chung 2010) Single-source multicast network results generalize to multi-source

165 165 / 191 Noisy Multi-source Multicast Network M j (X j,y j ) M 1 (X 1,Y 1 ) p(y 1,...,y N x 1,...,x N ) (X 2,Y 2 ) M 2 Network compress forward generalizes to this case (Lim Kim El Gamal Chung 2010) Single-source multicast network results generalize to multi-source Network compress forward Noisy network coding

166 166 / 191 Wireless Network As a model for wireless networks, consider the AWGN network: Y k = g jk X j +Z k j g jk : channel gain X j : power constraint P Z k : WGN noise

167 167 / 191 Wireless Network As a model for wireless networks, consider the AWGN network: Y k = g jk X j +Z k j g jk : channel gain X j : power constraint P Z k : WGN noise In general, it is extremely hard to find the capacity region, even for simple networks (interference channel, relay channel)

168 168 / 191 Wireless Network As a model for wireless networks, consider the AWGN network: Y k = g jk X j +Z k j g jk : channel gain X j : power constraint P Z k : WGN noise In general, it is extremely hard to find the capacity region, even for simple networks (interference channel, relay channel) Two approaches Bounds on the capacity (eg., network compress forward bound) Scaling laws

169 169 / 191 Wireless Network Scaling Law How does capacity scale in the number of users? N-user MAC: C(N) = O(log(N)) N-user IC: C(N) =?

170 170 / 191 Wireless Network Scaling Law How does capacity scale in the number of users? N-user MAC: C(N) = O(log(N)) N-user IC: C(N) =? 1 2 Random network approach (Gupta Kumar 2000) Randomly placed nodes Study typical behavior of most networks N

171 171 / 191 Wireless Network Scaling Law How does capacity scale in the number of users? N-user MAC: C(N) = O(log(N)) N-user IC: C(N) =? 1 2 Random network approach (Gupta Kumar 2000) Randomly placed nodes Study typical behavior of most networks N Wireless network information theory Effect of the path loss Relaying vs. direct transmission Interference management Protocols and architecture for large wireless networks

172 172 / 191 Wireless Network Scaling Law How does capacity scale in the number of users? N-user MAC: C(N) = O(log(N)) N-user IC: C(N) =? 1 2 Random network approach (Gupta Kumar 2000) Randomly placed nodes Study typical behavior of most networks N Wireless network information theory Effect of the path loss Relaying vs. direct transmission Interference management Protocols and architecture for large wireless networks Wireless ad-hoc networks (Smartdust, RFID)

173 173 / 191 Multi-hop Networks Multi-round two-way communication (relaying and interaction) (each node can be both a transmitter and a receiver) Independent messages over noiseless networks Independent messages over noisy networks Correlated sources over noiseless networks

174 174 / 191 Multi-hop Networks Multi-round two-way communication (relaying and interaction) (each node can be both a transmitter and a receiver) Independent messages over noiseless networks Independent messages over noisy networks Correlated sources over noiseless networks Correlated sources over noisy networks

175 175 / 191 Source Networks Multiple descriptions network 2 X 2 j X j X 1 R 12 R 14 4 N X N R 13 X 4 k 3 X 3 X k

176 176 / 191 Source Networks Multiple descriptions network 2 X 2 j X j X 1 R 12 R 14 4 N X N R 13 X 4 k 3 X 3 X k Two-way source coding M X l (X 1,M l 1 ) 1 X 2 Node1 M Node2 X l+1 (X 2,M l ) 2 X 1

177 177 / 191 Extensions and Applications Communication for computing X 1 X 2 Encoder Encoder M 1 Decoder ĝ(x 1,X 2 ) M 2 Distributed function computation (Orlitsky Roche 2001) communication complexity (Yao 1979) Distributed consensus (gossip) Sensor networks

178 178 / 191 Extensions and Applications Communication for computing Information theoretic secrecy M X p(y, z x) Y Z M I(M;Z) 0 Wiretap channel Distributed key generation Physical layer security

179 179 / 191 Extensions and Applications Communication for computing Information theoretic secrecy Asynchronous communication: M 1l Encoder 1 X 1i Delayd 1 X 1,i d1 p(y x 1,x 2 ) Y i ( M 11, M 2l ) Decoder M 2l Encoder 2 X 2i Delayd 2 X 2,i d2 Asynchronous multiple access Random packet arrivals

180 180 / 191 Network Information Theory Fundamental limits of information flow over networks Coding techniques, protocols, and system architectures

181 181 / 191 Network Information Theory Fundamental limits of information flow over networks Coding techniques, protocols, and system architectures Exciting challenges Many open problems Unification of information theory (statics) and network theory (dynamics) Implementation ideas to bridge the theory and practice

182 182 / 191 Network Information Theory Fundamental limits of information flow over networks Coding techniques, protocols, and system architectures Exciting challenges Many open problems Unification of information theory (statics) and network theory (dynamics) Implementation ideas to bridge the theory and practice Beginning to have impacts on practice

183 183 / 191 To Learn More Lecture Notes on Network Information Theory Top-down organization: background, single-hop, multi-hop, extensions

184 184 / 191 To Learn More Lecture Notes on Network Information Theory Top-down organization: background, single-hop, multi-hop, extensions Comprehensive coverage of key results: 24 chapters, slides

185 185 / 191 To Learn More Lecture Notes on Network Information Theory Top-down organization: background, single-hop, multi-hop, extensions Comprehensive coverage of key results: 24 chapters, slides Balanced introduction of new techniques and models

186 186 / 191 To Learn More Lecture Notes on Network Information Theory Top-down organization: background, single-hop, multi-hop, extensions Comprehensive coverage of key results: 24 chapters, slides Balanced introduction of new techniques and models Unified approach to coding theorems