Shannon Information Theory

Transcription

1 Comm. 5: Communication Theor Lecture Shannon Information Theor -Binar Erasure Channel, Binar Smmetric channel, Channel Caacit, Shannon Caacit

2 Noiseless Channel P P P Noiseless Channel P is the robabilit of theoutut fromthechannel given theinut

3 Nois Channels Source Receiver Channel ={,,, J- } ={,,, K- }

4 Discrete Memorless Channels Channel ={,,, J- } ={,,, K- } Discrete = Finite Alhabet for, Memorless = Current outut smbol deends onl on current inut smbol and not an of the revious ones.

5 Transition Probabilities Channel Matri ={,,, J- } ={,,, K- } Transition Probabilities:, P K K- J- J- K- J- Channel Matri or Transition Matri K,

6 Prior & Joint Probabilities Prior Probabilit Pr Joint Probabilit, Pr,= Pr = Pr

7 Marginal Probabilities The marginal robabilit of the outut random variable is obtained b averaging out the deendence of on as: This equation states that if we are given the inut a riori robabilities and the channel matri, then we ma calculate the robabilities of various outut smbols.,,,..., Pr Pr Pr J J K for

8 Classical Channels E Binar smmetric Channel Erasure Channel

9 Eamle: Binar Smmetric Channel - = = = - = It is secial case of Discrete Memor less Channel with J=K=. The Channel is smmetric because the robabilit of receiving given a zero is sent is the same as the robabilit of receiving a zero if a one is sent. This conditional robabilit of error is denoted b.

10 Transition Matri of Binar Smmetric Channel = = = = - - Transition Probabilit Diagram for a Binar Smmetric Channel Matri Channel K J

11 Erasure Channel BEC In BEC, a transmitter sends a a bit a zero or a one, and the receiver either receives the bit or it receives a message that the bit was not received "erased". Eas obtainable when feedbac! = q q -q = =e, erasure,,e = -q or received correctl = If Erasure, reeat until correct Channel Matri q q q q

12 Conditional Entro ow can we measure the uncertaint about after observing? Conditional Entro Given = J = log This quantit is itself a random variable that taes on the values =o,.., =K- with robabilities o,., K- resectivel. Remember: Entro of a source N i i log / i

13 Conditional Entro Cont. The mean of the entro = over all the outut smbols is therefore given b: K = K J = log is called the conditional entro. It reresents the amount of uncertaint about the channel inut after the channel outut has been observed. It is called equivocation K J =, log

14 Mutual Information The mutual information : measures how much information about the channel inut can be obtained b observing the channel outut. Mutual Information Mutual Information reresents the gain in information due to an observation,, I I, I

15 I, = - Mutual Information J K = log J K J K K J =, log -, log =, log J K =, log K, From Baes rule

16 Proerties of Mutual Information. The Mutual Information of a Channel is Smmetric i.e., I,=I,. The Mutual Information is alwas Non-Negative i.e., I, 3. I,=+-, Another form: For Proof: See Tet Boo K J,, log,

17 Channel Caacit Definition: The channel caacit of a discrete memorless channel is defined as the maimum rate at which the information can be transmitted through the channel. It is maimum mutual information I, over all ossible inut robabilit distributions C ma I, { }

18 Remember Maimum of the Entro Entro of a source For the binar source: Binar Source N i i / log i o v logv as v log log log log is maimum at =/

19 Binar Erasure Channel Caacit of I, =, log J K, -q,, q,, - -q, - q Substituting Pr = =, Pr = =- C q : o o o J q E, ma imum log log, at q I

20 Caacit of Binar Smmetric Channel Pr = =, Pr = =- nd Then K for J a,,..., Then, ] ma [ since ma. is when ma. is then, deendon not does log log log, log log since ] [ ma ], [ ma C I C K J = = = = - - log log C

21 Aroimated Formula to obtain the caacit Theorem: Without Proof Given J inut smbols and K outut smbols, the channel caacit of a smmetric discrete memorless channel is given b: C J J K log log K This formula is not tight

22 Eamle: A binar erasure channel has the following robabilities: =.75,e=.5, e=.5, and =.5. a- Draw the diagram of the channel. b-comute the equivocation c- Comute the channel caacit using two different methods. - Solution b- J=, K=3 e log log, J K J.5,, J then Since

23 Eamle Cont.: log log log log

24 Eamle Cont.:.34 have : we in values these b substitution of.5.5,.5,,.75, since log log

25 Eamle Cont.: Channel caacit: Prob. of error=q J=, K= q C e e q 3 log ] log log log log log log [ 3 log log log log log log K J C J K

26 Aendi: Maimization of o ]log [ ]log [ Then and since log log,,..., J K for

27 Aendi: Maimization of Cont. Let Then z z log / z zlog z z log z zlog z where z We maimize w. r. t is and the maimum at z / i. e. at maimum is :ma[ ] /

28 The Maimum Data Rate of a Channel The qualit of the channel indicates two tes: A. A Noiseless or Perfect Channel An ideal channel with no noise. The Nquist Bit rate derived b enr Nquist gives the bit rate for a Noiseless Channel. B. A Nois Channel A realistic channel that has some noise. The Shannon Caacit formulated b Claude Shannon gives the bit rate for a Nois Channel

29 Nquist Bit Rate The Nquist bit rate formula defines the theoretical maimum bit rate for a noiseless channel Caacit=Bitrate = Bandwidth Log M Where: Bitrate is the bitrate of the channel in bits er second Bandwidth is the bandwidth of the channel M is the number of signal levels.

30 Nquist Bit Rate Eamle: What is the maimum bit rate of a noiseless channel with a bandwidth of 5Kz transmitting a signal with two signal levels. Solution: The bit rate for a noiseless channel according to Nquist Bit rate can be calculated as follows: BitRate = Bandwidth Log M = 5 log = bs Kbs

31 Shannon Caacit Shannon s Formula for maimum caacit in bs maimum data rate: C = B log + SNR bit/sec Caacit can be increased b: Increasing Bandwidth Increasing SNR caacit is linear in SNR Maimum bandwidth Allowed b the channel

32 Shannon Caacit