Communication Theory II

Transcription

1 Communication Theory II Lecture 14: Information Theory (cont d) Ahmed Elnakib, PhD Assistant Professor, Mansoura University, Egypt March 25 th,

2 Previous Lecture: Source Code Generation: Lossless Data Compression Techniques o Limits of Lossless data compression: The entropy of the source establishes the fundamental limit on the removal of redundancy from the data o Prefix code Shannon s 1 st theorem; source coding theorem: L H S Huffman code (a prefix code with optimal shortest expected length) o Lempel-Ziv code: standard algorithm for file compression 2

3 Lecture Outlines osource Code Transmission over a Noisy Channel: Channel Coding Discrete memoryless channel Example: Binary symmetric channel Mutual Information Channel Capacity Channel-Coding Theorem 3

4 Channel coding (encoder/decoder) ochannel coding consists of : Channel encoder: mapping the incoming data sequence into a channel input sequence Channel decoder: inverse mapping the channel output sequence into an output data sequence in such a way that the overall effect of channel noise on the system is minimized 4

5 A simple Diagram for Channel Coding ofor simplicity, we have not include the source encoder/decoder in figure 5

6 A Discrete Memoryless Channel oa statistical model with an input R.V. X and an output R.V. Y that is a noisy version of X oevery unit of time, the channel accepts an input symbol X selected from an alphabet X and, in response, it emits an output symbol Y from an alphabet Y odiscrete: both of the alphabets X and Y have finite sizes omemoryless: the current output symbol depends only on the current input symbol and not any previous or future symbol

7 Example: Binary Symmetric Channel (BSC) Transition Probability Diagram obinary: X and Y of size (cardinality) of 2 osymmetric: the probability of receiving 1 if 0 is sent is the same as the probability of receiving 0 if 1 is sent 7

8 Channel Matrix (Stochastic Matrix) ocompletely specifies a channel: The transmission medium Signal specifications (binary, r-ary, simplex, orthogonal, kind of receiver used: receiver determines the error probabilities) o Defined by a set of transition probabilities: Where is the conditional probability that y k is received when x j is transmitted obsc: Channel Matrix 8

9 Channel Matrix Properties o Each row corresponds to a fixed channel input o Each column corresponds to a fixed channel output o Satisfy the axioms of probability: and The sum of elements along any row is always equal to one 9

10 Average uncertainity about the channel input: H(x) olet P(x i ) be the probability of the event that the channel input X=x i Prior probability: before observing the channel output oh(x): average uncertainty about the channel input (i.e., before observing the channel output op(y j )= i P(y j x i )P(x i )= i P(y j,x i ): marginal probability distribution of the output R.V Y 10

11 Equivocation of x with respect to y : H(x y) olet P(x i y j ) be a conditional probability that x i is transmitted when y j is received There is an uncertainty of log P(x i y j ) about x i when y j is received 1 oh(x y): Equivocation (to use unclear language) of x with respect to y Average uncertainty about the channel input after observing the channel output Average loss of information about a transmitted symbol when a symbol is received Average uncertainty about a transmitted symbol x when a symbol y is received P(x i,y j )=P(x i y j )P(x i ) is the joint probability distribution of the R.Vs X and Y ofor error free channel: average information is H(x) bits Due to noise, we lose an average of H(x y) bits per symbol 11

12 Mutual Information of a channel I(x; y) bits per symbol o Mutual Information of a channel (symmetric) Amount of information that the receiver receives A measure of the uncertainity about the channel input, which is resolved by observing the channel output I(x; y)=h(x) H(x y) bits per symbol and j P x i, y j = P(x i ) A measure of the uncertainity about the channel output, which is resolved by observing the channel input I(x; y)=i(y; x)=h(y) H(y x) bits per symbol For error free channel, no loss of information I(x; y)=i(y; x)=h(x)=h(y) bits per symbol 12

13 ononnegative Some Properties of Mutual Information (why?) H(x y) H(x): We cannot lose information, on the average, by observing the output of a channel more than the average transmitted information H(x) owhen does I(x; y)=0? When we receive no information at the receiver; i.e, lose all input information on the channel: H(x y) = H(x) This is satisfied if the input and output symbols of the channel are statistically independent (why?) If the input is given, this gives no idea about the output received information is zero 13

14 Channel Capacity o Property of a particular physical channel (i.e., function of channel matrix values; function of signal power and channel noise) that measure its intrinsic ability to convey information o Maximum information that can be transmitted by one symbol over the channel o The maximum value of the mutual information of the channel w.r.t P(x i ) r s r c C s o Let a source that produces r s symbols/second the enocoder transmits r c code digit/sec Code rate of the channel encoder R code = r s r c symbol/code digit(bit) o Channel Capacity per second (critical rate)=r c C s bits/second 14

15 Example: Channel Capacity of BSC o Let then: o For a given P e, Ω P e is fixed, and 15

16 Recall the Entropy of Bernoulli Random Variable p 0 1 p 0 H(X) H(X) Note: lim x 0 x log x = 0 Entropy function H(Po) 16

17 Channel Capacity of BSC o o is maximum if \ o For error free communications: C s max=1 bit/symbol We can transmit 1 bit of information per binary digit (symbol) for error free communications One binary bit can convey one of the two equiprobable messages o C s max=1 bit/symbol at P e =0 (noiseless) or P e =1 (good as noiseless. why? Output is known) o C s min=0 bit/symbol at P e =0.5 (why?) The transmitted and received signals are statistically independent (P(x)=P(y) (why?) If 0 is received, 0 and 1 are equally likely to have been transmitted information=0 17

18 Magnitude of Channel Capacity o o I(x; y)=h(x) H(x y) bits per symbol o C S is always less than or equal to the average information per input symbol o Example 1: C S H x bit/binary symbol; H(x y) 0 If we use binary symbols as input: the maximum value of H(x)=1 bit (when P(x o )=P(x 1 )) C S 1 bit/binary symbol o Example 2: If we use r-ary symbols as input: the maximum value of H(x)=1 r-ary unit (in case of equiprobable) C S 1 r-ary unit/binary symbol 18

19 Source Code Transmission over a Noisy Channel o A source with an entropy H(m) bits Can we achieve error free communication? If so, what is the cost? oredundancy can help to compact noise Adding a single parity check code (ensures that the total number of 1 s is always even or odd) can check a single error cost: L new = L+1 which means a decreased efficiency oimmunity against noise can be increased by increasing redundancy Example: adding a number of repeated guards for each binary symbol oshannon shows that it is theoretically possible to achieve error-free communications by adding sufficient redundancy 19

20 Shannon s 2 nd Theorem Channel-Coding Theorem r s r c S H(s) C s o Channel coding theorem: for error free communication (arbitrary small error) Source information rate channel capacity per second r s H(s) bits/sec r c C s bits/sec Otherwise if H(s) > r c C s, then error occurs o Messages from the source with entropy H(S) must be encoded by binary codes with a word length at least H(S)/C s o If condition satisfies, then good codes do exist o For discrete memoryless channels that produces equally likely symbols H(s)= 1 bit/symbol For error free communications R code C s bit/symbol We do not need to lower R code too much (a small reduction of the rate of information transmission is enough) Channel capacity C s Channel Capacity per second=r c C s bits/second r c code symbol/ sec r s symbol/ sec H(s) source entropy bit/symbol Source information rate=r s H(s) bits/sec Code rate R code = r s r c 20

21 Questions 21