Digital Communication Systems ECS 452

Transcription

1 Digital Communication Systems ECS 452 Asst. Prof. Dr. Prapun Suksompong 2. Source Coding 1 Office Hours: BKD, 6th floor of Sirindhralai building Monday 10:00-10:40 Tuesday 12:00-12:40 Thursday 14:20-15:30

2 Elements of digital commu. sys. Message Transmitter Information Source Recovered Message Destination Source Encoder Remove redundancy Source Decoder Channel Encoder Add systematic redundancy Receiver Channel Decoder Digital Modulator Digital Demodulator Channel Transmitted Signal Received Signal Noise & Interference 2

3 System Under Consideration Message Transmitter Information Source Recovered Message Destination Source Encoder Remove redundancy Source Decoder Channel Encoder Add systematic redundancy Receiver Channel Decoder Digital Modulator Digital Demodulator Channel Transmitted Signal Received Signal Noise & Interference 3

4 Main Reference Elements of Information Theory 2006, 2nd Edition Chapters 2, 4 and 5 the jewel in Stanford's crown One of the greatest information theorists since Claude Shannon (and the one most like Shannon in approach, clarity, and taste). 4

5 5 English Alphabet (Non-Technical Use)

6 US UK The ASCII Coded Character Set (American Standard Code for Information Interchange) [The ARRL Handbook for Radio Communications 2013]

7 Example: ASCII Encoder Character x Codeword c(x) E L O V MATLAB: >> M = 'LOVE'; >> X = dec2bin(m,7); >> X = reshape(x',1,numel(x)) X = Remark: numel(a) = prod(size(a)) (the number of elements in matrix A) 7 Information Source LOVE Source Encoder

8 English Redundancy: Ex. 1 J-st tr- t- r--d th-s s-nt-nc-. 8

9 English Redundancy: Ex. 2 yxx cxn xndxrstxnd whxt x xm wrxtxng xvxn xf x rxplxcx xll thx vxwxls wxth xn 'x' (t gts lttl hrdr f y dn't vn kn whr th vwls r). 9

10 English Redundancy: Ex. 3 To be, or xxx xx xx, xxxx xx xxx xxxxxxxx 10

11 11 Entropy Rate of Thai Text

12 Introduction to Data Compression 12 [ ]

13 Introduction to Data Compression 13 [ ]

14 ASCII: Source Alphabet of Size = 128 (American Standard Code for Information Interchange) [The ARRL Handbook for Radio Communications 2013]

15 15 Ex. Source alphabet of size = 4

16 Ex. DMS (1) X abcde,,,, p X x 1, x abcde,,,, 5 0, otherwise Information Source a c a c e c d b c e d a e e d a b b b d b b a a b e b e d c c e d b c e c a a c a a e a c c a a d c d e e a a c a a a b b c a e b b e d b c d e b c a e e d d c d a b c a b c d d e d c e a b a a c a d 16 Approximately 20% are letter a s [GenRV_Discrete_datasample_Ex1.m]

17 Ex. DMS (1) clear all; close all; S_X = 'abcde'; p_x = [1/5 1/5 1/5 1/5 1/5]; n = 100; MessageSequence = datasample(s_x,n,'weights',p_x) MessageSequence = reshape(messagesequence,10,10) >> GenRV_Discrete_datasample_Ex1 MessageSequence = eebbedddeceacdbcbedeecacaecedcaedabecccabbcccebdbbbeccbadeaaaecceccdaccedadabceddaceadacdaededcdcade MessageSequence = eeeabbacde eacebeeead bcadcccdce bdcacccaed ebabcbedac dceeeacadd dbccbdcbac deecdedcca eddcbaaedd cecabacdae 17 [GenRV_Discrete_datasample_Ex1.m]

18 Ex. DMS (2) X 1, 2,3, 4 Information Source p X x , x 1, 2 1, x 2, 4 1, x 3,4 8 0, otherwise 18 Approximately 50% are number 1 s [GenRV_Discrete_datasample_Ex2.m]

19 Ex. DMS (2) clear all; close all; S_X = [ ]; p_x = [1/2 1/4 1/8 1/8]; n = 20; MessageSequence = randsrc(1,n,[s_x;p_x]); %MessageSequence = datasample(s_x,n,'weights',p_x); rf = hist(messagesequence,s_x)/n; % Ref. Freq. calc. stem(s_x,rf,'rx','linewidth',2) % Plot Rel. Freq. hold on stem(s_x,p_x,'bo','linewidth',2) % Plot pmf xlim([min(s_x)-1,max(s_x)+1]) legend('rel. freq. from sim.','pmf p_x(x)') xlabel('x') grid on Rel. freq. from sim. pmf p X (x) x 19 [GenRV_Discrete_datasample_Ex2.m]

20 DMS in MATLAB clear all; close all; S_X = [ ]; p_x = [1/2 1/4 1/8 1/8]; n = 1e6; SourceString = randsrc(1,n,[s_x;p_x]); Alternatively, we can also use SourceString = datasample(s_x,n,'weights',p_x); 20 rf = hist(sourcestring,s_x)/n; % Ref. Freq. calc. stem(s_x,rf,'rx','linewidth',2) % Plot Rel. Freq. hold on stem(s_x,p_x,'bo','linewidth',2) % Plot pmf xlim([min(s_x)-1,max(s_x)+1]) legend('rel. freq. from sim.','pmf p_x(x)') xlabel('x') grid on [GenRV_Discrete_datasample_Ex.m]

21 A more realistic example of pmf: Relative freq. of letters in the English language 21 [

22 A more realistic example of pmf: Relative freq. of letters in the English language ordered by frequency 22 [

23 Example: ASCII Encoder Codebook Character x Codeword c(x) E L O V MATLAB: >> M = 'LOVE'; >> X = dec2bin(m,7); >> X = reshape(x',1,numel(x)) X = Remark: numel(a) = prod(size(a)) (the number of elements in matrix A) 23 Information Source LOVE Source Encoder c( L ) c( O ) c( V ) c( E )

24 The ASCII Coded Character Set [The ARRL Handbook for Radio Communications 2013]

25 A Byte (8 bits) vs. 7 bits >> dec2bin('i Love ECS452',7) ans = >> dec2bin('i Love ECS452',8) ans =

26 Geeky ways to express your love >> dec2bin('i Love You',8) >> dec2bin('i love you',8) ans = ans = &ga_filters=holidays+-supplies+valentine&ga_search_type=all&ga_view_type=gallery 26

27 27 Summary: Source Encoder Information Source source string LOVE Discrete Memoryless Source (DMS) The source alphabet is the collection of all possible source symbols. Each symbol that the source generates is assumed to be randomly selected from the source alphabet. Source Encoder An encoder is a function that maps each of the symbol in the source alphabet into a corresponding (binary) codeword. The list for such mapping is called the codebook. Source Symbol x Codeword c(x) E L O V w/o extension encoded string c( L ) c( O ) c( V ) c( E ) The codeword corresponding to a source symbol is denoted by. the length of Each codeword is constructed from a code alphabet. For binary codeword, the code alphabet is 0,1

28 Morse code (wired and wireless) Telegraph network Samuel Morse, 1838 A sequence of on-off tones (or, lights, or clicks) 28

29 Example 29 [

30 30 Example

31 Morse code: Key Idea Frequently-used characters are mapped to short codewords. 31 Relative frequencies of letters in the English language

32 Morse code: Key Idea Frequently-used characters (e,t) are mapped to short codewords. 32 Relative frequencies of letters in the English language

33 Morse code: Key Idea Frequently-used characters (e,t) are mapped to short codewords. Basic form of compression. 33

34 34 รห สมอร สภาษาไทย

35 Example: ASCII Encoder Character Codeword E L O V MATLAB: >> M = 'LOVE'; >> X = dec2bin(m,7); >> X = reshape(x',1,numel(x)) X = Information Source LOVE Source Encoder

36 Another Example of non-ud code Suppose we want to convey the sequence of outcomes from rolling a dice. x c(x) A sequence of throws such as is encoded as

37 Another Example of non-ud code Suppose we want to convey the sequence of outcomes from rolling a dice. x c(x) The encoded string 11 could be interpreted as 11: 1 1 3: 11 The encoded string 110 could be interpreted as 12: :

38 Another Example of non-ud code x c(x) A 1 B 011 C D 1110 E

39 Another Example of non-ud code x c(x) A 1 B 011 C D 1110 E Consider the encoded string It can be interpreted as CDB: BABE: [ ]

40 Game: 20 Questions 20 Questions is a classic game that has been played since the 19th century. One person thinks of something (an object, a person, an animal, etc.) The others playing can ask 20 questions in an effort to guess what it is. 40

41 41 20 Questions: Example

42 Shannon Fano coding Prof. Robert Fano ( ) Shannon Award (1976 ) Proposed in Shannon s A Mathematical Theory of Communication in 1948 The method was attributed to Fano, who later published it as a technical report. Fano, R.M. (1949). The transmission of information. Technical Report No. 65. Cambridge (Mass.), USA: Research Laboratory of Electronics at MIT. Should not be confused with Shannon coding, the coding method used to prove Shannon's noiseless coding theorem, or with Shannon Fano Elias coding (also known as Elias coding), the precursor to arithmetic coding. 42

43 Huffman Code David Huffman ( ) Hamming Medal (1999) MIT, 1951 Information theory class taught by Professor Fano. Huffman and his classmates were given the choice of a term paper on the problem of finding the most efficient binary code. or a final exam. Huffman, unable to prove any codes were the most efficient, was about to give up and start studying for the final when he hit upon the idea of using a frequency-sorted binary tree and quickly proved this method the most efficient. Huffman avoided the major flaw of the suboptimal Shannon-Fano coding by building the tree from the bottom up instead of from the top down. 43

44 Huffman s paper (1952) 44 [D. A. Huffman, "A Method for the Construction of Minimum-Redundancy Codes," in Proceedings of the IRE, vol. 40, no. 9, pp , Sept ] [ ]

45 Summary: All codes Nonsingular codes 45 [ ] A good code must be uniquely decodable (UD). Difficult to check. [Defn 2.18] [2.24] Consider a special family of codes: prefix(-free) code. Always UD. Same as being instantaneous. [Defn 2.30] Huffman s recipe UD codes Prefix-free codes Huffman codes No codeword is a prefix of any other codeword. Each source symbol can be decoded as soon as we come to the end of the codeword corresponding to it Repeatedly combine the two least-likely (combined) symbols. Automatically give prefix-free code. [Defn 2.36] [2.37] For a given source s pmf, Huffman codes are optimal among all UD codes for that source.

46 Huffman coding 46 [ ]

47 Ex. Huffman Coding in MATLAB [Ex. 2.31] Observe that MATLAB automatically give the expected length of the codewords px = [ ]; % pmf of X SX = [1:length(pX)]; % Source Alphabet [dict,el] = huffmandict(sx,px); % Create codebook %% Pretty print the codebook. codebook = dict; for i = 1:length(codebook) codebook{i,2} = num2str(codebook{i,2}); end codebook %% Try to encode some random source string n = 5; % Number of source symbols to be generated sourcestring = randsrc(1,10,[sx; px]) % Create data using px encodedstring = huffmanenco(sourcestring,dict) % Encode the data 47 [Huffman_Demo_Ex1]

48 Ex. Huffman Coding in MATLAB codebook = [1] '0' [2] '1 0' [3] '1 1 1' [4] '1 1 0' sourcestring = encodedstring = [Huffman_Demo_Ex1]

49 Ex. Huffman Coding in MATLAB [Ex. 2.32] px = [ ]; % pmf of X SX = [1:length(pX)]; [dict,el] = huffmandict(sx,px); % Source Alphabet % Create codebook %% Pretty print the codebook. codebook = dict; for i = 1:length(codebook) codebook{i,2} = num2str(codebook{i,2}); end codebook EL The codewords can be different from our answers found earlier. The expected length is the same. 49 [Huffman_Demo_Ex2] >> Huffman_Demo_Ex2 codebook = EL = [1] '1' [2] '0 1' [3] ' ' [4] '0 0 1' [5] ' ' [6] ' '

50 Ex. Huffman Coding in MATLAB px = [1/8, 5/24, 7/24, 3/8]; % pmf of X SX = [1:length(pX)]; [dict,el] = huffmandict(sx,px); % Source Alphabet % Create codebook %% Pretty print the codebook. codebook = dict; for i = 1:length(codebook) codebook{i,2} = num2str(codebook{i,2}); end codebook EL [Exercise] 50 >> -px*(log2(px)).' ans = codebook = [1] '0 0 1' [2] '0 0 0' [3] '0 1' [4] '1' EL =

51 x

52 Entropy and Description of RV 57 [ ]

53 Entropy and Description of RV 58 [ ]

54 59 Summary: Optimality of Huffman Codes Consider a given DMS with known pmf [Defn 2.36] A code is optimal if it is UD and its corresponding expected length is the shortest among all possible UD codes for that source. [2.37] Huffman codes are optimal. All codes [ ] Bounds on expected lengths: Nonsingular codes UD codes Prefix-free codes Huffman codes Expected length Expected length (per source (per source symbol) of an symbol) of a 1 optimal code Huffman code

55 Summary: Entropy 60 Entropy measures the amount of uncertainty (randomness) in a RV. Three formulas for calculating entropy: [Defn 2.41] Given a pmf of a RV, binary entropy function log. [2.44] Given a probability vector, log. [Defn 2.47] Given a number, Set 0log 0 0. log 1 log 1 [2.56] Operational meaning: Entropy of a random variable is the average length of its shortest description.

56 Examples Example 2.31 H X Huffman 1.75 X Efficiency = 100% Example 2.32 H X Huffman X Efficiency 97% 61

57 Examples Example 2.33 H X Huffman X Efficiency 99% Example 2.34 A B C D H X Huffman X Efficiency 93% 62

58 Summary: Entropy Important Bounds deterministic uniform The entropy of a uniform (discrete) random variable: The entropy of a Bernoulli random variable: binary entropy function 63

59 [Ex.2.40] Huffman Coding: Source Extension 1 1 X k i.i.d. Bernoulli p p L n n: order of extension

60 [Ex.2.40] Huffman Coding: Source Extension 1.8 X k i.i.d. Bernoulli p p L n H X 1 n 67 H X Order of source extension n

61 Summary: Source Extension The encoder operates on the blocks rather than on individual symbols. [Defn 2.39] -th extension coding: 1 block = successive source symbols = expected (average) codeword length per source symbol when Huffman coding is used with -th extension H X 1 n 68 H X 0.6 L n Order of source extension