UCSD ECE154C Handout #21 Prof. Young-Han Kim Thursday, April 28, Midterm Solutions (Prepared by TA Shouvik Ganguly)

Transcription

1 UCSD ECE54C Handout #2 Prof. Young-Han Kim Thursday, April 28, 26 Midterm Solutions (Prepared by TA Shouvik Ganguly) There are 3 problems, each problem with multiple parts, each part worth points. Your answer should be as clear and readable as possible. Please justify any claim that you make.. Binary codes (3 points). Consider a source that emits five symbols {A,B,C,D,E} with probabilities.3,.3,.2,., and., respectively. (a) Construct a binary Huffman code for this source, taking one source symbol at a time. What is the average codeword length for this code? (b) Repeat part (a) for a binary Shannon Fano code taking one source symbol at a time. (c) Construct a probability distribution p = (p A,p B,p C,p D,p E ) on {A,B,C,D,E}, for which the code that you constructed in part (a) has an average length equal to its binary entropy H(p). Solution : (a) One possible way to do the Huffman coding is shown below. A.3.6 B.3 C.2. D. E..2.4

2 This gives the encoding A B C D E The average codeword length is given by L = 2 ( )+3 (.+.) = 2.2 bits per symbol. Alternatively, the average length can be computed by adding the probabilities at the internal nodes, so we have L = = 2.2 bits per symbol. 2

3 (b) The procedure for constructing a binary Shannon Fano code is shown below. A B C D E..6.4 A B C D E.3 A.3 B.2.2 C D E D. E. This gives the encoding A B C D E, which is the same as the Huffman code. Thus, the average length is the same. Similar to part (a), the average length can also be computed by adding the probabilities at the internal nodes, giving L = = 2.2 bits per symbol. 3

4 (c) We see that for the Huffman code we constructed, the codeword lengths satisfy i 2 l i =. Hence, if we have a probability distribution with p i = 2 l i, our code will have average codeword length given by L = i p i l i = i p i logp i (since p i = 2 l i ) = H(p), i.e., the entropy of the distribution. This shows that the required probabilities are p A = p B = p C = /4, p D = p E = /8. This gives an average codeword length of L = 2.25 bits per symbol, which is the same as H(p A,p B,p C,p D,p E ). 2. One-bit quantizer (3 points). Let X be drawn according to the pdf f X (x) 2 f X (x) = { 2x, x,, otherwise. (a) Given the quantization regions R = {x : x < b} and R 2 = {x : b x }, find the quantization points a R and a 2 R 2 that minimize the mean squared error (MSE) in terms of b. (b) Giventhequantizationpointsa < a 2, findthequantizationregionsthatminimize the MSE in terms of a and a 2. (c) Using parts (a) and (b), find the optimal quantizer by specifying a,a 2, and b that minimize the MSE. Solution : x 4

5 (a) Using the centroid condition, we have and a 2 = a = b 2x2 dx b 2xdx = (2/3)b3 b 2 = 2 3 b b 2x2 dx b 2xdx = (2/3)( b3 ) b 2 = 2(b2 +b+). 3(b+) (b) Using the Voronoi condition, the quantization regions are given by R = [,b) and R 2 = [b,], where b = a +a 2. 2 (c) Plugging the expressions for a and a 2 computed in part (a) into the expression in part (b), we have 2b = 2 3 b+ 2(b2 +b+) 3(b+) = 4 3 b = 2(b2 +b+) 3(b+) = 2b = b2 +b+ b+ = 2b 2 +2b = b 2 +b+ = b 2 +b = 5 = b =. 2 Plugging this value into the expressions in part (a), we have a = 5 3 and a 2 = 2( 5 ). 3 5

6 3. Source coding for emergency (2 points). Consider a source that emits six symbols {A,B,C,D,E,F} with probabilities.3,.2,.2,.,., and., respectively. Here the symbol F only has the probability., but it is meant for an emergency message, say, Fire!, and hence we would like to spend few bits in encoding F. (a) Design a binary instantaneous code that takes one source symbol at a time with the shortest possible average codeword length under the constraint that F should be mapped to the codeword of length. (b) (Difficult.) Now assume that each source symbol i has some associated cost per letter c i of the codeword. The normal symbols A,B,C,D,E has a cost of c i = per codeword letter, while the emergency symbol F has a cost of c i = 4 per code letter. Hence, if the codeword for symbol i has the length i, then the cost spent for the symbol is c i l i. For example, the code that maps A B C D E F spends the cost of 4,5,4,3,2, and 8, respectively, for the symbols A,B,C,D,E, and F. Design a binary instantaneous code that minimizes the average cost p i c i l i of sending a symbol. Solution : i (a) Since F is mapped to and the code has to be instantaneous, all other codewords must start with. In order to ensure that we have the shortest possible average codeword length, we can perform Huffman coding on the source symbols other than F, and finally add at the beginning of these other codewords. Note that the total probability of the symbols other than F is not ; however, the Huffman coding algorithm does not require the numbers associated with the symbols to sum up to ; any set of non-negative weights can be used, and the resulting encoding will then give the code with the shortest average codeword length, where the average is computed using the corresponding weights. 6

7 One way to perform the Huffman coding is as follows. A.3.5 B.2 C.2.9 D..4.2 E. This, together with the preceding discussion, gives the encoding A B C D E F The average codeword length is given by L = 3 ( )+4 (.+.)+. = 3. bits per symbol. Alternatively, the average codeword length can also be computed as L = +( sum of probabilities at internal nodes of the Huffman coding tree ) = = 3. bits per symbol. (b) Wearerequired tominimize i p ic i l i = i q il i, where q i = p i c i. Asaconsequence of the argument provided in part (a), in order to minimize the average cost, we can use Huffman coding with the probability of a symbol replaced by the 7

8 corresponding weight q i. The procedure is shown below. A.3.5 B.2.3 F.4 C.2.8 D..4.2 E. This gives the encoding A B C D E F The average cost is given by C = 2 ( ) (.+.) = 3.2 per symbol. Alternatively, the average cost can be computed as the sum of the weights at the internal nodes, and is thus given by C = = 3.2 per symbol. 8