Solutions to Assignment-2 MOOC-Information Theory

Transcription

1 Solutions to Assignment-2 MOOC-Information Theory 1. Which of the following is a prefix-free code? a) 01, 10, 101, 00, 11 b) 0, 11, 01 c) 01, 10, 11, 00 Solution:- The codewords of (a) are not prefix-free as the codeword 10 is clearly a prefix of 101. Similarly in (b), the codeword 0 is prefix of the codeword 01. The codewords of (c) are prefix-free as no codeword is a prefix of another codeword. Hence the correct answer is (c). 2. Which of the following is a uniquely decodable code but not prefix-free? a) 01, 10, 101, 00, 11 b) 0, 11, 01 c) 01, 10, 11, 00 Solution:- As explained in solution (1), codes (a) and (b) are not prefix-free. The unique decodability of the code can be tested as below (a) 01, 10, 101, 00, 11 S 0 S 1 S 2 S 3 S

2 Since, the codeword 01 is present in both S 0 and S 2 (S 3, S 4 and so on). Hence, it is not uniquely decodable. (b) 0, 11, 01 S 0 S 1 S 2 S Here, none of the codewords are present in other columns. Hence, it is a uniquely decodable code. The correct answer is (b). 3. Given a binary variable length code consisting of codewords of following length w 1 = 1, w 2 = 2, w 3 = 2, w 4 = 3. Which of the following statement is correct? a) There exist a prefix-free code of given lengths b) There exist many prefix-free codes of given lengths c) There exist a uniquely decodable code of given lengths Solution:- The criteria for existence of a prefix-free code is given by Kraft s inequality K D wi 1 i=1 Here, K = 4 and D = 2. Substituting the values of w i s, we get 4 i=1 D wi = which is greater than 1 and hence no prefix-free or uniquely decodable code exist. So, the correct answer is (d). 4. A discrete memoryless source emits a symbol U that takes five different values u 1, u 2, u 3, u 4, u 5 with probabilities 0.25, 0.25, 0.25, 0.125, respectively. A binary Shannon-Fano code consists of codewords of following lengths a) 3, 3, 3, 4, 4 b) 2, 2, 2, 3, 3 c) 1, 1, 1, 2, 2 2

3 Solution:- The length, w i of the codeword u i is given as w i = log D (P U (u i )) Here, D = 2 (binary codes). So, substituting the respective probabilities in the above equation we get the lengths of codewords as 2, 2, 2, 3, 3. Hence, correct answer is (b). 5. Given a discrete memoryless source that emits a symbol U, that takes five different values u 1, u 2, u 3, u 4, u 5 with probabilities 0.25, 0.25, 0.25, 0.125, respectively. An optimal ternary prefix-free code is designed to represent U. What is the expected length of the code? a) 1.4 b) 1.5 c) 2.25 Solution:- Here, D = 3 (Ternary Code) and K = 5. So, the number of the unused leaves = Remainder( (K D)(D 2) D 1 ) = 0. So, the tree diagram of the optimal ternary prefix-code is represented as below. Hence, the expected length of the code using path length lemma is given as E[W ] = = 1.5 bits (Probabilities of the nodes is directed in red). Thus, the correct answer is (b). 3

4 6. Given a discrete memoryless source that emits 6-ary random variable U, {u 1, u 2, u 3, u 4, u 5, u 6 } with probabilities {0.25, 0.25, 0.125, 0.125, 0.125, 0.125} respectively. Which of the following is not an optimal binary prefix-free code for this source? (Prefix-free code is represented by binary tree.) Figure 1: 6(a) Figure 2: 6(b) Figure 3: 6(c) Figure 4: 6(d) Solution:- It can be easily checked that all the above codes are prefix-free. Now, the optimal code is one in which only the least likely (i.e, less probable) leafs/nodes are connected. In (a), we observe that at each depth only least likely leafs/nodes are joined. So, (a) is an optimal code. Similarly, we can observe in (c) and (d) that the least likely leafs/nodes are only connected at each depth of the tree. Now, in (b) we observe that the code tree has connected a node with probability 0.5 with another node with probability 0.25 at depth level 1, while another node with probability 0.25 is left to be connected at the root. Hence, the code in (b) is not optimal. Thus, the correct answer is (b). 4

5 7. Given a discrete memoryless source that emits 6-ary random variable U, {u 1, u 2, u 3, u 4, u 5, u 6 } with probabilities {0.25, 0.25, 0.125, 0.125, 0.125, 0.125} respectively. Which of the following is an optimal ternary prefix-free code for this source? Figure 5: 7(a) Figure 6: 7(b) Figure 7: 7(c) 7( Solution:- It can be easily checked that all the above codes are prefix-free. Here, D = 3 (ternary codes) and K = 6. Now, the number of the unused leaves = Remainder( (K D)(D 2) D 1 ) = 1. Now, an optimal code should have any unused leaves only at the maximum depth of the D-ary tree. Since, the code in (b) is of depth 2 but has a missing leaf at depth 1. So, (b) can t be an optimal code. Similarly, code in (c) is of depth 3 but also has a missing leaf at depth 1 from the root. So, (c) can t be an optimal code. Now, the code in (a) has a missing leaf at the maximum depth. Hence, the code shown in (a) is optimal. Thus, the correct answer is (a). 5

6 8. Given a discrete memoryless source that emits a symbol U, that takes four different values u 1, u 2, u 3, u 4 with probabilities 0.2, 0.2, 0.3, 0.3 respectively. Which of the following is an optimal binary block to variable length code for this source but not Huffman code. Codewords of u 1, u 2, u 3, u 4 are respectively? a) 00, 01, 10, 11 b) 00, 11, 01, 10 c) 0, 10, 110, 111 d) 111, 110, 10, 0 Figure 8: 8(a) Figure 9: 8(b) Figure 10: 8(c) Figure 11: 8(d) Solution:- The expected length of the codes by path length lemma for (a)=2, (b)=2, (c)=2.4 and (d)=2.1. Thus, the codewords of (c) and (d) are not optimal. The code in (a) is a Huffman code as the two least likely codewords of (a) differs by 1 bit only (i.e 00, 01). Whereas the code in (b), although being optimal has it s two least likely codewords differ by 2 bits (i.e 00, 11). Hence, the code in (b) is an optimal, non-huffman code. Thus, the correct answer is (b). 6

7 9. Consider a bag containing 100 balls of red and blue colors. Let, X i = 1 or 0 depending upon whether the i-th ball is of red color or blue color. Let, X i be independent with Pr{X i = 1} = You are asked to find the set of all red balls. Any yes/no question is admissible. What is the minimum average number of questions required to identify the set of red balls? a) 80 b) 81 c) 82 Solution:- Since, it s a binary based setup. So, Pr{X i = 0} = The minimum average number of questions required to identify the set of red balls is given as N = H(X 1, X 2,..., X 100 ) Since, the X i s are independent so we have N = i=100 i=1 H(X i ) where H(X i ) = Pr{X i = 0} log 2 (Pr{X i = 0}) Pr{X i = 1} log 2 (Pr{X i = 1}) = 1 4 log log = N = Hence, the correct answer is (c). 10. Which of the following cannot be a valid Huffman code? a) 01, 10 b) 0, 10, 11 c) 01, 10, 11, 00 Solution:- The binary tree diagram is shown below. The unused leafs of each tree is shown by an empty circle, while the used leafs are shown with dotted circle. The codewords of (a) is shown to have two unused leaves and thus the codeword length can be shortened as shown in the tree diagram with the 7

8 Figure 12: 10(a) Figure 13: 10(b) Figure 14: 10(c) 10( red arrow. Hence, the codewords of (a) cannot be of a Huffman code since, it is not optimal. While, the codewords of (b) and (c) are shown to be using all the leaves and thus are valid Huffman codewords. Hence, correct answer is (a). 8