Coding for the Slepian-Wolf Problem With Turbo Codes

Transcription

1 Coding for the Slepian-Wolf Problem With Turbo Codes Jan Bajcsy and Patrick Mitran Department of Electrical and Computer Engineering, McGill University Montréal, Québec, HA A7, {jbajcsy, Abstract This paper proposes a practical coding scheme for the Slepian-Wolf problem of separate encoding of correlated sources. Finite-state machine () encoders, concatenated in parallel, are used at the transmit side and an iterative turbo decoder is applied at the receiver. Simulation results of system performance are presented for binary sources with different amounts of correlation. Obtained results show that the proposed technique outperforms by far both an equivalent uncoded system and a system coded with traditional (non-concatenated) coding. I. INTRODUCTION Slepian and Wolf investigated the problem of separate encoding of correlated sources, shown in fig., and derived the achievable region of fig. for this problem []. It was later shown by Csiszar that linear block codes can be used for this problem and that their performance is bounded by an error exponent []. Orlitsky showed in [] that for most cases of practical interest an error probability of can be achieved only asymptotically. In particular, if an error probability of exactly is desired, the achievable region for the Slepian-Wolf compression is often reduced to encoders that operate at the entropy rate of their sources. Oohama and Han then improved on the bound provided by [] and considered the case of universal coding where the encoders and decoder do not depend on the source statistics [4]. Practical code designs for the Slepian-Wolf problem have mostly concentrated on a zero-error objective [4-8]. Pradhan and Ramchandran focused on a special case when the correlation between X and is specified as a prescribed maximal Hamming distance and explored the use of linear block codes and trellis codes in this setting [4,5]. an and Berger studied necessary and sufficient conditions for the existence of a lossless instantaneous code for binary sources and gave sufficient conditions for nonbinary ones [7]. Zhao and Effros proposed a tree-structured algorithm to construct a class of lossless multiple access source codes in [8] and studied use of these codes in the near-lossless case [9]. Independently of the work presented in this paper, Garcia- Frias considered a related problem of joint source-channel decoding for data that were separately turbo coded and transmitted over noisy channels []. In that case, low rates of the codes were justified by the need for error correction. The most powerful channel coding techniques known today are based on concatenation and iterative decoding. Turbo codes and their extensions have reached performance close to the Shannon This work has been supported in part by the Natural Sciences and Engineering Research Council (Canada), the Canadian Institute for Telecommunications Research under the NCE program of the Government of Canada, and the McGill University Faculty of Graduate Studies and Research. limit for the AWGN channel [5,6] and have been successfully applied to systems in wireless communication and digital recording [8], [9], etc. In this paper, we propose using ideas from turbo coding to achieve data compression in the Slepian-Wolf problem. Section II introduces suitable notation and explains how parallel concatenated codes with iterative decoding can be applied to the Slepian-Wolf problem. In section III we describe the experimental setup, present simulation results and discuss their significance. Finally, in section IV we discuss possible directions for improvement and conclude the significance of this work. The design of finite stage machine () encoders used throughout this paper is discussed in the appendix. (X,) Enc X Enc Rate Rate R Decoder Fig.. Schematic block diagram of the Slepian-Wolf problem of separate encoding of correlated sources. H() H( X) R H(X ) +R =H(X,) H(X) Fig.. The achievable region for separate encoding of correlated sources. II. TURBO CODES APPLIED TO THE SLEPIAN-WOLF PROBLEM A. Preliminaries We consider two information sources, (X,), each generating a sequence of outputs X,X,X, and,,,. If each source is taken separately and is encoded/decoded without knowledge of the other, clearly the minimum rate needed to encode these two sources is H(X) + H(). In the case that X and are not independent, joint source coding/decoding can reduce the rate to H(X,) < H(X) + H(). This is of particular interest when the encoding must be done separately for X and (fig. ). We focus on achieving the corner points of the rate region in fig.., since any rate point in between //$7. IEEE 4

2 can be achieved using time-sharing techniques []. This motivates the idea of encoding sequence X with turbo codes and transmitting the sequence at a rate arbitrarily close to its entropy H() since techniques to perform the latter are well known [7]. We denote an encoder as a mapping from a message set to a codebook, i.e., f : M C () where C is a subset of all sequences S that can be transmitted. Similarly, a decoder is a mapping from the set of received data R into a decision set, ϕ : R D () A decoder may evaluate logarithmic ratios of aposteriori bit probabilities Pm ( = r) Pm ( = r) Pm ( = r) k Λ ( m r) = log, log,..., log. Pm ( = r) Pm ( = r) Pm ( = r) k In case of an encoder with trellis structure and a memoryless channel, implementation of such a decoder can be based on the BCJR algorithm []. B. Proposed Encoding Scheme Because of the time-sharing argument mentioned in the previous subsection, we focused on the following scenario. One of the sources is transmitted error-free at a rate arbitrarily close to its entropy, while the other source is coded with two finite-state machine () encoders concatenated in parallel and separated by an interleaver, as depicted in fig.. Both encoders are identical and output n binary symbols for every k binary input symbols, k > n. Note that in the region of interest, the total number of bits output per input symbol by both encoders combined is less than H(X), i.e., less than the number of input bits in case of a binary source. Throughout the rest of this paper, we shall follow the channel coding convention by saying that the individual encoders operate at a rate of k/n and that the total rate of the parallel-concatenated encoder for X is then k/n. (Note that for comparison with the achievable region in fig., one must convert the rate of the encoders into information bits per source symbol, i.e., the rate of the individual encoders is n/k and the rate of the concatenated scheme is n/k bits per source symbol.) (X,) X INTER- LEAVER ZERO ERROR Fig.. Proposed encoding structure for the Slepian-Wolf problem using parallel concatenated codes. r s t R / / / / / / / / / / / / / / / / Fig. 4. Example of a trellis section for a simple rate encoder (k = and n = ). encoders, used by our system, were custom designed, as it is discussed in Appendix A. An example of a trellis diagram for such an encoder is shown in fig. 4. Following design guidelines in the appendix, the input state transition matrix was chosen to be a Latin square of size k by k. Because it is difficult to design the output transition matrix such that each entry appears only k-n times per row and column, a compromise was made. Each entry appears only k-n times per row and each row is permuted randomly and independently. C. Iterative Decoder Setup At the receiver side, the turbo decoding principle is used as depicted in fig. 5 to recover X from the compressed vectors r, s and side-information y. Assuming for simplicity a binary source alphabet {;}, the proposed iterative decoder can be formally written as w = ϕ (, s y, z ) i i zi () z (,, ) i = ϕ r y w w (4) i i with the initial condition to be chosen as z = (,,...,). Decoding functions ϕ and ϕ evaluate the log-ratios of aposteriori probabilities ϕ (, s yz, ) (,, ) i x syzi ϕ =Λ (5) (, r y, w ) ( x r, y, w ) i i =Λ, (6) and can be implemented using the BCJR algorithm. Following the turbo decoding principle outlined in [5], extrinsic information vectors zi and w i are treated as if they contained independent realizations of a fictitious Gaussian vector with unit covariance matrix and mean x ( /,/,...,/). After each iteration, decisions are formed by thresholding entries of Λ ( x r, y, w ) i. 4

3 r s DECODER w i DECODER z i X^ (X,) X r y BCJR DECODER X^ t DECODER z i- DELA Fig. 5. Iterative turbo decoder applied to the proposed Slepian-Wolf coding scheme. III. PERFORMANCE RESULTS A. Experimental Setup To test the proposed coding scheme for the Slepian-Wolf problem, we applied it, operating at a fixed rate, to different pairs of correlated binary memoryless sources (X,). The amount of correlation between the sources depended on their joint probability mass function (PMF), given by the following matrix p p P = p p, (7) where p is a constant, < p < ¼. Note, that the entropy of each separate source is bit and that H(X ) = H( X) = h(p) H(X,) = +h(p), where h(.) is the binary entropy function. Fig. 6. Coding scheme with one for the Slepian-Wolf problem. B. Simulation Results Fig. 7 shows the performance plot for turbo coding where each is operating at a rate k/n = 4 and the overall encoder outputs half as many bits as are input. From information theory results, the ideal curve would asymptotically reach a of below H(X ) = n/k =.5. Fig. 8 shows the performance plot for a single code operating at an equivalent rate. It is worthwhile to note that the single scheme performance improved when the number of states was increased. Despite increasing the number of states for the single code, the decline in is still more pronounced for turbo coding with fewer states, as it can be observed by comparing fig. 7 and fig 8. Contrasting these figures, the superior performance of parallel-concatenated codes versus traditional non-concatenated codes is apparent. - - For the proposed turbo coded system, the block-length of the encoded message X was chosen to be the nearest integer to that is divisible by k. To close the trellis, a fixed state-transition sequence leading to state was used for each state. The interleaver was chosen as a random permutation and the decoding functions were implemented using the BCJR algorithm. The performance of a system with fixed rate is evaluated by plotting bit-error rate () versus conditional entropy H(X ) of various source pairs. To establish the effectiveness of our coding scheme in a fair manner, we also compared it to two more traditional coding approaches, an equivalent uncoded system and a single- coded system. The uncoded system achieves a data rate of k/n by throwing away (k-n)/k of the bits at the transmitter side. The optimal method to recover these bits is by maximizing conditional PMF p X (x i y i ) using the side information sequence from source. For a source whose PMF is given by (7), this procedure will produce an error with probability p (provided p <.5). Hence, the overall fraction of incorrect bits is given by ref = n p k. (8) The other reference system, based on a single encoder and corresponding BCJR decoder, is shown in fig. 6. In this case, a single rate k/n encoder encoded packets of 4 bits from source X into r. The decoder used both the coded sequence and the side information sequence to recover the message by thresholding Λ ( x r, y), evaluated with the BCJR algorithm iteration iterations iterations 4 iterations Fig. 7. Performance plot for turbo code with s operating at rate 4 (k=4, n=) and overall rate Uncoded Reference k =, n = k = 4, n = k = 6, n = Fig. 8. Performance plot for single encoder with k/n = /, 4/, 6/. 4

4 Fig. 9 and fig. show performance plots for proposed turbo coding schemes operating at rates 5/4 and /. For the turbo code with rate /, source X could almost always be recovered at a conditional entropy H(X ) =.49, provided that enough iterations were performed in the decoding process. Fig. shows the as a function of the number of iterations for a rate / turbo code at H(X ) =.49. Beyond 7 iterations, the message was exactly recovered in all simulations. This indicates that the rate / code is only.8 bits short of the ideal performance predicted by information theory Iteration iteration iterations Fig. 9. Performance plot for turbo code with rate 5/4 and each component at rate 5/ (k = 5, n = ). Fig.. vs. number of iterations for turbo code with overall rate / at H(X ) =.49. IV. CONCLUSION The use of turbo codes as applied to the Slepian-Wolf problem was investigated. The proposed scheme was found to perform dramatically better than either alternative, an equivalent uncoded scheme and a single encoder with BCJR decoding. In light of this observation, one is led to conclude that turbo coding is a viable technique for encoding/decoding in Slepian-Wolf networks. In fact, there is no penalty in using such a technique for Slepian-Wolf networks, since one can only asymptotically approach zero-error probability [] in the achievable region of interest Although general design rules for designing encoders were established, it is worthwhile noting that the constructed encoders can still be improved upon. The output state transition matrix and the interleaver were not optimum, so there is room left for future development. Other extensions of the presented work include simulation experiments for the cases of non-binary and Markov sources, as well as universal encoding/decoding in proposed scheme. iteration iterations -4 iterations 4 iterations Fig.. Performance plot for turbo code with rate / and each component with rate (k =, n = ). ACKNOWLEDGEMENTS One of the authors would like to thank S. Verdu for initial suggestions and discussions on coding for the Slepian-Wolf problem. APPENDIX A. Designing Encoders Using Latin Squares The encoders used in our concatenated scheme perform data compression and produce only n output bits for every k input bits where n<k. Since there are more possible encoder input sequences than output sequences, several encoder input sequences must map into a single output sequence. Instead of puncturing heavily the output of a (recursive) convolutional encoder to achieve this, e.g., to achieve k= and n=, we chose to custom-design time-invariant encoders. A couple of design rules, described in the following paragraph, were chosen to guarantee good performance of the constructed encoders. First, the number of states was chosen to be k to avoid dealing with parallel edges. Hence, every k-bit input sequence drives the 4

5 encoder from a given state into a different state, as shown, for instance, in fig. 4. The next design rule was chosen to guarantee that the encoder introduces sufficient amount of memory into coded data. This involved avoiding trellises whose one state collapses in a manner shown in fig., i.e., a trellis that always enters state for input sequence. This effect removes most memory of the previous states and reduces the efficiency of the encoding. (Even if a decoder knew the input bits were, there is still ambiguity as to which state the encoder was in originally.) An extreme case of a collapsed encoder would have the next state determined uniquely by the input symbols at given time instant or over a finite time window. The last is the case for traditional convolutional (non-recursive) codes and the limited amount of memory in known to limit their performance in concatenated encoding schemes [6]. We describe an encoder using two matrices: an input state transition matrix whose (i,j) th entry corresponds to the k-bit input sequence when the encoder makes a transition from state i to state j, and an output state transition matrix whose (i,j) th entry corresponds to the n-bit output sequence for the described transition. Our particular requirements translate into the fact that these matrices have dimensions k by k with every entry unique in its column for the input state transition matrix. Since the input state transition matrix must also have every entry unique in its row (otherwise the same input sequence would lead to two different states), this matrix must be a Latin square. These have been extensively studied in combinatorial mathematics [] and one can use them directly when constructing an encoder satisfying chosen rules. / / / / Fig.. An example of a trellis section unsuitable for source coding. The output state transition matrix, on the other hand, has only n different elements, n < k. It is therefore impossible to require that each entry be unique in its row and column. In fact, in any given row or column, one symbol must occur at least k-n times and we chose that each output symbol occurs exactly k-n times per row and column. An optimised scheme for designing both encoders jointly was considered and presented in []. REFERENCES [] D. Slepian and J. K. Wolf, Noiseless Coding of Correlated Information s, IEEE Trans. Information Theory, vol. 9, pp , July 97. [] I. Csiszar Linear Codes for s and Networks: Error Exponents, Universal Coding, IEEE Trans. Information Theory, vol. 8, pp , July 98. [] A. Orlitsky, Average-Case Interactive Communication, IEEE Trans. Information Theory, vol. 8, pp , September 99. [4]. Oohama and T. S. Han, Universal Coding for the Slepian- Wolf Data Compression System and the Strong Converse, IEEE Trans. Information Theory, vol. 4, pp , November 994. [5] C. Berrou, A. Glavieux, P. Thitimajshima, Near Shannon Limit Error Correcting Coding and Decoding: Turbo Codes, Proc. International Conference on Communications, Geneva, Switzerland, pp. 64-7, May -6, 99. [6] S. Benedetto, D. Divsalar, G. Montorsi, and F. Pollara, Serial Concatenation of Interleaved Codes: Performance Analysis, Design, and Iterative Decoding, IEEE Trans. Information Theory, pp , May 998. [7] T. M. Cover and J. A. Thomas, Elements of Information Theory, John Wiley and Sons, 99. [8] T. Souvignier, M. Oberg, P. H. Siegel, R. E. Swanson, and J. K. Wolf, Turbo Decoding for Partial Response Channels, IEEE Trans. Commun., vol. 48, pp. 97-8, Aug.. [9] J. Bajcsy, C.V. Chong, D.A. Garr, J. Hunziker and H. Kobayashi, On Iterative Decoding is Some Existing Systems, IEEE Journ. on Selected Areas in Comm., pp.88-89, May. [] S. Shamai (Shitz), S. Verdu, Capacity of Channels with Uncoded Side Information, European Trans. on Telecomm., vol. 6, pp , Sept.-Oct [] L.R. Bahl, J. Cocke, F. Jelinek and J. Raviv, Optimal Decoding of Linear Codes for Minimizing Symbol Error Rate, IEEE Trans. Inform. Theory, Vol., pp , March 974. [] J. H. van Lint and R. M. Wilson, A Course in Combinatorics, Cambridge University Press, 99. [] P. Mitran and J. Bajcsy Design of Fractional Rate Encoders Using Latin Squares, ISIT Recent Results Session, Washington, DC, June. [4] S. S. Pradhan and K. Ramchandran, Distributed Coding using Syndromes (DISCUS) Design and Construction, Proc. IEEE Data Compression Conference, pp , Snowbird, UT, March 999. [5] S. S. Pradhan and K. Ramchandran, Distributed Coding: Symmetric Rates and Applications to Sensor Networks, Proc. IEEE Data Compression Conference, pp. 6-7, Snowbird, UT, March. [6] A. Kh, Al Jabri and S. Al-Issa, Zero-Error Codes for Correlated Information s, Proc. of Cryptography, pp. 7-, Cirencester, UK, Dec [7]. an and T. Berger, On Instantaneous Codes for Zero-Error Coding of Two Correlated s, Proc. IEEE International Symposium Information Theory, p. 44, Sorrento, Italy, June. [8] Q. Zhao and M. Effros, Lossless Coding for Multiple Access Networks, Proc. IEEE International Symposium Information Theory, p. 44, Washington, DC, June. [9] Q. Zhao and M. Effros, Optimal Code Design for Lossless and Near-Lossless Coding in Multiple Access Networks, Proc. IEEE Data Compression Conference, pp. 6-7, Snowbird, UT, March. [] J. Garcia-Frias, -Channel Decoding of Correlated s over Noisy Channels, Proc. IEEE Data Compression Conference, pp. 8-9, Snowbird, Utah, March. 44