Statistical Communication Theory

Statistical Communication Theory Mark Reed 1 1 National ICT Australia, Australian National University 21st February 26

Topic Formal Description of course:this course provides a detailed study of fundamental statistical principles that underpin this subject. Emphasis will be on a modern treatment of detection and estimation theory. Elements of modulation theory and channel coding will be introduced at specific points in the course to link up with practical realities of digital communications. Informal Description of course: The outcome of the course for interested students is a fundamental understanding of detection and estimation theory. The course will examine the underlying statistical tools to perform detection and estimation, including probability theory, detection theory, maximum likelihood detection with memory, MAP detection with memory, turbo processing, parameter estimation theory, synchronisation, channel state estimation and multiuser detection.

Housekeeping 1 lectures, 2hrs each, over approx. 1 weeks (see schedule) Mon 3th Jan 2-4pm Mon 6th Jan 2-4pm Mon 13th Feb 2-4pm Thur 23rd Feb 2-4pm Mon 27th Feb 2-4 Mon 6 March 2-4pm Mon 13 March 2-4pm Tue 21st March 1-12 Mon 1th April 2-4pm Mon 17th April 2-4pm Assessment, 4 assignments handed out during the course

Topic The proposed course outline is: - Introduction (Week 1) Historical Evolution of Digital Communications: How and Why The Challenge in Designing High-performance Receivers Detection and Estimation using Statistics Probability Theory and Statistics (Week 2) The Inference Problem Bayesian Theory Hypothesis Testing Hypothesis-Testing (M-ary) Parameter Estimation Channel Models Markov Models and Chains

- Detection Theory (Week 3) Gram-Schmidt Orthoganalization Matched Filtering Sufficient Statistics Detection Criteria The Receiver in its most Basic Form Receiver-operating Characteristic (ROC) and its Properties Bounds on Receiver Performance ML Detection with Memory (Week 4) Channel Memory The ML Criterion extended to deal with memory State and Trellis Diagram Dynamic Programming The Viterbi Algorithm Computational Limitations The SOVA Algorithm Theme Example - Viterbi Equalization for a Wireless Channel

MAP Detection with Memory (Week 5) The MAP Criteria Extended to Deal with Memory Two Basic Notions of Estimation Markov Chain Revisited The (BCJR) APP Algorithm Variants of the APP Algorithm Forward-backward Interpretation of the APP Algorithm Need for pre-processing Theme Example - MAP Equalization for a Wireless Channel Turbo Processing (Week 6) The Turbo Principle Concatenated Transmitter configurations Likelihood-ratio Calculations Turbo Decoding Turbo Equalization Analysis Schemes Complexity Reduction Strategies Theme Example: Turbo Equalization for a Wireless Channel

Parameter Estimation Theory (Week 7) Parameter Estimation in Digital Communication Receivers MSE, ML and MAP Estimation Procedures, and Their Statistical Properties Amplitude Estimation in Noise Phase/Frequency Estimation Timing Estimation Performance Bounds Synchronization (Week 8) The Receiver Synchronization Problem Signal Acquisition Signal Tracking Turbo Synchronization Theme Example: Turbo Phase Synchronization for a Wireless Channel

Channel State Estimation (Week 9) Pilot Assisted (Supervised) Training Approach Semi-blind State Estimation State-space Model of the Channel Kalman Filtering Strategy Particle Filtering Strategy Particle Filtering as a Feed-forward Method for Signal Tracking Theme Example: Turbo Phase Synchronization for a Wireless Channel Revisited Multi-user Detection (Week 1) The Multi-user Detection Problem Maximum Likelihood Detector (Optimum Detector) The Conventional Detector (Matched Filter) Successive Interference Cancellation Strategy Parallel Interference Cancellation Turbo Multi-user Detection Reduced Complexity Turbo Multi-user Receivers

ML Detection with Memory (Week 4) Channel Memory The ML Criterion extended to deal with memory State and Trellis Diagram Dynamic Programming The Viterbi Algorithm Computational Limitations The SOVA Algorithm Theme Example - Viterbi Equalization for a Wireless Channel

Detection for Channels without Memory Channel Model y j = b j + n j A-Posteriori Prob. (APP) Maximum- A-Posteriori (MAP) Pr{b j = b (m) y j } = p(y j b j = b (m) ) Pr{b j = b (m) } p(y j ) arg max b (m) Pr{b j = b (m) y j } = arg max b (m) p(y j b j = b (m) ) Pr{b j = b (m) } Maximum Likelihood (ML) Detection arg max Pr{b j = b (m) y j } = arg max b (m) p(y j b j = b (m) ).5.45 APP/MAP A-prior information provides hint of real data value Normalised Likelihood.4.35.3.25.2 p(y mx=+1) =.243 y p(y j b j ) is a point from a Gaussian Distribution.15.1 mx= 1 mx=+1.5 p(y mx= 1)=.4 5 4 3 2 1 1 2 3 4 5 Received value Turbo Receiver Design:From Principles to Practice c 24 Mark C. Reed. 24

The Channel Model is Detection for Channels with Memory y = b + n The APP outputs are Pr{b j = b y 1,, y j+m }. The MAP decision criterion is then ˆbj = arg max b ±1 Pr{b j = b y 1,, y j+m } = arg max b ±1 p(y 1,, y j+m b j = b) Pr{b j = b} p(y 1,, y j+m ) = arg max b ±1 p(y 1,, y j+m b j = b) Pr{b j = b}. Turbo Receiver Design:From Principles to Practice c 24 Mark C. Reed. 25

Dynamic Programming Published by Andrew Viterbi in 1967 as a decoder for convolutional codes Explained in more detail by Forney in 1972 Method also know as Maximum Likelihood sequence Estimation Finds best sequence but may not minimise probability of errors Method collapses infinite tree into a trellis for decoding Tree methods include Fano, Sequential decoding (Stack) algorithms

The Viterbi Algorithm 1 Compute the transition metric for bit k of the actual received signal and state v of the equalizer, where v =, 1, 2,..., 2 l 1 and l is the memory of the channel. 2 Compute the accumulated transition metric for every possible path in the trellis representing the equalizer. The metric for a particular path is defined as the squared Euclidean distance between the estimated received waveform represented by that path and the actual received waveform. For each node in the trellis, the Viterbi equalizer compares the two paths entering that node. The path with lower metric is retained, and the other path is discarded. 3 Repeat the computation for every bit of the received signal. 4 The survivor, or maximum likelihood sequence, path determined by the algorithm defines the most likely l-bit sequence. Typically the survivor path is selected after a delay equal to five times the memory of the channel.

Computational Limitations States of trellis grow exponentially with Memory of Channel If we are transmitting high order constellations (16-QAM) then number of states increase with M n M = cardinality of symbols (16-QAM this equals 4) n = memory of Channel e.g. memory 8, 16 QAM, 65536 States!! Lower complexity solutions are often required (eg) Multi-user detection for CDMA)

The SOVA Algorithm Produces A-Posteriori Probability (reliability value) for each bit Overcomes hard decision output from VA for a turbo system Saves delta value = max ACS value - min. ACS value Computes Likelihood See Hagenauer and Hoeher A Viterbi Algorithm with soft-decision outputs and its applications in Proc. IEEE Globecom (Dallas, U.S.A.) pp168-1686, 1989

Theme Example - Viterbi Equalization for a Wireless Channel

CHAPTER 1 THEMES This Chapter just contains all the themes I have to develop in one central place, latter on they will be shifted to each appropriate chapter. 1.1 THEME EXAMPLE - VITERBI EQUALIZATION FOR A WIRELESS CHANNEL 1.1.1 Introduction The Viterbi Equalizer (VE) determines the maximum likelihood data sequence for an Inter- Symbol Interference channel. In a similar way to Viterbi decoding the VE searches over all possible transmitted data symbols to determine the maximum likelihood path. The fundamental differences in the channel model between a convolutional code and an ISI channel are:- 1. With an ISI Channel the code rate equals one (one output bit for every input bit) 2. The channel is a real code, where the different delayed outputs are added together as real numbers instead of modulo-2 arithmetric. 1.1.2 Transmitter and Channel Model Figure 1.1. shows a simple modulator and ISI channel, the modulator simply converts binary values to real values where a 1 and 1 +1. The input binary data is modulated, Statistical Communication Theory, First Edition. By Simon Haykin and Mark C. Reed ISBN -471-45435-4 c 26 John Wiley & Sons, Inc. 1

2 THEMES in this case we consider only the real channel, however this can be easily extended to a complex channel with complex valued coefficients, the input data can also be complex, representing QPSK or higher order modulation schemes. The ISI channel contains a delay line where each delay of the signal is multiplied by a different channel coefficient, labeled c 1 to c 4. In a mathematical form the ISI channel can be represented as y t = i=t+m i=t c i d i + n t (1.1) where d i represents the modulated data stream, c i represents the channel coefficients, and n t represents the additive white Gaussian noise term. The output of the ISI channel is /1 Input Data Stream Modulator -1/+1 T T T c 1 c 2 c 3 c 4 Sum Output Data Stream AWGN Figure 1.1. Modulator and ISI Channel. essentially the input data smeared with the ISI channel coefficients plus the additive white Gaussian noise (AWGN). This means that the received data is effected by the channel coefficients as well as the data values directly before and after the bit of interest. In this example we assume we know the coefficients of the ISI channel and the challenge is to detect the data values. If we can correctly detect the data values then we can achieve the performance of the channel without ISI (i.e. just effected by AWGN) otherwise if we just use conventional detection our performance will be severely degraded. Figure 1.2. shows the effect of ISI on a transmitted signal using the channel [.48,.815,.48]. The modulated data symbols with four times over sampling are shown as impulses with amplitude equal to one, this signal is then convolved with a low pass filter (in this case a root raised cosine filter with four times oversampling). The received signal without any ISI effects is then filtered again with the root raised cosine filter by again convolving this filter by the received waveform. This is shown as the dotted line and perfect sample restoration is possible (i.e. the filter output passes directly through the modulated data peaks). Now with the ISI channel inserted and convolving the received signal by the channel as well as the root raised cosine filter the received signal with ISI can be observed (solid line). It is clear here that the received signal has been significantly disturbed by the ISI. For example data sample three and four have different signs than the data transmitted. This diagram clearly shows the effect of ISI on a transmitted signal and the need for a method of removing it s effect. 1.1.3 Viterbi Equalization The Viterbi equalizer assumes knowledge of the channel coefficients and checks each possible data combination to determine the most likely sequence, essentially the steps in performing the Viterbi equalization function are:-

THEME EXAMPLE - VITERBI EQUALIZATION FOR A WIRELESS CHANNEL 3 2.5 2 Modulated Data with 4x oversampling Matched Filtered Output No ISI Pulse and Channel Matched Filtered Output with ISI 1.5 1.5 Amplitude.5 1 1.5 2 2.5 1 2 3 4 5 6 7 8 Sample Number (4x Oversampling) Figure 1.2. Signals with and without ISI. 1. Compute the transition metric for bit k of the actual received signal and state v of the equalizer, where v =, 1, 2,..., 2 l 1 and l is the memory of the channel. 2. Compute the accumulated transition metric for every possible path in the trellis representing the equalizer. The metric for a particular path is defined as the squared Euclidean distance between the estimated received waveform represented by that path and the actual received waveform. For each node in the trellis, the Viterbi equalizer compares the two paths entering that node. The path with lower metric is retained, and the other path is discarded. 3. Repeat the computation for every bit of the received signal. 4. The survivor, or maximum likelihood sequence, path determined by the algorithm defines the most likely l-bit sequence. Typically the survivor path is selected after a delay equal to five times the memory of the channel. The trellis diagram for a ISI channel with memory two is shown in Figure 1.3. The states are labeled in the first column. For time point three the current state is represented by the data bits at time one and two while the transition represents the data at time three. For the time epoch four the current state now represents the data at time two and three, while the transition represents the data at time four. This continues for as long as the block of data maybe. In systems like telephone line modems the data is continuous so there is no end to this data sequence. For mobile phone systems like GSM [?] the data is blocked so special handling of the bits at the start and end of the block is needed to minimise errors and perform traceback. Figure 1.4. shows a bit error rate (BER) performance curve of the Viterbi equaliser, on the x-axis is the E b /N, or the energy per information bit divided by the noise, where the noise variance is σ 2 n = N /2. The y-axis shows the bit error rate or the probability of error.

4 THEMES d d d d d 3 d 4 d 4 d 5 d 5 d6 d 6 d 7 d 7 d 8 1 2 d 3 2 3 d 4 d 5 d 6 d 7 d 8 1 1 1 11 1 1 1 Figure 1.3. Trellis Diagram for a memory two ISI channel. Ideally we want to have a low E b /N and have a low probability of error, thus the lower left hand corner of the graph is where we would ideally like to be. Our performance is however limited by the modulation and coding methods we use as well as the channel conditions. In this example we have no channel coding and our modulation is simply real BPSK. The ISI channel used is h = [.47,.815,.47], that is, a memory two channel with three channel coefficients. In the BER plot the AWGN performance is shown, this is to indicate the best possible performance that can be achieved if the equalizer can fully remove the interference. Also shown is the performance of a conventional detector, this detector simply makes the following decision, if the data value is positive then the data transmitted was +1, while if the data is negative then the data transmitted was 1. This conventional detector works fine in AWGN channels, however, with the introduction of the ISI the degradation in performance is substantial. So much so that performance at normal operating points of one error in 1 (BER = 1 2 or one error in 1 1 4 is just not possible, regardless of the signal to noise ratio. 1 1 1 AWGN Simulated Performance Conventional Receiver Performance Viterbi Equalizer Performance. AWGN Performance BER 1 2 1 3 1 4 1 2 3 4 5 6 7 8 E /N b Figure 1.4. Bit Error Rate performance of Viterbi Equalizer.

THEME EXAMPLE - VITERBI EQUALIZATION FOR A WIRELESS CHANNEL 5 For this channel the conventional receiver cannot overcome the ISI and the error rate is fixed to approximately.4 (four errors in every 1 bits) regardless of the signal to noise ratio. The Viterbi Equalizer can achieve low bit error rates for a given E b /N, for example for an operating point of one error in 1 bits (P e = 1 2 ) the Viterbi equalizer requires an E b /N = 7dB, while if there was no interference then the E b /N = 4dB. This demonstrates that although the Viterbi equalizer removes a large proportion of the interference, there is still some residual interference that remains. We will later demonstrate how this can be overcome. 1.1.4 A Small Example This example illustrates the advantage of performing equalization at the receiver. Ten data symbols are modulated and transmitted over an ISI channel, as used above. In Table 1.1. the trace values for the Viterbi Equalizer are shown. These indicate for the first ten data values received the most likely previous state from the current state and time interval. Therefore from the tenth symbol both state three and four say that the maximum likelihood sequence must go to state four, while state one and two believe the maximum likelihood sequence in the previous time interval must pass through state three. Table 1.1. Prefered Previous State values states \ time 1 2 3 4 5 6 7 8 9 1 s=1 1 1 1 3 3 1 3 3 3 3 s=2 1 1 1 3 3 1 1 3 3 3 s=3 2 2 2 2 2 2 2 2 2 2 s=4 2 2 2 2 2 2 2 2 2 2 In Table 1.2. the accumulated metric values are shown for the first ten time intervals, each state has a value which corresponds to the trace values provided in Table 1.1. Here for example at time symbol ten the lowest metric is state three, therefore this is where the traceback should start in the preferred previous state Table 1.1. Tracing back the values from Table 1.2. Accumulated Transition Metric states \ time 1 2 3 4 5 6 7 8 9 1 s=1.2.4.6 1.96.5.7 1.87.6 1.78.9 s=2.95.48.4 1.27.77.5 1.12.8 1.13.63 s=3 3.22 3.19 1.27.5 2. 1.45.7 1.13.8 1.68 s=4 6.81 6.29 3.38.69 4.5 3.47 1.2 1.95.76 3.55 state three at the tenth time interval the previous states are then [3, 2, 3, 2, 1, 3, 2, 1, 1, 1] in reverse order read from right to left. The final transition is from state two at time one to state one at time zero. this transition represents a transmitted bit of +1(1), therefore this is the resultant answer of the Viterbi Equalizer. Table 1.3. shows the resultant detected values for the first ten data symbols transmitted, using the traceback path. Figure 1.5. shows the trellis for the ISI channel and the traceback path for our example. The output values can also be seen here, which are computed based on the path on the traceback. Typically only the last value in the traceback is output, however, at the end of a

6 THEMES Table 1.3. Detected Values Time 1 2 3 4 5 6 7 8 9 1 Detected Values 1 1 1 block the final traceback path is used to compute all values to the end of the block, in much the same way we have performed the task here. The bits closest to the end of the block have the least reliability as the traceback length is limited and therefore errors in the values can result (unless termination of the data is used). Time 1 2 3 4 5 6 7 8 9 1 1 1 1 1 Output 1 1 1 Figure 1.5. ISI Trellis showing Traceback and Output.

Assignment 1 Implement an orthogonisation algorithm and compare to qr() in MATLAB use a random matrix as input say a 1 by 1 provide the code of your implementation

Assignment 2 Implement an AFC loop to track a frequency offset in baseband Symbol Rate 15Hz, Frequency offset 5Hz Assume BPSK and sequence is known at rx Assume perfect timing information Provide Performance Plots (freq vs symbols) Determine 2nd order Filter Parameters Provide source code

References H. V. Poor An Introduction to Signal Detection and Estimation D. MacKay, Information Theory, Inference, and Learning Algorithms U. Mengali, Synchronization Techniques for Digital Receivers, Kluwer Academic/Plenum Publishers, 1997 http://www.sss-mag.com/index.html C.. Helstrom, Elements of Signal Detection and Estimation Prentice-Hall G. Golub, C. Van Loan Matrix Computations R. Peterson R. Ziemer D. Borth Introduction to Spread Spectrum Communications Prentice-Hall