A New Method for Calculating Symbol Error Probabilities of Two-Dimensional Signalings in Rayleigh Fading with Channel Estimation Errors 1

Transcription

1 A New Method for Calculating Symbol Error Probabilities of Two-Dimensional Signalings in Rayleigh Fading with Channel Estimation Errors 1 Xiaodai Dong, Member, IEEE, Norman C. Beaulieu, Fellow, IEEE Department of Electrical and Computer Engineering, University of Alberta, Edmonton, Alberta, Canada T6G 2V4 xdong, Tel: Fax: This work was presented in part at the 2002 IEEE Symposium on Advances in Wireless Communications, Victoria, Canada, September 2002, and the 2003 IEEE International Conference on Communications ICC 2003), Anchorage, USA, May 2003.

2 1 Abstract A general analytical framework for evaluating the performance of coherent two-dimensional signaling in frequency flat Rayleigh fading with channel estimation is proposed in this paper. A new and simple analytical expression for the symbol error rate of an arbitrary polygonal two-dimensional constellation in Rayleigh fading in the presence of channel estimation errors is presented. This framework is applicable to many current channel estimation methods such as pilot symbol assisted modulation and minimum mean square error estimation where the fading estimate is a complex Gaussian random variable correlated with the channel fading. The sensitivity of various high-level two-dimensional signaling formats to static and dynamic channel amplitude and phase estimation errors in Rayleigh fading can be easily studied using the derived formula. The new exact expression makes it possible to optimize constellation parameters and various parameters associated with channel estimation schemes. It also provides insights into choosing an appropriate signaling format for a fading environment with practical channel estimation methods used at the receiver. Index Terms Channel estimation errors, fading channels, high order modulation, imperfect channel estimation, minimum mean square error MMSE), M-ary phase shift keying MPSK), pilot symbol assisted modulation PSAM), quadrature amplitude modulation QAM), symbol error probability, two-dimensional 2-D) modulation formats. I. INTRODUCTION The perfect coherent detection of arbitrary polygonal two-dimensional 2-D) signaling in frequency flat fading channels has been well studied. Tractable analytical results for determining the performances of 2-D signaling schemes are available for various fading models [1]-[2]. These results assume perfect fading channel state information. In reality, channel state information is only available through estimation algorithms with inherent channel estimation errors. Therefore, it is of practical interest and importance to study the performances of coherent signaling formats in the presence of channel estimation errors. Common channel estimation methods studied in the literature are pilot symbol assisted modulation PSAM) and minimum mean square error MMSE) estimation. A classical paper on pilot symbol assisted modulation for Rayleigh fading was presented by Cavers in 1991 [3] where he derived the optimum Wiener

3 2 interpolation filter to minimize the variance of the estimation error and studied the system performance for binary phase shift keying BPSK), quaternary phase shift keying QPSK) and 16-ary quadrature amplitude modulation QAM). The optimum Wiener filter requires a priori information of the channel and is computationally complex. Therefore, several suboptimal interpolation filters have been proposed among which the sinc interpolator is most widely used due to its simple implementation and close-to-optimum performance [4]. The bit error rate BER) of pilot symbol assisted M-ary QAM MQAM) in Rayleigh fading using the sinc interpolator has been analyzed in [5]. An upper bound to the symbol error rate SER) for pilot symbol assisted QAM in Rayleigh fading was presented in [6], and the optimum MMSE interpolator using an infinite number of pilot symbols was examined. The performance of MMSE channel estimation has been investigated for M-ary PSK MPSK) and MQAM in Rayleigh fading in [7] and for Ricean fading in [8]. The analyses in these two papers employed the orthogonality between the channel estimate and the channel estimation error. Their work has been further extended by Wilson and Cioffi [9] to eliminate the orthogonality requirement. The probability density functions PDF s) of the receiver decision variables for Rayleigh fading channels with maximal ratio diversity combining, and for single branch Ricean fading channels in the presence of imperfect channel estimation were derived. The PDF s were employed to calculate the SER and BER of 16-QAM with channel estimation errors. However, the analysis in [9] is only applicable to MQAM. In summary, results are only available for MPSK and MQAM with small M values; results for arbitrary 2-D signaling are not available. Furthermore, there has been no analysis that is able to unify these previously published results and to account for more general case of arbitrary 2-D signalings. We present in this paper a general method of analysis for determining symbol error probabilities of arbitrary 2-D signalings in Rayleigh fading with static or dynamic channel estimation errors. It is applicable to any fading estimate which is jointly Gaussian with the actual fading. This includes, but is not limited to, MMSE and pilot symbol assisted modulation schemes. Our analytical method is new and applies very generally to modulation formats not previously studied as well as to modulation formats previously studied using other methods which are not applicable to the general modulations considered here. The paper is organized as follows. Section II derives the error probability expression for coherent 2-D signal-

4 3 ing in Rayleigh fading with channel estimation errors. Section III applies the newly derived formula to the analysis of dynamic estimation errors when PSAM and MMSE estimation are employed as well as to the analysis of static estimation errors. Section IV discusses the numerical SER results for various 8-ary and 16-ary constellations that are of practical interest or are commonly studied in the literature. Finally, conclusions are given in Section V. II. ERROR PROBABILITY OF 2-D SIGNALING This section begins with the system modeling. We then present our symbol error probability analysis of 2-D signalings in Rayleigh fading with channel estimation errors. Assume perfect symbol timing, intersymbol interference ISI) free transmission, and slow channel fading that is almost constant over one symbol duration. The received signal z at any symbol interval given signal s i s Ii js Qi i 1 M transmitted can be expressed as z gs i n 1) where the zero-mean Gaussian random variable R.V.) g g I jg Q represents the complex fading introduced by the channel with variance E s in both of its real and imaginary parts, and n n I jn Q is a zero-mean Gaussian noise R.V. with variance N 0 in both of its real and imaginary parts. The fading g is independent of the additive Gaussian noise n. Signal s i is assumed to be normalized to yield unit average energy of the constellation. Various channel estimation methods, for instance, pilot symbol assisted modulation and minimum mean square error estimation, generate a fading estimate ĝ that is also Gaussian and correlated with the true fading g. The specific expression for ĝ depends on the particular estimation method used. The most often used method for calculating the probability of error of a signaling format is based on the conditional error probability given the channel fading g and the fading estimate ĝ. Under this conditioning, the problem is similar to the well known additive white Gaussian noise AWGN) case. The probability of error is then obtained by averaging the conditional error probability over the joint distribution of g and ĝ. Previous publications [5]-[6] follow this approach. However, in the general case of arbitrary 2-D constellations, this approach often leads to an intractable analysis. Averaging over the joint

5 4 distribution of g αe jφ and ĝ ˆαe jˆφ usually involves a four-fold nested integration, and the conditional probability P e α ˆα φ ˆφ for arbitrary 2-D signaling usually has another level of integration. That is, P e P e α ˆα φ ˆφ f α ˆα φ ˆφ α ˆα φ ˆφ dαd ˆαdφd ˆφ 2) and P e α ˆα φ ˆφ requires a single integration for general 2-D signaling as seen in [2], [1]. We propose in this paper a completely different approach to solving the problem and arrive at a surprisingly simple yet general expression for the symbol error probability of arbitrary 2-D signaling in Rayleigh fading with channel estimation errors. We begin with defining a decision variable as D z ĝ g ĝ s i n ĝ s i t 3) And then, where t t z ĝ s i D s i 4) re jθ can be considered as an equivalent noise term that is superimposed on the signal s i and includes the effects of both the fading estimation error and the AWGN. Assuming equally-likely transmission of signals, the maximum likelihood ML) decision rule of a constellation in AWGN and in fading with perfect channel sate information is equivalent to the minimum Euclidean distance criterion. In the case of imperfect channel estimation, however, minimum Euclidean distance rule is no longer the optimum. Nevertheless, in practice a receiver will likely use the estimated fading as if it were perfect channel state information and draw up decision boundaries as in the AWGN case. This is because the receiver may not have the necessary information, such as the maximum Doppler frequency, the signal-to-noise ratio SNR) and so on, needed in the PDF of the decision variable to determine the optimum decision boundaries. Therefore, we consider the practical case where the minimum Euclidean distance rule is used in the receiver. If D falls into the decision region of signal s i, a correct decision will be made; otherwise, a symbol error will occur. The decision variable D was also written as the sum of the signal and an equivalent noise term in [7] and [8]. Furthermore, the probability density function of the noise term t was obtained from the conditional

6 5 PDF as f t r θ f t r θ α ˆα φ ˆφ f α ˆα φ ˆφ α ˆα φ ˆφ dαd ˆαdφd ˆφ 5) which can be simplified for MMSE estimation as discussed in [7] and [8]. However, the four-fold nested integration in 5) usually cannot be solved for other practical channel estimation schemes. Here we present a new derivation of the PDF of t. Now, recognizing the fact that the decision variable D in 3) is the ratio of two complex-valued Gaussian random variables z and ĝ, we first derive the distribution of the ratio of two complex Gaussian R.V. s X z and Y ĝ. Define variable D in a general form as D X Y 6) where X and Y are correlated zero-mean complex-valued Gaussian random variables. The joint PDF of zero-mean complex Gaussian R.V. s X and Y was given by Wooding in a simple and compact form as [10] f X Y X Y 2 L 1 exp V H L 1 V 7) where V X Y H with representing complex conjugation and A H the Hermitian transpose of matrix A. Matrix L is the Hermitian covariance matrix given by L m xx m xy mxy m yy where m xx E XX, m yy E YY and m xy E XY are the second moments of the zero-mean complexvalued Gaussian R.V. s X and Y. Eqn. 7) can also be written as 8) f X Y X Y 2 L 1 exp L 1 m yy X 2 m xx Y 2 mxyxy m xy X Y 9) and in polar coordinates X r x e jθ x and Y r y e jθ y, f X Y r x θ x r y θ y r x r y 2 L 1 exp L 1 m yy r 2 x m xx r 2 y 2R mxyr x r y e j θ x θ y! " 10)

7 To obtain the PDF of D r d e jθ d the following relationships X Y, we define another complex variable F r f e jθ f Y. Then, we have 6 r d r x r y θ d θ x θ y 11a) 11b) r f r y 11c) θ f θ y 11d) The joint PDF of D and F is obtained from 10) as f D F r d θ d r f θ f f X Y r d r f θ d θ f r f θ f J r d 2 r L 3 f exp # m yy rd 2 m xx 2R mxyr d e jθ d! r 2 f L $ 12) where J is the Jacobian of the transformation 11) and it can be shown that J 1 r y 1 r f. The PDF of the complex variable D is then obtained by integrating the joint density 12) with respect to the complex variable F, f D r d θ d 0 L 0 2 f D F r d θ d r f θ f dθ f dr f m yy r 2 d m xx The next step is to derive the PDF of the combined noise term t D D x jd y given by 13) in rectangular coordinates, we have f D D x D y L m yy D 2 x D 2 y m xx 1 r d 2R mxyr d e jθ d! 2 13) 2R m xy! D x re jθ in 4). Rewriting the PDF of 2I m xy! D y 2 14) It can be shown that 14) is equivalent to [9, eqn. 3)] but 14) is derived using a different method. Moreover, our later application of Craig s method to general 2-D signaling schemes will require the PDF of the

8 7 combined noise term t in polar coordinate form. Since t t x jt y D s i, one has t x D x t y D y s Ii s Qi 15a) 15b) The Jacobian of transformation 15) is J 1, and thus the PDF of t can be written as L f t t x t y m yy tx 2 ty 2 2 m yy s Ii R m xy! t x 2 m yy s Qi I m xy! t y m yy s i 2 m xx 2R m xy! s Ii 2I m xy! s Qi 2 16) Furthermore, the PDF of the complex-valued variable t expressed in polar coordinates can be obtained from 16) by another change of variables r &% tx 2 ty 2 and θ tan 1 t y' t x as f t r θ L r cr 2 b θ r a 2 17a) where a m yy s i 2 m xx 2R m xy! s Ii 2I m xy! s Qi 17b) b θ 2 m yy s Ii cosθ m yy s Qi sinθ R m xy! cosθ I m xy! sinθ 17c) c m yy 17d) Once the PDF 17) of the equivalent noise term t accounting for both the channel estimation error and the additive white Gaussian noise is available, we apply Craig s method [11], [1], [2] to analyze the probability of symbol error for arbitrary 2-D signaling. The main idea of Craig s method is to work directly with the PDF of the noise term that is superimposed onto the signal s i and divide erroneous decision regions into smaller subregions. Each subregion can be conveniently represented by polar coordinates [11]. Since we have expressed the decision variable D as the sum of s i and t, we shift the origin of the coordinates to signal s i s Ii js Qi to form new coordinates and work directly with t. Fig. 1 illustrates how to partition

9 / 8 the erroneous decision region of signal s i into subregions. Following analysis similar to that presented in [2], the probability of error for the j-th erroneous subregion is given by P e j R θ θ 2) j θ 1) j R θ x j sinψ j f t r θ drdθ sin * θ θ 1 j ψ j 18a) 18b) where angles θ 1 j θ 2 j ψ j and distance x j are parameters specific to the geometry of the j-th erroneous subregion and are illustrated in Fig. 1 for different erroneous subregions. It can be seen from 17) that the PDF of the combined noise t is no longer Gaussian and does not have circular symmetry as in the perfect channel state information case. The integration limits θ 1 j and θ 2 j must be angles related to the j-th subregion in the new coordinates s i is the origin). In the open decision region case shown in Fig. 1b), θ 1 j is defined as the angle formed by the horizontal axis of the new coordinates and the line from signal s i to the intersection of two decision boundaries. The absolute value is needed in 18a) as θ 1 j might be larger than θ 2 j, resulting in a negative value of the integral in 18a). Subregion 5 in Fig. 1b) is one example where θ θ 2 5. Substituting 17) into 18) and solving the inner integral with respect to r, we get P e j θ 2) j θ 1) j L, 2a b θ R θ a b θ R θ cr 2 θ b θ 3-2 where [12, eqns ) and 2.172)] have been used and 4ac b θ 2 2b θ 3- tan 1. b θ 2cR θ dθ 19a) 4 m xx m yy m xy 2 4 m yy s Ii sinθ s Qi cosθ I m xy! cosθ R m xy! sinθ b) 2 L m xx m yy m xy 19c) m xx E zz 19d) m yy E ĝĝ 19e) m xy E zĝ 19f)

10 / / 9 The symbol error probability of any coherent two-dimensional constellation in Rayleigh fading with channel estimation errors is simply a weighted sum of terms given by eqn. 19a) [2], P e N w j P e j 20) j3 1 where N is the total number of distinct erroneous subregions of the 2-D constellation and w j is the weighting coefficient [2]. The new single integral expression 19) is an exact solution to calculating the SER of coherent 2-D signaling in Rayleigh fading with imperfect channel estimation. Note that only one level of integration is required, in contrast to four levels of nested integration required by previous methods. As a special case of 2-D signalings, the symbol error probability of MPSK in Rayleigh fading with channel estimation errors can be obtained by assuming s i where P e MPSK M L, b 1 θ M b 1 θ L, 1 as 2a 1 sin 2 θ ' M b 1 θ sin ' M sin θ ' M 1 a 1 sin 2 θ ' M b 1 θ sin ' M sin θ ' M csin 2 ' M 2b 1 θ 3-1 tan. b 1 θ sin θ ' M 2csin ' M 2 dθ sin θ 1 ' M 1 0 $ 2a 1 sin 2 θ ' M b 1 θ sin ' M sin θ ' M 1 a 1 sin 2 θ ' M b 1 θ sin ' M sin θ ' M csin 2 ' M 2b 1 θ 3-1 tan. b 1 θ sin θ ' M 2csin ' M 2 dθ sin θ 1 ' M 1 0 $ 21a) a 1 m yy m xx 2R m xy! 21b) b 1 θ 2 m yy cosθ R m xy! cosθ I m xy! sinθ 21c) c m yy 1 4 m xx m yy m xy 2 4 m yy sinθ I m xy! cosθ R m xy! sinθ 2 21d) 21e) Note that the two integrals on the right side of 21a) are equal when the second moment m xy is real, i.e., I m xy! 0 and, hence, b 1 θ b 1 θ. Thus, when a channel estimation method has the property that

11 10 the cross-correlation between the received signal z and the channel estimate ĝ is real, the probability of symbol error for MPSK is given by doubling the first integral. As will be shown in the next section, this is the case for pilot symbol assisted modulation. It is worth pointing out that for MPSK, it is actually easier to work with the decision variable D rather than t, because the erroneous decision region is more conveniently described by the non-shifted original coordinates and the SER is given by P e MPSK M M M 0 f D r d θ d dr d dθ d f θ d dθ d L # 2 2 M L # 2 2 M b 2 θ d b 2 θ d f θ d dθ d M 2b 2 θ d b 2 θ d f D r d θ d dr d dθ d 1 tan b 2 θ / d 2 $ 1 tan b 2 θ / d 2 $ dθ d dθ d 22a) 22b) where f θ d is the marginal PDF of θ d, the phase difference between the received signal and the channel phase estimate, and b 2 θ d 22c) 2 R m xy! cosθ d I m xy! sinθ d 22d) 2 4m xx m yy 4 R m xy! cosθ d I m xy! sinθ d 2 22e) Another special case is the symbol error probability of M-ary rectangular QAM, which is often referred to as MQAM. This is readily obtained using 20) and 19), though the details are omitted for brevity. More generally, we have written a computer program that calculates these geometric parameters automatically for arbitrary 2-D constellations. III. APPLICATIONS TO CHANNEL ESTIMATION ERROR ANALYSIS This section applies the newly derived method to the performance analysis of 2-D signalings in the presence of three different kinds of channel estimation errors.

12 11 A. Static Channel Estimation Errors Two parameters, amplitude error tolerance and phase error tolerance, were defined in [2] to describe how sensitive a 2-D constellation is to carrier amplitude and phase tracking errors. These two parameters provide some information about the robustness of a modulation format, but the effect of channel estimation errors on the average SER of the modulation format is not known with accuracy. Here we investigate the performance of 2-D signaling in Rayleigh fading with constant channel estimation errors caused by the carrier amplitude and phase tracking components. Another scenario where static channel estimation error is relevant is when amplifier nonlinearity is present in the system and the estimated amplifier nonlinearity differs from the true amplifier nonlinearity by some constant error. In all these cases, the estimated fading is related to the real fading by a constant factor. That is, ĝ g qexp jφ where q is the amplitude estimation error and φ is the phase estimation error. Parameters q 1 and φ 0 correspond to the perfect channel estimation case. Therefore, the fading estimate ĝ is also a zero-mean complex Gaussian R.V. correlated to the true fading g. Letting X z gs i n and Y ĝ transmitted are g qexp jφ, the second moments given signal s i m xx 2E s s i 2 2N 0 23a) m yy m xy 2E s q 2 23b) 2E s s i q e jφ 23c) The signal-to-noise ratio SNR) per symbol is given by E s' N 0. The SER of arbitrary 2-D signaling in the presence of static channel estimation errors can be readily obtained from 19) and 23). The second moments are obtained from 23) with s i 1 for MPSK. The probability of error for MPSK in Rayleigh fading with static channel estimation errors is then given by 21) and 23). B. Pilot Symbol Assisted Modulation In a PSAM system, pilot signals are periodically inserted into the data stream for every L 1 data symbols. Hence, a data frame of length L consists of a pilot symbol at its zero-th position and L 1 data

13 12 symbols at positions from 1 to L 1. The estimated channel fading at the pilot position of the k-th frame is obtained by ĝ k p z k 0' p k where p k is the pilot symbol of the k-th frame and z k 0 is the received signal at this pilot position. The received signal at the l-th data symbol position of the current frame is given by z l g l s l n l 24) where g l is the zero-mean complex Gaussian R.V. representing the multiplicative channel fading, s l is the l-th data symbol taking values from an M-ary signal set and n l is the additive Gaussian noise. Channel fading g l is estimated from K pilot symbols, that is, the previous K 1 54 K 1 ' 26 pilot symbols, the current pilot symbol and the subsequent K 2 74 K' 26 pilot symbols as ĝ l K 2 k3 K 1 h k lĝ k p K 2 k3 K 1 h k l 8 g k p n k p p k 9 25) where 4;: 6 is the floor function, g k p is the actual channel fading at the k-th pilot position, and h k l kl is the interpolator coefficient for ĝ k p and is dependent on the current symbol position l. Proposed interpolators include the optimum Wiener interpolator [3], a Gaussian interpolator [13] and the simple sinc interpolator [4]. As an example, the sinc interpolator with a Hamming window is given by h l h n w n : sinc < n L= K 1 L > n > K 2 L 26a) where w n cos 8 2n KL KL 2 6 KL b) It can be seen from 25) that ĝ l is also a zero-mean complex Gaussian R.V. because it is a linear combination of ĝ K 1 p ĝ K 2 p that are jointly Gaussian R.V. s. Let X z l and Y ĝ l. To calculate the SER of

14 13 PSAM 2-D signaling, the only quantities needed from 19) are the second moments m xx 2E s s l 2 2N 0 27a) m yy H l CH T 2N 0 l p k 2 H 2 l 27b) m xy K 2 2E s h k ls l J 0 2 f D kl l T 27c) k3 K 1 H l h K 1 l h K2 l 27d) C kn E g k pg n p 2E s J 0 2 f D k n T 27e) where C kn is the element with index k n of the covariance matrix C for fadings at pilot symbols, f D is the maximum Doppler frequency and T is the symbol duration. Pilot symbol p k is assumed to have unit average energy, i.e., p k 2 1 in 27). The signal-to-noise ratio per symbol is given by E s' N 0. Since the second moments m yy and m xy are dependent on the symbol position l where l symbol error probability should be averaged over the L 1 positions within a frame. 1 L 1, the overall In the case of pilot symbol assisted MPSK, the second moments are given by 27) with s l 1. Since m xy in 27c) is real, the two integrals in 21a) are equal. The probability of error for pilot symbol assisted MPSK in Rayleigh fading is thus given by two times the first integral in 21a). C. Minimum Mean Square Error Estimation It is well known that the estimate and the estimation error are uncorrelated in minimum mean square error estimation and that the average estimation error power is the difference of the power of the variable to be estimated and the power of the estimate. Since the fading and its estimate are complex Gaussian R.V. s, MMSE estimate ĝ and the estimation error e g ĝ are independent. Denote the mean power of the estimation error by 2σ 2 e and the fading power by 2E s. The estimation error power is a measure of the quality of the estimation algorithm. Let X z and Y ĝ. The second moments required by 19) in the

15 / / 14 case of MMSE channel estimation are given by This leads to m xx 2E s s i 2 2N 0 28a) m xy E gs i n ĝ E e ĝ s i g ˆ m yy s i 28b) m yy 2E s 2σ 2 e 28c) a m xx m yy s i a) Substituting 29) into 19), we have for MMSE estimation b 0 29b) c m yy c) e j P MMSE θ 2) j θ 1) j L sin 2 * θ θ 1 j ψ j 2c asin 2 * θ θ 1 j ψ j cx 2 j sin2 ψ Eqn. 30) can be further simplified to a closed-form expression given by Pe MMSE j θ 2 j 2 θ 1 tan 1 1 cu j 2% cu 2 j a a cu 2 j tan * θ 2 j θ 1 j ψ j dθ tan ) a cu 2 j tanψ j CBD 31) where u j x j sinψ j + 0 and [12, eqn )] has been used. For MMSE MPSK, the symbol error probability can be written in closed-form as Pe MMSE MPSK M 1 M cu / cu 2 a, tan 1.FE 1 where u sin M, and a and c can be obtained from 29) and 28) with s i 1. a cu 2 tan M 0G1 Both 31) and 32) are new results. Substituting 29) into 17), we obtain an equivalent PDF expression in polar coordinates for the combined noise term to that given by [7, eqn. 22)]. Polar coordinates facilitate 32)

16 15 the present analysis and lead to simplified and generalized results for arbitrary 2-D signaling. While [7, eqn. 22)] is valid only for MMSE channel estimation, 17) applies to any channel estimations that are jointly Gaussian with the actual channel fading. Note that our new closed-form SER expression 31) applies to arbitrary polygonal 2-D constellations with MMSE estimation in Rayleigh fading, compared to the single integral results given in [7] for SER s of 4-PSK, 8-PSK, 8-AMPM, 16-PSK and 16-QAM. IV. NUMERICAL EVALUATIONS We present in this section some numerical results for various 8-ary and 16-ary constellations as applications of the theoretical analysis presented in Sections II and III. The 8-ary signalings considered are 8PSK, the 8-ary rectangular set, the 8-ary max-density, the 8-ary triangular set, 4,4) and 1,7). The 16-ary signalings considered are the 16-ary hexagonal set, 16 rectangular QAM, the 16-ary triangular set, 4,12), the 16-ary max-density, 5,11), 16 star-qam, rotated 8,8) and 1,5,10). The signal space diagrams for these constellations can be found in [2] and are not presented here due to space limitations. Figs. 2-4 show the SER s of the 8-ary and 16-ary signal sets as functions of the average signal-to-noise ratio per bit, E b N 0, in the presence of static channel estimation errors. The SNR per bit E b N 0 is related to E s N 0 by a factor of log 2 M 1. The derived single integral 19) with finite integration limits is easy to evaluate numerically and generates results with high accuracy. Fig. 2 demonstrates the effect of constant channel estimation errors on the SER performance of 8-ary signal sets in Rayleigh fading. The performance of the 4,4) constellation will depend on the ring ratio employed. The ring ratio of 4,4) is set at , the optimized value for SNR 20 db. It is evident that the relative performances of these constellations with channel estimation errors are distinct from those with perfect channel state information as given in [2]. Given constant amplitude error q error φ 2 db and phase 10H, 4,4) and 1,7) have better error performance than 8PSK and the 8-ary rectangular set in a Rayleigh fading environment. The 1,7) set saves about 2.04 db power over 8PSK at SER 4,4) saves around 1.96 db over 8PSK as shown in Fig. 2. 1e 3 and The performances of some 16-ary constellations are presented in Figs. 3 and 4 for the ring ratios speci-

17 16 fied in the figure captions. The performance differences among 16-ary signal sets in Figs. 3 and 4 are more pronounced than those of 8-ary signal sets in the presence of the same static channel estimation errors, as expected. Circular constellations with two rings such as 16 star-qam, rotated 8,8), 5,11) and 1,5,10) are clearly good choices that are robust to constant channel estimation errors as shown in Figs. 3 and 4. The 16-ary hexagonal set, 16 rectangular-qam, the 16-ary max-density and the 16-ary triangular set no longer yield acceptable performance in the presence of the amplitude error q φ 2 db and the phase error 10H. Sixteen star-qam and rotated 8,8) signaling have comparable error performances. Comparing Fig. 3 with Fig. 4, it is seen that under-estimating the channel amplitude has less adverse effect on the SER than over-estimating the channel amplitude with the same q db. Some simulation results are plotted in Figs. 2 and 4 to verify the analytical SER results. Figs. 5-9 present SER results for pilot symbol assisted modulation schemes which introduce dynamic channel estimation errors to receiver detection. As described in Section III-B, we use a sinc interpolator with Hamming windowing in our numerical evaluations. Note that the general SER expression 19) is applicable to any linear interpolator and our study shows that Hamming windowing produces better performance than rectangular windowing. Fig. 5 depicts the SER s of pilot symbol assisted 16 rectangular-qam and 16 star-qam as a function of SNR in Rayleigh fading. Sixteen rectangular-qam slightly outperforms 16 star-qam for the perfect channel state information case, and the benefit is greater at higher SNR. However, it is clear from Fig. 5 that when dynamic channel estimation errors inherent in PSAM schemes are present, the performance degradation of 16 rectangular-qam is quite severe. The SNR difference by numerical evaluation for star-qam with and without channel estimation error at SER=1e 3 is about 2.4 db, while for rectangular-qam, the SNR difference is about 3 db. This shows that star-qam is more robust to channel estimation errors than rectangular-qam. Pilot symbol assisted 16 star-qam has a slightly better performance than pilot symbol assisted 16 rectangular-qam, and at very high SNR both constellations exhibit error floors. The error floor of 16 star-qam, however, is less than that of 16 rectangular-qam. Simulation results agree well with our theoretical analysis. It is also observed in our study that performance differences between the two constellations are more pronounced at higher fading

18 17 rates when channel estimation errors increase. The choice of two parameters, frame length L and interpolation order K, depends on a number of factors. The larger the frame length L, the less power loss incurred from the pilots inserted in the data 1 stream. However, L is upper bounded by the Nyquist Sampling Theorem at L > 2 f D T. For example, L should be less than 16 for f D T Therefore we choose L 15. We found that the PSAM system would not work with larger values of L, and smaller L does not lead to much performance gain. Once L is determined, the SER performance of 16 rectangular-qam as a function of interpolation order K is plotted in Fig. 6. Parameter K determines the buffer size and the detection delay and should be minimized without sacrificing the error performance. As can be seen from Fig. 6, the smaller the average SNR of the channel, the smaller the value of K that is sufficient. It is obvious that choosing K 30 works well for SNR values less than 40 db, but is not sufficient for SNR = 50 db, where K 34 reduces error floors noticeably. It is observed in our study that K depends on the channel fading rate, the SNR and the frame size L. It does not seem to depend strongly on the signal constellation used. The ring ratio of pilot symbol assisted 16 star-qam requires optimization for minimum SER. Note that in slow Rayleigh fading with perfect channel state information, the optimum ring ratios are a function of SNR and approach an asymptotic value as SNR gets very large [2]. The asymptotic optimum ring ratio is in Rayleigh fading. Fig. 7 shows the optimum ring ratio of PSAM 16 star-qam as a function of SNR for f D T 0 03, L 15, K 30 and 34. Obviously the optimum ring ratios no longer approach an asymptotic value and are quite different from those of the perfect channel knowledge case. The optimum ring ratio of 16 star-qam is chosen at 2.18 in Fig. 5. Fig. 8 shows that pilot symbol assisted 4,4) achieves the best SER performance among 8-ary signal sets with f D T 0 03, L 15, and K 30 in Rayleigh fading. The ring ratio used for 4,4) is , optimized for SNR SER 20 db. There is about 1 6 db power savings for PSAM 4,4) over PSAM 8PSK at 1e 3, which is rather significant when it comes to system design for high speed data. Error floors are observed for all signalings at high SNR. The SER s of pilot symbol assisted 16-ary signal sets in Rayleigh fading are plotted in Fig. 9 with

19 18 f D T 0 03, L 15, and K 30. Fig. 9 shows that 16 star-qam has very comparable performance to the rotated 8,8) set and achieves the lowest SER among the five signal sets in the medium to high SNR range. Although not plotted here, it is also found in our study that the constellation 5,11) curve is slightly above that of the 16-ary max-density, and 1,5,10) almost overlaps with rotated 8,8). The ring ratios used for 16 star-qam and rotated 8,8) in Fig. 9 are and , respectively. Fig. 10 plots the SER curves of 16 rectangular-qam, 16 star-qam and QPSK in Rayleigh fading when MMSE estimation errors are present. Suppose the variance of the estimation error is 5 db, 16 star-qam suffers 4.86 db power loss while 16 rectangular-qam loses 5.8 db power for SER at 1e 3, compared to their corresponding perfect coherent detection. In addition, for symbol error probabilities smaller than 1e 2, 16 star-qam saves about 0.6 db power over 16 rectangular-qam with σ 2 e 5 db. The QPSK curves obtained agree with the results in [8, Fig. 6], except that the horizontal axis in Fig. 10 is SNR per bit whereas it is SNR per symbol in [8, Fig. 6]. V. CONCLUSION The impact of channel estimation errors on the performance of coherent 2-D signaling has been investigated in this paper. A new, simple, general method of analysis has been developed for performance evaluation of arbitrary polygonal 2-D signaling in Rayleigh fading with imperfect channel estimation. The symbol error probability of an arbitrary polygonal 2-D constellation is given by a single integral with finite integration interval and is well-suited to numerical evaluation. The practical effects of channel estimation errors on signaling performance have been examined by considering static channel estimation errors, pilot symbol assisted modulation and minimum mean square error estimation. The SER expression of arbitrary 2-D signaling with MMSE estimation was further simplified to a closed form. The symbol error probabilities of six 8-ary signal sets and nine 16-ary signal sets have been numerically evaluated and plotted under different kinds of channel estimation errors. Pilot symbol assisted 4,4) displays significant power advantage over conventionally used PSAM 8PSK. Pilot symbol assisted 16 star-qam has been shown to be more immune to channel estimation errors than PSAM 16 rectangular-qam and its performance

20 19 is comparable to PSAM rotated 8,8). It has also been demonstrated that parameters crucial to channel estimation schemes and ring ratios of circular constellations can be easily and accurately determined using the analytical results presented. ACKNOWLEDGEMENT The first author wishes to thank Mr. Lei Xiao for his assistance in simulations. REFERENCES [1] X. Dong, N.C. Beaulieu and P.H. Wittke, Error Probabilities of Two-Dimensional M-ary Signaling in Fading, IEEE Trans. Commun.,vol. 47, pp , Mar [2] X. Dong, N.C. Beaulieu and P.H. Wittke, Signal constellations for fading channels, IEEE Trans. Commun., vol. 47, pp , May [3] J.K. Cavers, An Analysis of Pilot Symbol Assisted Modulation for Rayleigh Fading Channels, IEEE Trans. Veh. Technol., vol. 40, pp , Nov [4] Y.S. Kim, C.J. Kim, G.Y. Jeong, Y.J. Bang, H.K. Park and S.S. Choi, New Rayleigh Fading Channel Estimator Based on PSAM Channel Sounding Technique, Proc. IEEE Int. Conf. Communications ICC 97), pp , June [5] X. Tang, M.S. Alouini and A. Goldsmith, Effect of channel estimation error on M-QAM BER performance in Rayleigh fading, IEEE Trans. Commun., vol. 47, pp , Dec [6] K. Yu, J. Evans and I. Collings, Performance Analysis of Pilot Symbol Aided QAM for Rayleigh Fading Channels, Proc. IEEE Int. Conf. Communications ICC 02), Apr [7] A. Aghamohammadi and H. Meyr, On the Error Probability of Linearly Modulated Signals on Rayleigh Frequency-Flat Fading Channels, IEEE Trans. Commun., vol. 38, pp , Nov [8] M.G. Shayesteh and A. Aghamohammadi, On the Error Probability of Linearly Modulated Signals on Frequency-Flat Ricean, Rayleigh, and AWGN Channels, IEEE Trans. Commun., vol. 43, pp , Feb./Mar./Apr [9] S.K. Wilson and J.M. Cioffi, Probability Density Functions for Analyzing Multi-Amplitude Constellations in Rayleigh and Ricean Channels, IEEE Trans. Commun., vol. 47, pp , Mar [10] R.A. Wooding, The Multivariate Distribution of Complex Normal Variables, Biometrika, vol. 43, pp , June [11] J.W. Craig, A New Simple and Exact Result for Calculating the Probability of Error for Two-Dimensional Signal Constellations, Proc. IEEE Milit. Commun. Conf. MILCOM 91), Boston, MA, pp , [12] I.S. Gradshteyn and I.M. Ryzhik, A. Jeffrey editor), Table of Integrals, Series, and Products. Fifth Edition, San Diego, CA: Academic Press, [13] S. Sampei and T. Sunaga, Rayleigh fading compensation for QAM in land mobile radio communications, IEEE Trans. Veh. Technol., vol. 42, pp , May 1993.

21 20 2 B C E 1 S i O θ 1,1 θ 2,1 θ x 1 R ψ 1 3 A D 4 R = OE x = OA 1 0 a) 5 F 6 ψ 5 x 5 θ x 1,5 S 7 G 7 θ 2,5 i O θ1,7 ψ θ 2,7 7 x = OF 5 x = OG 7 0 b) Fig. 1. Decision regions and geometric parameters of a a) closed region and b) open region.

22 PSK 8PSK simulation 8 Rectangular 8 Max density 8 Triangular 4,4) 1,7) Symbol Error Probability q=2 db, φ=10 o Average E b /N 0 db) Fig. 2. Average SER s of 8-ary signal sets in Rayleigh fading with amplitude error q 3 2 db and phase error φ 3 10I.

23 Symbol Error Probability q=2 db, φ=10 o 16 rect. QAM Triangular Hexagonal 4,12) Max density 5,11) 16 star QAM Rot. 8,8) 1,5,10) Average E b /N 0 db) Fig. 3. Average SER s of 16-ary signal sets in Rayleigh fading with amplitude error q 3 2 db and phase error φ 3 10I. The ring ratios used for 1,5,10), rotated 8,8), 16 star-qam, 5,11) and 4,12) are , , , and , respectively.

24 Symbol Error Probability q= 2 db, φ=10 o 16 Hexagonal 16 rect. QAM 16 rect. QAM simulation 16 Max density 16 Triangular 4,12) Rot. 8,8) 16 star QAM 16 star QAM simulation 5,11) 1,5,10) Average E b /N 0 db) Fig. 4. Average SER s of 16-ary signal sets in Rayleigh fading with amplitude error q 3 2 db and phase error φ 3 10I. The ring ratios used for 1,5,10), rotated 8,8), 16 star-qam, 5,11) and 4,12) are , , , and 3.497, respectively.

25 rect. QAM, simulation 16 star QAM, simulation 16 rect. QAM, theory 16 star QAM, theory 16 rect. QAM, perfect 16 star QAM, perfect Symbol Error Probability L=15, K=30, f D T= Average E b /N 0 db) Fig. 5. Average SER s of PSAM 16 rectangular-qam and 16 star-qam in Rayleigh fading with f D T 3 0J 03 L 3 15 K 3 30J

26 Symbol Error Probability rect. QAM f D T=0.03 L= SNR=10 db SNR=20 db SNR=30 db SNR=40 db SNR=50 db K Fig. 6. Average SER s of PSAM 16 rectangular-qam as a function of K in Rayleigh fading with f D T 3 0J 03 L 3 15J

27 K=30 K= star QAM L=15 f D T=0.03 Optimum Ring Ratio Average E b /N 0 db) Fig. 7. Optimum ring ratios of PSAM 16 star-qam in Rayleigh fading with f D T 3 0J 03 L 3 15 K 3 30 and 34J

28 PSK 8PSK simulation 8 Rectangular 1,7) 8 Max density 8 Triangular 4,4) Symbol Error Probability L=15, K=30, f D T= Average E b /N 0 db) Fig. 8. Average SER s of PSAM 8-ary signal sets in Rayleigh fading with f D T 3 0J 03 L 3 15 K 3 30J

29 Hexagonal 16 rect. QAM Max density Rot. 8,8) 16 star QAM Symbol Error Probability L=15, K=30, f D T= Average E b /N 0 db) Fig. 9. Average SER s of PSAM 16-ary signal sets in Rayleigh fading with f D T 3 0J 03 L 3 15 K 3 30J

30 rect. QAM, MSE=5 db 16 star QAM, MSE=5 db 16 star QAM, perfect 16 rect. QAM, perfect QPSK, MSE=1/3 [8] QPSK, perfect Symbol Error Probability Average E b /N 0 db) Fig. 10. Average SER s of 16 star-qam and 16 rectangular-qam in Rayleigh fading with MMSE estimation and σ 2 e 3 5 db, and the SER of QPSK with σ 2 e