THE problem of noncoherent detection of frequency-shift

IEEE TRANSACTIONS ON COMMUNICATIONS, VOL. 45, NO. 11, NOVEMBER 1997 1417 Optimal Noncoherent Detection of FSK Signals Transmitted Over Linearly Time-Selective Rayleigh Fading Channels Giorgio M. Vitetta, Member, IEEE, Umberto Mengali, Fellow, IEEE, Desmond P. Taylor, Fellow, IEEE Abstract Linearly time-varying fading models are used to investigate noncoherent detection of frequency-shift keying (FSK) signals transmitted over frequency-flat fading channels. The structure of the optimal noncoherent FSK detector is derived a novel analytical technique is proposed to compute the error performance of noncoherent FSK detectors in fast fading. Error performance results obtained by computer simulation are in excellent agreement with the analytical predictions. Index Terms Diversity reception, fading channels, FSK signals, noncoherent detection. I. INTRODUCTION THE problem of noncoherent detection of frequency-shift keying (FSK) signals has been widely studied in the technical literature of the fifties the sixties [1] [8]. Most of the research efforts have been centered on optimal diversity noncoherent detection for slow time-selective fading channels (see, for instance, [1] [4]). The slow fading model holds true when the channel variations over a symbol interval are very limited, in consequence, have negligible effects on the receiver error performance. Otherwise, the fading is deemed fast. In the presence of fast fading, the error rate curves of conventional matched-filter receivers exhibit a floor as the signal-to-noise ratio (SNR) increases. The effect of the fading spectrum on this floor is discussed in [9] an analytical technique is proposed to evaluate the error performance of noncoherent receivers in fast fading. The error floor problem is also encountered in other noncoherent FSK receivers, namely differential discriminator detectors [10]. Their performance in fast fading has been tackled in more recent publications [11] [13]. In this paper, the power series fading model proposed by Bello [14] is applied to noncoherent detection of FSK signals in the presence of fast fading. Following this approach a noncoherent receiver structure (called double-filter receiver) is derived which is optimum for linearly time-selective fading channels performs better than conventional noncoherent Paper approved by N. C. Beaulieu, the Editor for Wireless Communication Theory of the IEEE Communications Society. Manuscript received June 18, 1996; revised December 16, 1996. This paper was presented in part at the IEEE ICC 97, Montreal, Canada, June 1997. G. M. Vitetta U. Mengali are with the Department of Information Engineering, University of Pisa, 56126 Pisa, Italy. D. P. Taylor is with the Department of Electrical Electronic Engineering, University of Canterbury, Christchurch, New Zeal. Publisher Item Identifier S 0090-6778(97)08204-4. detectors [1] in fast fading. Also, a novel technique is proposed for computing the error performance of both the matched filter detector [1] the double-filter detector in fast fading. The benefit of receiver diversity is investigated. The paper is organized as follows. Section II describes the mathematical model of the fading process establishes basic notations for FSK signals. In Section III, we propose a new method for computing the error probability of a conventional FSK noncoherent detector in fast fading. The double-filter detector its error performance are studied in Section IV. Performance comparisons between the single-filter doublefilter receivers are made in Section V. Finally, Section VI offers some conclusions. II. CHANNEL AND SIGNAL MODELS The complex baseb expression for an -ary FSK signal is given by [10] (2.1) (2.2) (2.3) In these equations, is the information bearing function, is the signal energy, is the symbol period, is the modulation index, is the th data symbol belonging to the -ary alphabet, is the signal phase in the th signaling interval. This description encompasses the class of continuous phase FSK (CPFSK) signals [10] for which varies according to (2.4) In this paper, only binary FSK signals are considered although the receiver structures described in the following are easily extended to general -ary FSK. We assume that the receiver is provided with independently fading replicas of the same information bearing signal. Thus, the transmission of signal (2.1) results in the following received waveforms: (2.5) 0090 6778/97$10.00 1997 IEEE

1418 IEEE TRANSACTIONS ON COMMUNICATIONS, VOL. 45, NO. 11, NOVEMBER 1997 Fig. 1. Block diagram of the noncoherent single-filter receiver. for, is the explicit diversity order of the receiver, is the fading distortion affecting the th channel, is additive white Gaussian noise (AWGN) with two-sided spectral density (independent of ). The following assumptions are made. is a zero-mean Gaussian wide-sense stationary (WSS) process whose sample functions are differentiable in mean square sense [15]; the fading processes are mutually independent; the noise processes are mutually independent. Denoting as the fading distortion over the symbol interval centered about, i.e., exping into a Taylor series yields (2.6) (2.7) (2.8) are Gaussian rom variables. In Appendix A, it is shown that the cross correlation between is expressed by index for which the maximum of the sum attained, i.e., is (3.2) In the following, the detector following this strategy is referred to as the single-filter receiver because there is one matched filter (in each diversity branch) for each symbol of the alphabet. B. Error Performance To compute the error performance of the single-filter detector in fast fading, we adopt a linearly time-selective channel model [14]. This involves approximating the fading distortion by the first two terms of its Taylor expansion. In other words, we set (3.3) Without loss of generality, we assume that. Correspondingly, it can be shown that the matched filter outputs take the form (3.4) (2.9) is the autocorrelation function of the fading process. III. SINGLE-FILTER NONCOHERENT FSK DETECTION A. Receiver Structure In this subsection, we overview the maximum-likelihood noncoherent detector [1] for slow time-selective fading channels. The receiver structure is shown in Fig. 1 employs two (complex-valued) matched filters (3.1) according to whether equals 0 or 1. Let denote the output sample from on the th diversity branch at. Then the optimal decision strategy [2] is to decide, is that (3.5) (3.6) (3.7) (3.8) is the noise contribution to the output of the filter in the th receiver branch. It turns out that is

VITETTA et al.: OPTIMAL NONCOHERENT DETECTION OF FSK SIGNALS 1419 a zero mean Gaussian rom variable with variance such that probability is computed by letting reads in (D.7) The detector error probability is expressed as (3.9) (3.10) (3.17) are the eigenvalues of. The following observations are of interest. 1) In the case of orthogonal FSK signalling over a slow fading channel, (3.16) can be simplified. In fact, setting in the eigenvalue formulas (B.5) (B.6) substituting into (3.16) yields is a matrix with elements. The last term in (3.10) can be computed with the methods indicated in [4] [16]. For this purpose, we denote by the autocorrelation matrix of (whose coefficients are given in Appendix B) define Next, the characteristic function of [17] [8, pp. 593 595]) (3.11) is found to be (see (3.12) are the eigenvalues of. Finally, the error probability (3.10) is computed as [16] (3.13) is the set of all the distinct eigenvalues of the matrices. In (3.13), the following notations have been used: (3.14) (3.15) is the residue of for. A general formula for the computation of these residues is given in Appendix D. In the sequel, we concentrate on two particular cases: 1) single-filter receiver no diversity ; 2) single-filter receiver with double diversity fading branches with the same statistics. In the first case, the error probability is obtained letting in (D.7). This produces (3.16), the eigenvalues of, are computed in Appendix B. It is worth noting that (3.16) coincides with Barrett s formula in [16]. With double diversity, the error (3.18) which is a well-established result in the communication theory literature (see, for instance, [8, p. 407]). 2) The technique proposed in this section to compute is numerically simpler than that illustrated by Bello Nelin in [9]. In fact, the present method requires only the parameters, which can be computed from the fading autocorrelation as. On the other h, Bello Nelin s technique requires the computation of integrals involving both the autocorrelation function the transmitted waveforms. IV. DOUBLE-FILTER FSK DETECTOR A. Receiver Structure The single-filter detector of the previous section is suboptimal as it does not exploit the time-varying nature of the channel fading. In the following, we derive the maximum likelihood noncoherent detector for a linearly time-selective fading channel. Assuming that explicit diversity branches are available at the receiver, it is easily shown that the maximum likelihood noncoherent strategy consists of making decision, (4.1) (4.2) represents the joint probability density function of the rom variables [see (3.3)]. In Appendix C, it is shown that (4.1) can also be written in the form (4.3)

1420 IEEE TRANSACTIONS ON COMMUNICATIONS, VOL. 45, NO. 11, NOVEMBER 1997 Fig. 2. Block diagram of the noncoherent double-filter receiver. with (4.4) (4.5) model of the fading distortion. In particular, we adopt a quadratically time-selective channel model [14] which is obtained by truncating the Taylor series (2.7) to the third term (4.6) (4.8) Note that in Section III-A matched filter has been already defined is the output sample from the Assuming found that, after considerable manipulations it is (4.9) (4.7) with. This detector, whose block diagram is shown in Fig. 2, may be classified as a generalized quadratic combining receiver (see [8, p. 510]). The following remarks are in order. The novel detection strategy exploits the intrinsic or implicit time diversity provided by time-selective fading through the filters. The separate contributions of these filters in (4.4) are a consequence of the orthogonality of the rom variables. 1 Under the assumption of a linearly time-selective fading channel the filter outputs represent the sufficient statistics for the ML noncoherent detector. Since the novel receiver employs two matched filters for each symbol of the alphabet, it will be called a double-filter receiver in the sequel. B. Error Performance The error performance of the th-order double-filter receiver in fast fading is now assessed using a power series 1 In general, such a property holds when the carrier frequency (implicit in the complex envelope representation of the signals) coincides with the centroid of the fading Doppler spectrum [14]. This certainly occurs if the Doppler spectrum is symmetrical (i.e., the fading autocorrelation function is even) as is assumed in the paper. (4.10) (4.11) (4.12) (4.13) (4.14) (4.15), is the noise sample from filter in the th receiver branch. The bit-error probability of the double-filter detector is given by (4.16)

VITETTA et al.: OPTIMAL NONCOHERENT DETECTION OF FSK SIGNALS 1421 (4.17). Denoting, the autocorrelation matrix of (whose elements are given in Appendix B) letting (4.18) it can be shown that the characteristic function of given by [9] is (4.19) are the eigenvalues of. Then, following the method in Section III-B, the bit-error probability can be found by replacing in (3.13) by (4.20) For single diversity, the error probability is obtained by letting in (D.7). This yields Fig. 3. BER performance of MSK differential discriminator detectors. (4.21) are the eigenvalues of. This coincides with Barrett s formula [16]. With double diversity, is obtained by letting in (D.7): (4.22) are the eigenvalues of (note that ). V. SIMULATION RESULTS A. Simulation Model Extensive simulations have been run to assess the performance of both the single-filter double-filter receivers. In doing so, a time-selective Rayleigh channel has been adopted. The fading process on each diversity branch is generated by passing two independent real Gaussian processes through two identical third-order Butterworth filters. The 3-dB bwidth of these filters,, is taken as a measure of the fading rate. The autocorrelation function of the fading process is the same for all the diversity channels is expressed by (5.1) Fig. 4. BER performance of the MSK single-filter receiver. The coefficients (2.9) involved in the analytical expression of the error probability are given in Appendix A. The error performance results illustrated in the following correspond to Doppler spreads in the range, i.e., in a fast-fading environment. Bit error rate (BER) results are derived as a function of the ratio of the average signal energy per bit to noise power density. B. Performance Comparisons A binary CPFSK with modulation index (MSK) has been considered because of its practical interest.

1422 IEEE TRANSACTIONS ON COMMUNICATIONS, VOL. 45, NO. 11, NOVEMBER 1997 Fig. 5. BER performance of the MSK double-filter receiver. Fig. 7. BER performance of the MSK double-filter receiver with double diversity. Fig. 6. BER performance of the MSK single-filter receiver with double diversity. BER curves with differential [11] limiter-discriminatorintegrator [12] detectors have been computed by simulation. In both cases, a fourth-order Butterworth FIR structure has been employed as a receiver front-end filter its 3-dB bwidth has been chosen equal to so as to minimize BER for around 30 db. Fig. 3 indicates that the error performance of these detectors is poor in the presence of fast fading. Poor performance is also obtained with a single-filter detector, as illustrated in Fig. 4. To the contrary, Fig. 5 shows that a double-filter receiver allows a substantial lowering of the error floor. A further Fig. 8. Floor level versus normalized fading bwidth for MSK single double-filter receivers. reduction in the error floor could be obtained by designing a ML noncoherent detector for a quadratically time-selective channel. However, as explained in Appendix B, the use of a such a channel model (or of higher order models) does not lead to a closed form solution for the detection strategy. The performance gap between single- double-filter receivers increases in the presence of explicit diversity. This is seen by comparing Figs. 6 7 which illustrate BER curves for single- double-filter receivers double diversity. As is seen, the error floor with single-filtering is clearly visible

VITETTA et al.: OPTIMAL NONCOHERENT DETECTION OF FSK SIGNALS 1423 10 (only analytical results are shown). As increases, the asymptotic error performance with double-filtering improves much more than with single-filtering. Fig. 9. Floor level versus normalized fading bwidth for MSK single-filter double-filter receivers with double-diversity. VI. CONCLUSION Power series models for fading distortion prove to be very useful in deriving noncoherent detection schemes for FSK signals transmitted over fast frequency-flat fading channels. In fact, we have found that: they provide novel analytical tools for accurate estimation of the error performance of noncoherent detectors in fast fading channels; the optimal noncoherent detector for linearly timeselective fading channels ensures a substantial lowering of the error floor over a conventional (single-filter) noncoherent detector; the performance gap between single- double-filter receivers increases with the number of explicit diversity channels /or the modulation index; the double-filter receiver exploits the implicit diversity due to Doppler spreading. It should be noted that in static or time-invariant fading, there is no performance gain due to this effect as the coefficient of (3.3) then goes to zero. APPENDIX A In this Appendix, we prove formula (2.9) in the text. Using (2.8), we have (A.1) Fig. 10. Floor level versus modulation index for binary FSK single- double-filter receivers. for 5 10 10 as it is not with double-filtering. The substantial improvement in the error floor provided by a double-filter detector is further stressed by Figs. 8 9. These figures show the floor level versus normalized fading bwidth for single- double-filter receivers. The continuous lines are obtained by setting to zero the noise level in the analytical formulas of the error probability. Finally, the dependence of the floor level on the modulation index is illustrated in Fig. 10 for 5 10 from which the desired result follows by interchanging the expectation differentiation operations. It is worth noting that: Equation (A.1) can also be derived from the correlation properties of the -power series models illustrated by Bello in [14, p. 390]; vanishes for odd because is an even function all its odd derivatives are zero at the origin. Also note that, with an autocorrelation function as in (5.1), (2.9) yields. APPENDIX B In this Appendix, we compute the elements of the autocorrelation matrices. With straightforward algebra, the elements of are found to be (recall that, as is Hermitian, its elements are related by ) (B.1) (B.2) (B.3)

1424 IEEE TRANSACTIONS ON COMMUNICATIONS, VOL. 45, NO. 11, NOVEMBER 1997 Correspondingly, the eigenvalues of result in (B.4) (B.5) the rom variables, have the same joint probability density function. Then, as are uncorrelated, the joint probability density function of may be written as (C.1) Next, consider the following integral which appears in (4.2): (B.6) Note that have always opposite sign (see [8, p. 515]). In a similar way, the elements of the autocorrelation matrix are found to be (again, recall that ) (C.2) are defined in Sections III-A IV- A, respectively. Substituting (C.1) (C.2) into (4.2) performing some manipulations yields (B.7) (B.8) (B.9) (B.10) (B.11) (B.12) (B.13) (B.14) (B.15) (C.3) is a constant independent of. Finally, taking the logarithm of the product in (4.1) using (C.3) yields (4.4). It is interesting to note that the simple structure of (C.3) is a consequence of both the orthogonality of the rom variables the orthogonality of the functions. It can be shown that the detection strategy of the ML noncoherent detector cannot be expressed in closed form if higher order terms are included in the truncated Taylor series of the fading distortion. APPENDIX D In this Appendix, we provide a useful formula for the computation of the residues of the function (D.1) (D.2) (B.16) Application of the residue theorem yields, after considerable manipulations, the following expression for the residue of for : (B.17) (B.18) APPENDIX C In this Appendix, we give a sketch of the derivation of the optimal noncoherent receiver for FSK signals transmitted on a linearly time-selective fading channel. To begin, we note that (D.3)

VITETTA et al.: OPTIMAL NONCOHERENT DETECTION OF FSK SIGNALS 1425 with if if (D.4) (D.5) (D.6) The solution of this problem allows a numerical evaluation of the bit-error performance of both single-filter double-filter detectors for binary FSK in the presence of diversity branches with the same noise fading statistics. In these circumstances, represents the set of the eigevalues of the matrix in (3.11) ( for ) for the single-filter detector of the matrix in (4.18) ( for ) for the double-filter detector. The error probability is found as follows. Suppose that is the characteristic function of some rom variable. Then, the probability that be negative is given by [16] (D.7) [10] J. B. Anderson, T. Aulin, C. E. Sundberg, Digital Phase Modulation. New York: Plenum, 1986. [11] I. Korn, Error probability of M-ary FSK with differential phase detection in satellite mobile channel, IEEE Trans. Veh. Technol., vol. 38, pp. 76 85, May 1989. [12], M-ary frequency shift keying with limiter-discriminatorintegrator detector in satellite mobile channel with narrow-b receiver filter, IEEE Trans. Commun., vol. 38, pp. 1771 1778, Oct. 1990. [13], Error floors in the satellite l mobile channels, IEEE Trans. Commun., vol. 39, pp. 833 837, June 1991. [14] P. A. Bello, Characterization of romly time-variant linear channels, IEEE Trans. Commun. Syst., vol. CS-9, pp. 360 393, Dec. 1963. [15] A. Papoulis, Probability, Rom Variables Stochastic Processes, 2nd ed. New York: McGraw-Hill, 1984. [16] M. J. Barrett, Error probability for optimal suboptimal quadratic receivers in rapid rayleigh fading channels, IEEE J. Select. Areas Commun., vol. SAC-5, pp. 302 304, Feb. 1987. [17] G. L. Turin, The characteristic function of Hermitian quadratic forms in complex normal variables, Biometrika, vol. 47, pp. 199 201, June 1960. [18] C. E. Pearson, Hbook of Applied Mathematics, 2nd ed. New York: Van Nostr Reinhold, 1990. Giorgio M. Vitetta (S 89 M 91) was born in Reggio Calabria, Italy, in April 1966. He received the Dr.Ing. degree in electronic engineering (cum laude) the Ph.D. degree from the University of Pisa, Pisa, Italy, in 1990 1994, respectively. In 1992 1993, he spent a period at the University of Canterbury, Christchurch, New Zeal, doing research for digital communications on fading channels. He is currently a Research Fellow at the Department of Information Engineering of the University of Pisa. His interests include coded modulation, channel equalization, algorithms for synchronization. REFERENCES [1] G. L. Turin, Error probabilities for binary symmetric ideal reception through nonselective slow fading noise, Proc. IRE, vol. 46, pp. 1603 1619, Sept. 1958. [2] J. N. Pierce, Theoretical diversity improvement in frequency-shift keying, Proc. IRE, vol. 46, pp. 903 910, May 1958. [3] J. N. Pierce S. Stein, Multiple diversity with nonindependent fading, Proc. IRE, vol. 48, pp. 89 104, Jan. 1960. [4] G. L. Turin, On optimal diversity reception, II, IRE Trans. Commun. Syst., vol. CS-10, pp. 22 31, Mar. 1962. [5] P. M. Hahn, Theoretical diversity improvement in multiple frequency shift keying, IRE Trans. Commun. Syst., vol. CS-10, pp. 177 184, June 1962. [6] W. D. Lindsey, Error probability for rician fading multipath reception of binary N-ary signals, IEEE Trans. Inform. Theory, vol. IT-10, pp. 339 350, Oct. 1964. [7], Error probability for incoherent diversity reception, IEEE Trans. Inform. Theory, vol. IT-11, pp. 491 499, Oct. 1965. [8] M. Schwartz, W. R. Bennett, S. Stein, Communication Systems Techniques. New York: McGraw-Hill, 1966. [9] P. A. Bello B. D. Nelin, The influence of fading spectrum on the binary error probabilities of incoherent differentially coherent matched filter receivers, IRE Trans. Commun. Systems, vol. 10, pp. 160 168, June 1962. Umberto Mengali (M 69 SM 85 F 90) received the degree in electrical engineering from the University of Pisa, Pisa, Italy. In 1963, he obtained the Libera Docenza in telecommunications from the Italian Education Ministry. Since 1963, he has been with the Department of Information Engineering at the University of Pisa, he is a Professor of Telecommunications. His research interests include digital communication theory, with emphasis on synchronization methods modulation techniques. He has served for six years as Editor of the IEEE TRANSACTIONS ON COMMUNICATIONS. He is now an Editor of the European Transactions on Telecommunications. He has been a consultant to industry in the area of communications. Prof. Mengali is a member of the Communication Theory Committee. He is listed in American Men Women in Science. Desmond P. Taylor (M 65 SM 90 F 94), for photograph biography, see p. 102 of the January 1997 issue of this TRANSACTIONS.