ABSTRACT NONLINEAR EQUALIZATION BASED ON DECISION FEEDBACK EQUALIZER FOR OPTICAL COMMUNICATION SYSTEM. by Xiaoqi Han

ABSTRACT NONLINEAR EQUALIZATION BASED ON DECISION FEEDBACK EQUALIZER FOR OPTICAL COMMUNICATION SYSTEM by Xiaoqi Han Nonlinear impairments in optical communication systems have become the major performance limiting factor of optical communication systems. The equalizer based on linear filters has limited compensation capability. Therefore, nonlinear models based equalizers are proposed for performance enhancement. In this thesis, a popular nonlinear processing tool, the Volterra model, is investigated. This research proposes an equalization scheme which integrates a nonlinear filter and a Decision Feedback Equalizer (DFE) to alleviate nonlinear distortions. The error detection schemes are applied to reduce the error propagation problem of DFE. Different equalization configurations consisting of nonlinear/linear filter and DFE are investigated. The results presented in this thesis can be used to design equalizers for the next generation s optical communication system.

NONLINEAR EQUALIZATION BASED ON DECISION FEEDBACK EQUALIZER FOR OPTICAL COMMUNICATION SYSTEM A Thesis Submitted to the Faculty of Miami University in partial fulfillment of the requirements of the degree of Master of Science Department of Electrical & Computer Engineering by Xiaoqi Han Miami University Oxford, Ohio 2013 Advisor Dr. Chi-Hao Cheng Reader Dr. Donald Ucci Reader Dr. Kuang-Yi Wu

Table of Contents Chapter 1... 1 Introduction... 1 Chapter 2... 4 Optical Communication System Review... 4 2.1 Optical Communication System... 4 2.2 Optical Fibers... 5 2.2.1 Clarification of Optical Fibers... 5 2.2.2 Characteristics of Optical Fibers... 6 2.3 Optical Receivers... 8 Chapter 3... 10 Signal Processing Tool Review... 10 3.1 Equalizer Model... 10 3.1.1 Linear Model... 10 3.1.2 Volterra Model... 10 3.2 Nonlinear Equalizers with DFE... 11 3.3 Error Detection... 14 3.3.1 Perfect Decision Error Detection... 14 3.3.2 Parity Check... 14 3.4 Update Algorithms... 15 3.4.1 Recursive Least Square Algorithm... 15 3.4.2 Least Mean Square Algorithm... 16 Chapter 4... 18 Simulation in MATLAB Using Benedetto Model... 18 4.1 System Setup in MATLAB... 18 4.2 Signal Generator... 18 4.3 Channel Model and Noise... 20 4.4 Equalization and Performance Analysis... 21 Chapter 5... 26 Simulations in OptiSystem... 26 II

5.1 Introduction of OptiSystem... 26 5.2 Incoherent System Performance in OptiSystem... 26 5.2.1 System Setup for On-Off Keying System... 26 5.2.2 Simulation Result and Analysis... 29 5.3 Coherent System Performance in OptiSystem... 29 5.3.1 System Setup for Coherent 16 QAM... 32 5.2.2 User Define Signal Generator... 33 5.2.3 Simulation Results and Analysis in MATLAB for 3dBm Laser Power... 35 5.2.4 Analysis of Relationship between Laser Power and System Performance... 40 Chapter 6... 44 Conclusions... 44 References... 46 III

Tables Table 4. 1 Average BER for six equalizers without error detection... 22 Table 4. 2 Mean values of BER for six equalizers with error detection... 25 Table 5. 1 OptiSystem simulation parameters for OOK optical system... 28 Table 5. 2 Simulation parameters of Optiwave DFE module... 28 Table 5. 3 BER of OOK system from OptiSystem... 29 Table 5. 4 Four DFE Schemes Tested in OptiSystem... 29 Table 5. 5 Simulation setting of OptiSystem coherent 16-QAM system... 33 Table 5. 6 Relation between parity bit and transmitted bits... 34 Table 5. 7 Mean BERs for four schemes under different error detection capability... 37 Table 5. 8 The number of errors missed by parity check... 37 Table 5. 9 Input power per channel under different laser powers... 40 IV

Figures Figure 2. 1 General scheme of optical communication system... 4 Figure 2. 2 Light transmission in SMF... 5 Figure 2. 3 Light transmission in MMF... 6 Figure 2. 4 Constellation diagram of 16 QAM... 9 Figure 3. 1 DFE scheme... 12 Figure 3. 2 Equalization with training sequence... 13 Figure 3. 3 Equalization with the decision device known as the slicer... 13 Figure 3. 4 Equalization combined with forward and feedback path... 14 Figure 4. 1 Basic scheme of simulation in MATLAB... 18 Figure 4. 2 Constellation diagram of 16QAM... 19 Figure 4. 3 Constellation diagram for 16QAM generated by MATLAB... 19 Figure 4. 4 16-QAM constellation diagram after nonlinear channel and noise... 21 Figure 4. 5 Equalizer output constellation diagram for six types of equalizers... 23 Figure 4. 6 Equalizer output constellation diagram for six types of equalizers with error detection... 24 Figure 5. 1 Incoherent OOK system diagram... 27 Figure 5. 2 Screen shot of OptiSystem OOK system simulation diagram... 27 Figure 5. 3 FF DFE scheme... 30 Figure 5. 4 FV DFE scheme... 30 Figure 5. 5 VF DFE scheme... 31 Figure 5. 6 VV DFE scheme... 31 Figure 5. 7 Simulation diagram of 16-QAM system with coherent detection... 32 Figure 5. 8 OptiSystem Simulation diagram screen shot for coherent system... 33 Figure 5. 9 OptiSystem constellation diagram for 16-QAM with even parity check... 34 Figure 5. 10 Distorted signal constellation diagram from OptiSystem... 35 Figure 5. 11 MATLAB Constellation diagram of transmitted signals under 3dBm laser power... 36 Figure 5. 12 Distorted signal constellation diagram under 3dBm laser power from MATLAB... 36 Figure 5. 13 Signal constellations after equalization with no error detection under FF and VF systems... 38 Figure 5. 14 Signal constellations after equalization with perfect error detection under FF and VF systems... 39 Figure 5. 15 Signal constellations after equalization with parity check under FF and VF systems... 39 Figure 5. 16 BER vs. power: Uncompensated systems and systems with DFE schemes without error detection... 41 Figure 5. 17 BER vs. power: Uncompensated systems and system with DFE schemes using perfect error detection... 41 V

Figure 5. 18 BER vs. power: Uncompensated systems and system with DFE schemes using parity check error detection... 42 Figure 5. 19 BER vs. power: System with DFE schemes using parity check and perfect error detection. 42 VI

Acknowledgements I would like to give my sincere gratitude to my supervisor: Dr. Chi-Hao Cheng. His patient and invaluable guidance help me a lot in pursuing my research objective. When I got stuck, he would give his maximum support and inspire me with new ideas. The most important thing I learn from him is that attitude is the most critical in research and life. The well-disciplined attitude obtained from him will be beneficial to my life. I would like to thank Drs. Kuang-Yi Wu and Donald Ucci for serving on my committee. They provide me with insightful comments and suggestions for my research. With their keen help, I get further improvement on my research and thesis writing. Thank you for spending your valuable time giving so much support. I sincerely thank all the ECE faculty, staff, and graduate students for helping me through my graduate study at Miami University. Special appreciation is given to Mr. Hang Yin. Without your encouragement and understanding, I cannot go through all the struggles in hard time. In the end, I would like to thank my parents for paying your love to me disregarding repay. Without your endless support and love, I cannot be where I am. Thank you for your trust in me. Thank you for always being here. VII

Chapter 1 Introduction Fiber optic communications utilize an optical fiber as the medium and pulses of light as the carrier to transmit information [1], enabling high speed communication. Nonlinearities in optical fibers deteriorate system performances and become a major performance limiting factor [2]. In this project, compensation for nonlinear distortions in optical communication systems is investigated. Linear equalizer based Decision Feedback Equalizer (DFE) is widely utilized to equalize nonlinear distortions [3]. In this project, we explore a nonlinear filter based DFE for nonlinear distortions compensation. Fiber optics communication was first developed in 1970s which brought great breakthroughs for telecommunications industry [1]. A fiber optics communication system consists of a transmitter which is used for encoding information into optical signals for transmission and a receiver whose function is to decode messages from received signals. Based on the modulation scheme, fiber optics communication can be categorized as an incoherent fiber optics system or a coherent fiber optics system depending on whether the receiver recovers the transmitted signal phase or not [4]. Transmission impairments cannot be avoided in communication systems. Channel impairments in a fiber optics communication system can be classified into two groups: linear impairments and nonlinear impairments. Chromatic dispersion and fiber attenuation belong to the group of linear impairments [1, 5]. Over the years, engineers have made significant progress in developing techniques for compensating linear impairments. The development of optical communication systems, with higher transmission speed, and longer transmission length demand bring researchers attention to nonlinear impairments including self-phase modulation, cross-phase modulation, etc. [5]. This project focuses on the development of nonlinear equalizers to minimize channel distortions for high speed optical communication systems. The equalizers are used to reduce signal distortions in transmission systems [6-7]. The equalizers based on linear filter structure such as Finite Impulse Response (FIR) filter is 1

considered as a linear equalizer. Linear equalizers are easy to implement and thus cost-effective. However, its capability of compensation of nonlinear distortion is limited. Therefore, nonlinear filter based equalizer deserves more investigations. The Volterra model is a popular nonlinear signal processing tool [8-9]. Although the complexity of the Volterra model prevents its wide deployment, it has shown a great potential to compensate nonlinear distortions and decrease Bit Error Rate (BER) in optical communication systems [8, 10-12], and the rapid progress in electrical might make a Volterra model based equalizer more feasible in the near future. In this research project, a DFE based on Volterra model is proposed and studied. Periodically transmitting training sequences to update equalizer coefficients can efficiently decrease system BER. However, it decreases bandwidth utilization. DFE, which updates equalizer coefficients based on the decoded symbols, does not request periodic transmission of training sequence, thus improving bandwidth utilization. Although the conventional DFE is developed based on linear filters, the DFE is a nonlinear device due to slicer s nonlinear nature. However, as shown in our research results, a slicer alone does not fully compensate for fiber nonlinear distortions. In this research project, we propose to use a DFE consisting of a nonlinear filter and a slicer for nonlinear distortions compensation. Different DFE configuration consisting of nonlinear/linear filters and slicer will be considered and tested in a coherent Single Mode Fiber (SMF) communication system. Decision errors due to noise and system distortions can significantly degrade a DFE performance and this phenomenon is referred to as the error propagation [13]. To alleviate this issue, error detection scheme can be integrated into DFE. Both perfect error detection and parity check are studied in this project. Our study shows that the error detection functionality can greatly improve DFE s performance. The rest of the thesis is organized as follows. Chapter 2 introduces optical communication system, including incoherent system and coherent system in Single-Mode and Multimode fibers. Detailed explanation of signal processing tools including equalizer schemes, adaptive algorithms, and error detection is presented in Chapter 3. Chapter 4 presents simulation results in MATLAB using Benedetto model as a general communication system. Chapter 5 focuses on performances 2

of equalizers in an optical communication system simulated with OptiSystem, a commercial optical communication simulation tool. Chapter 6 concludes this thesis. 3

Chapter 2 Optical Communication System Review 2.1 Optical Communication System Optical communication is a technique utilizing light wave as information carrier [5]. Its early form can be traced back to smoke signals, beacon fires, and semaphore lines etc. Microwave and coaxial cables functioned as the carrier for long distance communication before fiber optics. However, their transmission speed is limited to ~100 Mb/s which is not enough for supporting high speed communications. Optical communication system opens a door to operations on more than 10 Tb/s bit rate [1]. Figure 2.1 shows the general scheme of optical communication system which consists of an optical transmitter, a communication channel, and an optical receiver [1]. Optical transmitter converts the electrical input into the appropriate optical form for transmission. Light-emitting diodes (LEDs) or laser diodes performs as optical source which supplies optical carrier. Optical fibers including Single-Mode Fiber (SMF) and Multi-Mode Fiber (MMF) act as the communication medium. The loss of optical fibers is the main limitation for transmission distance. Besides, chromatic dispersion and fiber nonlinearity have significant effects on system performance and maximum transmission distance in the optical communication system. Optical receiver converts the received optical signals back to original electrical form. Main component of optical receiver is photo-detector which converts optical signals to electrical signals. There are two types of demodulation schemes: incoherent system and coherent system which will be described in later sections. Figure 2. 1 General scheme of optical communication system 4

2.2 Optical Fibers Optical fibers act as communication medium in optical communication systems. It is generally composed of a core and a cladding layer. Utilizing characteristic that the index of refraction of the core is higher than that of the surrounding cladding layer, light is transmitted within the core due to total internal reflection [1]. In the transmission, optical fiber introduces distortions. This section will give a detailed explanation in the clarification of optical fibers: single mode fibers and multimode fibers and characteristics for optical fibers. 2.2.1 Clarification of Optical Fibers Optical fibers broad bandwidth and lower attenuation compared to coaxial cables make it advantageous in long distance communications. According to the different modes of lights transmitted, optical fibers can be classified into two categories: Single-Mode Fibers (SMF) and Multi-Mode Fibers (MMF). SMF has a smaller core that allows only one mode of light transmitted through the core. Because of this characteristic, signal fidelity is better preserved since modal dispersion is greatly reduced in SMF [14]. It is capable of higher bandwidth capacity. SMFs are mostly used for long haul transmission and higher bandwidth applications. Generally diameter of single-mode fiber s core is limited in 8-10 micrometers [14]. In the research project, we only consider an optical communication system with SMF. A SMF diagram is illustrated in Figure 2.2. Figure 2. 2 Light transmission in SMF Compared to SMFs, MMF has a much larger core usually greater than 10 micrometers. It can carry more than one mode of light at the same time. The light transmission in a MMF is illustrated in Fig.2.3. Simplified connection based on larger core size provides more allowances 5

for low-cost optics-electronic devices [14]. The property of transmitting numerous models of lights at one time causes modal dispersion, which greatly restricts transmission distance. However, since MMF allows the transmission of the signal in different modes, it has the potential to carry significantly more information in one fiber as compared to SMF. As a result, many researchers believe MMF system will be the future of long haul optical communication system [14]. Figure 2. 3 Light transmission in MMF 2.2.2 Characteristics of Optical Fibers Although optical fibers have many advantages over other communication tool, it still has some limiting factors: Fiber attenuation, fiber dispersion, and nonlinearity, to name a few. In this section detailed information about the undesirable characteristics of optical fibers will be discussed. Fiber attenuation is the loss of optical power brought by optical fibers in transmission process. It depends on the material of optical fibers, signal wavelength, and travelling distance [15]. A longer transmission distance results in higher attenuation. Generally speaking, fiber attenuation is in inverse relationship with wavelength. SMF operate on longer wavelength region [14]. Thus fiber attenuation for SMF is much less than MMF. This is also the reason why SMF is more suitable for long distance transmission while MMF is designed for shorter distance. Fiber chromatic dispersion is the most important factor in linear distortions for optical fibers. Since optical signals occupy a broad bandwidth, frequency-dependent phase delay in optical fibers results in signal distortion [1, 5]. Most common phenomenon caused by chromatic dispersion is pulse broadening which cause interferences between adjacent symbols thus decreasing the communication signal quality. 6

High optical intensity in optical fibers introduces nonlinear effects especially in Wavelength- Division Multiplexing (WDM) systems. In a WDM system, one fiber transmits many channels occupying different wavelengths. Accumulation of nonlinearity over long transmission distance cannot be neglected in a high-speed optical communication system. With significant progress engineers made to compensate chromatic dispersion and fiber attenuation, fiber nonlinearity becomes one of the most important remaining limiting factors for optical communication systems. Based on different physical characteristics, fiber nonlinearities are divided into two categories: the Kerr effect and stimulated scattering [5]. The Kerr effect is generated from a change of refractive index of materials [16]. It can be expressed by the following equations. n n n * I (0.1) 0 2 where n is the refractive index, denotes the linear refractive index, denotes the nonlinear refractive index, and I is the wave intensity. The Kerr effect includes Self-Phase Modulation (SPM), Cross-Phase Modulation (XPM), and Four-Wave Mixing (FWM) [16]. Self-phase modulation is a nonlinear effect caused by interaction with the wave itself. Simply speaking, the change of refractive index results in nonlinear phase shift which makes the signal undistinguishable to optical receivers. Compared to the self-phase modulation for singlewavelength signal, cross-phase modulation comes from the modulation between signals among adjacent channels. The cross-phase modulation in WDM system induces interchannel crosstalk. Except for a nonlinear phase shift brought about by a variation of refractive index, the fiber nonlinearity also generates a signal based on the interaction between signals in different wavelength in optical fibers. It is called four-wave mixing phenomenon. Stimulated scattering is the result of inelastic scattering of the photon. It is composed of two parts: stimulated Raman scattering and stimulated Brillouin scattering [17]. When light travels through optical fibers, the photons will interact with themselves and give rise to scattering such as Stimulated Raman scattering. It will cause energy distribution in random directions. However, besides interaction between photons themselves, they will also have interaction with fiber s silica molecules which generates scattering like Stimulated Brillouin scattering in 7

direction opposite propagation direction. When bit rates are high, effects by stimulated scattering cannot be ignored. 2.3 Optical Receivers Function of optical receivers is to restore electrical signals from received optical signals. There are two types of demodulation schemes used in optical communication systems which are distinguished by their needs for preserving phase of the carrier at receiver end. Demodulation schemes not requiring the carrier phase are referred to as incoherent communication system [4]. An incoherent optical system uses intensity to transmit information and intensity modulated signals are detected via a photo-detector. Major advantage of an incoherent communication system is its simplicity. It dominated the optical communication system for several decades. However, the fact that it encodes information without using carrier phase significantly reduces its bandwidth utilization efficiency. To achieve high transmission rate and bandwidth efficiency, it is necessary to deploy a coherent communication system. In this research project, an On-Off Keying (OOK) system is employed for incoherent detection. It only has two amplitudes: 0 and 1. Its immunity to noises is high but it can only act on 1 bit which is not suitable for high bit rate in advanced optical communication systems. Demodulation schemes requiring accurate recovered carrier phase are referred to as coherent communication system [4]. Coherent detection allows user to encode information in both carrier amplitude and phase, thus improving bandwidth efficiency. Recent progress in optical phase lock loops paves the road for the development of a coherent optical communication system [18]. Using the same transmitted power, a coherent system enjoys a higher Signal-to-Noise Ratio (SNR) compared to the incoherent system. However its sensitivity to laser phase noise and fiber dispersions greatly limits its improvement. Moreover, it has a far more complicated structure, and is more expensive [19]. In this research, we will study a coherent system using 16- Quadrature Amplitude Modulation (16-QAM). In a QAM system, signal modules two carrier waves (in-phase, I, and quadrature, Q) whose phases are off by 90 degree and the coherent detector needs to separate these two carriers to decode transmitted signal. Figure 2.5 shows the constellation diagram for 16-QAM. Coherent detection is demandable for various amplitudes and 8

phases for different signals in 16-QAM. Signal information encoded in both amplitude and phase help improve bandwidth efficiency. Figure 2. 4 Constellation diagram of 16 QAM 9

Chapter 3 Signal Processing Tool Review 3.1 Equalizer Model This section provides detailed information of the two basic equalizer models: linear model and Volterra model. Different forward/feedback equalizer combinations, DFE, and update algorithms employed for the research project are also discussed. 3.1.1 Linear Model Linear model equalizer used in this project has the form same as the FIR filter. It employs the difference between a desired signal, and the equalizer output, to update coefficients for the equalizer. The relation between the linear equalizer output and input is given by the convolution summation as follows: out N n (0.2) k 0 e n w k x n k where is the equalizer output signal, denotes the input signal, and is the weight for the FIR filter at time instant n. The error signal used for the update is the difference between the desired signal and the equalizer output, where e n r n e n (0.3) is the desired signal. If a training sequence is used, we can get an accurate error signal to optimize the equalizer coefficients. 3.1.2 Volterra Model An N-th order discrete Volterra filter with input, output and memory length M can be described as [9, 20] out 10

N M M (0.4) e n h h k,, k. x n k x( n k ) out 0 r 1 r 1 r r 1 k1 0 kr 0 where are the order Volterra kernels. In this research project, a third order band limited Volterra Model is utilized. The 3 rd order band limited Volterra model [21] with input, output and memory length M can be represented as [22, 23] M M M M e ( ),, n k out (0.5) * n h k x n k h k k k x n k x n k x 1 1 1 3 1 2 3 1 2 3 k1 0 k1 0k2 0k3 0 where and are the linear and cubic Volterra kernels respectively. denotes the complex conjugation of. Like the linear equalizer, the difference between desired signal and equalizer output is used to update equalizer coefficients. 3.2 Nonlinear Equalizers with DFE A DFE consisting of a linear filter and a slicer is the most common equalizer for removing nonlinear effects from optical fibers. Its basic principle is to correct errors of future symbols based on the previous errors. The slicer determines the most likely transmitted symbol based on the received symbol and the slicer output serves as the desired signal. Since a slicer is a nonlinear device, a DFE can eliminate nonlinear impairments in optical fibers to some degree. However, as shown in our research work, the linear filter based DFE can only achieve limited success in terms of compensating for a nonlinear distortion. The use of the slicer reduces the need to periodically transmit training sequences. However the DFE is sensitive to the initial condition of its coefficients and can easily to be ruined by a wrong decision made by slicer. This phenomenon is referred to error propagation [13]. The basic scheme for a DFE is shown in Figure 3.1. 11

Figure 3. 1 DFE scheme In this research, we expand the conventional linear filter based DFE to a Volterra model based DFE and a DFE consisting of both linear and Volterra models. We also investigate the method to alleviate the impact of slicer s decision error. General diagrams for equalizer models using training sequence and a nonlinear equalizer with DFE are shown below in Figures 3.2 and 3.3 respectively. 12

Figure 3. 2 Equalization with training sequence Figure 3. 3 Equalization with the decision device known as the slicer Further research is conducted about filters used in equalizer s forward path and feedback path. Four combinations of basic models are included: linear model in both forward path and feedback path, linear model in forward path and third-order Volterra model in feedback path, third-order Volterra model in forward path and linear model in feedback path, and third-order Volterra model in both forward path and feedback path. Figure 3.4 shows the general schemes of introductions above. From the scheme of equalization with decision device, it is clear to see the importance of decision device in the whole system especially when there is feedback path. An error may propagate with the signal transmission. Thus investigations on the alleviating the impact of slicer s wrong decision is also conducted in this project. 13

Figure 3. 4 Equalization combined with forward and feedback path 3.3 Error Detection From the discussions above, making decisions based on the current received symbol correctly is of great significance. The slicer output will affect the direction of updating coefficients which has impact on the next received symbol. For an equalizer with a feedback path using slicer output, severe error propagation may ruin the performance of whole communication system. Error detection in decision device is therefore critical. Here two ways of error detection used in our simulation: perfect decision error detection and parity check. 3.3.1 Perfect Decision Error Detection An original sequence is transmitted along the equalization process for error detection check. We compare the decision symbol with actual transmitted symbol to determine if a decision error happens. Although the perfect decision error detection is not feasible, we use it in our study as a benchmark. 3.3.2 Parity Check A parity bit is an adjunct bit added to the end of a binary sequence to keep the amount with value 1 is odd or even. Parity check is to utilize concatenating parity bit with transmitted sequence [24]. Based on the oddness and evenness of the specific value it can be classified into two categories: odd parity check and even parity check. In this research project, we employ even 14

parity check. Bit with value one will be added if the count of value one is odd. Otherwise bit with value zero will be added at the end of transmitted sequence. On the receiver side, if count of value 1 is not even, then it is concluded that an error happens in decision process. Although a parity bit reduces bandwidth utilization. The parity check provides a simple scheme to detect decision errors. 3.4 Update Algorithms Adaptive algorithms are used to update equalizer coefficients based on error signals. Two most popular adaptive signal processing algorithms are the Recursive Least Square( RLS ) and the Least Mean Square ( LMS ) methods. 3.4.1 Recursive Least Square Algorithm The general idea for RLS is to update coefficients recursively to minimize the sum of the squared errors. This is also known as the cost function. Assume e(n) represents the error signal at time instant n. The cost function can be denoted as follows: C n i 2 n w n e ( i). (0.6) i 0 where is the forgetting factor which reduces the weight of previous errors. Setting the derivative of the cost function with respect to the equalizer coefficients to zero, can guarantee minimization of the cost function. w n k C 2 e i x i k 2 d i w m x i m x i k 0. n n p n i n wn i 0 i 0 m 0 n i (0.7) After differentiation based on a matrix inversion formula [22, 25], we can derive the RLS algorithm below: T n d n n ( n). x w (0.8) T n n n n n n * * 1 g P 1 x { x P( 1) x }. (0.9) 15

1 T 1 n n n n n P P 1 g( ) x P 1. (0.10) n n n n w w 1 g. (0.11) where,, p is the filter order. The initial value for is where is the identity matrix. The sequence is the desired signal, which can be either the training sequence or the slicer outputs, depending upon the equalization scheme. 3.4.2 Least Mean Square Algorithm The popular other adaptive update algorithm is the Least Mean Square (LMS) Algorithm. The premise of LMS algorithms is that it estimates the gradient vector in the steepest descent algorithm and uses them to update the weights. The goal of LMS algorithm is to minimize the mean square error. The cost function is defined as the expected value of square of error signal as shown in (3.11). where T T T 2 X W W [ X( ) X ] W. (0.12) 2 2 C E e n E d n E d n n E n n and When the value of the first derivative of C is 0, C will reach its smallest value. This will find the optimal values for weight coefficients and provide for the best performance of the adaptive filter. p k 0 n C en d n w k x i k 2e n 2e n 2 e n X ( n). (0.13) w ( k) w k w k n n n In summary, the LMS algorithm can be expressed as below: wn 1 k wn k 2 e n x n k. (0.14) where denotes the weight coefficient at time instant. The number is the step size which determines the speed of convergence. The sequence is the difference between desired signal and equalizer output signal at time instant. In the DFE based models, the difference between the output signal of the slicer and equalizer output signals at time instant denoted as. is 16

The largest advantage of LMS is its low computation and cost. However, it has only one adjustable variable, step size, which is employed to control the convergence rate, and the convergence rate is slower compared to RLS. RLS is more complicated in computation in LMS. But it can achieve better convergence speed. In this project, we deploy RLS as the adaptive algorithm to achieve better performance. 17

Chapter 4 Simulation in MATLAB Using Benedetto Model 4.1 System Setup in MATLAB In order to test the performance of the equalizers introduced above, we employ 16-QAM signals transmitted through a nonlinear channel. The distorted channel output corrupted by Gaussian noises will be used as the equalizer input. The equalizer output should be identical to the original signal in an ideal case. The basic scheme of simulation system is shown in Figure 4.1. Figure 4. 1 Basic scheme of simulation in MATLAB 4.2 Signal Generator The signal generator generates 16-QAM signals. QAM is a digital transmission technique that modulates amplitudes on two orthogonal carrier waves. The two carrier waves are sinusoid with the same frequency but with a 90 phase difference. A constellation diagram, shown in Figure 4.2, represents the QAM signals. Every point in the constellation diagram corresponds to 4 bits of binary data. Figure 4.3 shows the constellation diagram generated by MATLAB. 18

Figure 4. 2 Constellation diagram of 16QAM Figure 4. 3 Constellation diagram for 16QAM generated by MATLAB 19

4.3 Channel Model and Noise For our research, the channel model we employ is Benedetto s Volterra series model of a nonlinear telecommunication channel [26] as a testing system. Benedetto s model is a satellite channel which has loose BER requirements. It provides a bandpass nonlinear model based on the Volterra model. The Benedetto s model is given in (4.1)-(4.4). 3 1 (0.15) k 0 l ( i) f ( k) x( i k). out out 2 * 2 * 2 * 3 1 3 1 3 2 3 3 3 3 2 2 * 2 * 4 3 5 2 3. c i f x i x i f x i x i f x i x i f x i x i f x i x i 3 3 (0.16) out f i f x i x i (0.17) 1 3 3 [ * 2 ] 2. 5 ( ). outc i l i c i f i (0.18) out out out is the input signal complex envelope. denotes the complex conjugation of. The channel model coefficients are Gaussian noise is added to the nonlinear channel output signal, and the SNR is set to around 23.67dB. The distorted signal after passing through the nonlinear channel and corrupted by Gaussian noise is shown in Figure 4.4. From the graph, we find that the constellation diagram is rotated and there are clouds caused by noise around the signal points. 20

Figure 4. 4 16-QAM constellation diagram after nonlinear channel and noise In reality, the channel characteristics change over time. Here, we change the phase of each coefficient randomly within the range of (-15, 15 ) to simulate time-varying nature and use a DFE to compensate the time varying communication channel. 4.4 Equalization and Performance Analysis A DFE cannot be functional if the slicer cannot determine the transmitted signal with reasonable accuracy. Therefore, the performance of the DFE is sensitive to the equalizer s initial settings. Because of the sensitivity of the DFE to initial condition of its coefficients, a training sequence is transmitted in the initialization phase to determine the equalizer coefficients. It is worthy of notice that, with a DFE, the need to periodically send training sequence is reduced. The performance of a DFE highly depends on the accuracy of the slicer output. A wrong decision may destroy the DFE performance, especially the 3 rd order Volterra model based DFE, which is more sensitive to decision error. Using Benedetto s channel model as the starting point, we changed channel model coefficients phases. Figure 4.5 (a) and (b) show the output signal constellations of the linear equalizer and the 3 rd order Volterra model based equalizer whose coefficients are updated with the difference between the actual input signal and the equalizer 21

output. This is not a realistic situation but is included in simulations as a benchmark. Figure 4.5 (c) and (d) show the output signal constellations of a linear equalizer and a 3 rd order Volterra model based equalizer, whose coefficients are fixed and the same as the initial settings. Figure 4.5 (e) and (f) show the output signal constellations of a linear equalizer and a 3 rd order Volterra based equalizer whose coefficients are updated with the difference between the slicer output and the equalizer output. Unsurprisingly, the linear equalizer based DFE and 3 rd order Volterra model based DFE updated with actual input signals have the best performance. As shown in Figure 4.5 (e) and (f), the linear equalizer based DFE and the 3 rd order Volterra model based DFE actual dramatically deteriorate the system performance. As shown in our later simulation results, this deterioration is caused by the decision error made by the slicer. In order to analyze the performance of the six equalizers quantitatively, we conduct the simulation fifty times to calculate the average BER of the system with different DFEs and the results are included in Table 4.1. Each time, 2500 symbols are transmitted and the phases of the nonlinear channel coefficients are changed randomly within +/- 15 degrees. Table 4.1 clearly shows that, both the linear equalizer based and 3 rd order Volterra model based DFEs deteriorate system performance. The equalizers updated with the actual input sequence improve the system performance and the linear equalizer updated with the actual input signal outperforms the 3 rd order Volterra model based equalizer slightly due to the fact that the nonlinear equalizer is more sensitive to noise added at the receiver side. But it is not realistic as receiver does not have the actual transmitted signals. Table 4. 1 Average BER for six equalizers without error detection Type of equalizer Linear equalizer with fixed coefficients Linear equalizer updated with actual transmitted signal Linear equalizer updated with slicer output 3 rd order Volterra equalizer with fixed coefficients 3 rd order Volterra equalizer updated with actual transmitted signal Mean BER 3 rd order Volterra equalizer updated with slicer output 22

1.5 linear updating with training 2 3rd volterra updating with training 2 linear equalization without updating 1 1.5 1.5 0.5 1 1 0 0.5 0 0.5 0-0.5-0.5-0.5-1 -1-1 -1.5-1.5-1 -0.5 0 0.5 1 1.5 (a) Equalizer output constellation diagram for linear equalizer updated with real signal 1.5 3rd volterra without updating -1.5-1.5-1 -0.5 0 0.5 1 1.5 (b) Equalizer output constellation diagram for 3rd order Volterra model based equalizer updated with real signal 2 linear equalization update using slicer -1.5-2 -1 0 1 2 (c) Equalizer output constellation diagram for linear equalizer with fixed coefficients 3 3rd volterra equalization update using slicer 1 0.5 0-0.5-1 1 0-1 2 1 0-1 -1.5-1.5-1 -0.5 0 0.5 1 1.5 (d) Equalizer output constellation diagram for 3rd order Volterra model based equalizer with fixed coefficients -2-2 -1 0 1 2 (e) Equalizer output constellation diagram for linear equalizer updated with slicer output -2-2 -1 0 1 2 3 (f) Equalizer output constellation diagram for 3rd order Volterra model based equalizer updated with slicer output Figure 4. 5 Equalizer output constellation diagram for six types of equalizers As shown previously, the DFE does not deliver satisfactory performance and we speculate the decision error is the cause. To verify this assumption, we changed the setting of the DFE as follows. When the slicer makes a correct decision, the equalizer coefficients are updated. Otherwise, we restore the equalizers coefficients to the previous setting. In other words, the slicer has error detection capability and determines whether the equalizer coefficients should be updated or not. We simulate the program again with the same settings. The resulting signal constellations are shown in Figure 4.7. 23

1.5 linear updating with training 1.5 3rd volterra updating with training 2 linear equalization without updating 1 0.5 1 0.5 1 0 0 0-0.5-1 -0.5-1 -1-1.5-1.5-1 -0.5 0 0.5 1 1.5 (a) Equalizer output constellation diagram for linear equalizer updated with real signal 3rd volterra without updating 1.5 1 0.5 0-0.5-1 -1.5-1.5-1 -0.5 0 0.5 1 1.5 (d) Equalizer output constellation diagram for 3rd Volterra model based equalizer with fixed coefficients -1.5-1.5-1 -0.5 0 0.5 1 1.5 (b) Equalizer output constellation diagram for 3rd Volterra model based equalizer updated with real signal linear equalization update using slicer 1.5 1 0.5 0-0.5-1 -1.5-1.5-1 -0.5 0 0.5 1 1.5 (e) Equalizer output constellation diagram for linear equalizer updated with slicer output with error detection -2-2 -1 0 1 2 (c) Equalizer output constellation diagram for linear equalizer with fixed coefficients 3rd volterra equalization update using slicer 1.5 1 0.5 0-0.5-1 -1.5-1.5-1 -0.5 0 0.5 1 1.5 (f) Equalizer output constellation diagram for 3rd Volterra model based equalizer updated with slicer with error detection Figure 4. 6 Equalizer output constellation diagram for six types of equalizers with error detection Figure 4.7 (e) and (f) show the output signal constellations of the linear equalizer and the 3 rd order equalizer using slicer output to update coefficients. When the decision error is present the coefficients will not be updated. As shown in the figures, the equalizers have rotated the constellation diagram back to its normal position. 24

Table 4. 2 Mean values of BER for six equalizers with error detection Type of equalizer Linear equalizer with fixed coefficients Linear equalizer updated with actual transmitted signal Linear equalizer updated with slicer with error detection capability 3 rd order Volterra equalizer with fixed coefficients 3 rd order Volterra equalizer updated with actual transmitted signal 3 rd order Volterra equalizer updated with slicer with error detection Mean BER In order to analyze the performance of the six equalizers quantitatively, we re-conducted the simulation fifty times to calculate mean values of BER of system with different DFEs and the results are included in Table 4.2. Each time, 2500 symbols are transmitted and the phases of the nonlinear channel coefficients are changed randomly within +/- 15 degree. We can draw two conclusions from Table 4.2: The linear equalizer based DFE and the 3 rd Volterra model based DFE can effectively decrease BER compared to those with fixed initial settings. A 3 rd Volterra model based DFE outperforms a linear equalizer based DFE apparently when decision error can be detected. The 3 rd Volterra equalizer with fixed coefficients works better than linear equalizer with fixed coefficients. The 1 st order equalizer updated with actual input signal slightly outperforms the 3 rd order equalizer due to the fact that the nonlinear equalizer is more sensitive to noise and decision error. The decision accuracy of the slicer determines the performance of the DFE especially when the BER is high. Although it is not feasible to know transmitted signal in advance, some error detection schemes will be necessary to improve the performance of DFE. A more realistic error detection scheme, parity check, will be considered in the optical system simulation which will be presented next. 25

Chapter 5 Simulations in OptiSystem 5.1 Introduction of OptiSystem OptiSystem is a popular optical communication system simulation tool. It simulates characteristics of optical devices and allows researchers to simulate an optical communication system before conducting experiments which might be extremely expensive. In this project, we use OptiSystem to test and verify our equalization techniques. The research work conducted with OptiSystem is described in this Chapter. 5.2 Incoherent System Performance in OptiSystem For an incoherent On-Off Keying (OOK) communication system, we developed a conventional linear filter based DFE program in MATALB and compared it with the DFE module of OptiSystem. If both deliver similar performance, we can then have more confidence about our MATALB DFE program. Since OptiSystem does not provide DFE modules for coherent system nor with nonlinear filters, it is necessary to develop our own DFE design for coherent system in MATALB. 5.2.1 System Setup for On-Off Keying System We apply a DFE technique to an incoherent OOK system whose transmission length is 700km. The schematic diagram is shown in Figure 5.1. The attenuation of the Dispersion Compensation Fiber (DCF) and transmission fiber is compensated by amplifiers. The transmission fiber dispersion is partially compensated by the DCF. The transmission rate is 10 Gb/s. The screen shot of OptiSystem OOK system diagram is illustrated in Figure 5.2. The simulation settings of the coherent OOK communication system are summarized in Table 5.1 and the OptiSystem DFE module setting is listed in Table 5.2. 26

Figure 5. 1 Incoherent OOK system diagram Figure 5. 2 Screen shot of OptiSystem OOK system simulation diagram 27

Table 5. 1 OptiSystem simulation parameters for OOK optical system Bit Rate 10 Gb/s Wavelength of optical fiber 1552.5243 nm Dispersion Parameter for transmitting fiber 16.75 ps/nm/km Dispersion Parameter for DCF -78 ps/nm/km Transmitting Distance for one loop overall 700 km Electrical Gain 0.000144 Table 5. 2 Simulation parameters of Optiwave DFE module Training Sequence Length 1024 Step Size for LMS 5e6 Higher level input for optisystem 0.00014 Lower level input for optiystem 0 Number of forward coefficients 4 Number of feedback coefficients 2 Initial value for forward coefficients [1,0,0,0] Initial value for feedback coefficients [0,0] The settings of MATLAB DFE module are kept identical with those of OptiSystem DFE module. 28

5.2.2 Simulation Result and Analysis The BERs of the same incoherent OOK system with the OptiSystem and MATLAB DFE modules are listed in Table 5.3. Due to signal conversion between Optiwave and MATLAB, we cannot achieve identical BER using the OptiSystem and MATLAB DFE modules. However, two BER numbers listed in Table 5.3 are close enough to provide some confidence about the correctness of our MATLAB DFE program which serves as a foundation for the Volterra model based DFE. The BERs here are calculated from Q factors. Table 5. 3 BER of OOK system from OptiSystem Situation for calculation Before equalization DFE module offered by Optiwave DFE from MATLAB component BER 5.3 Coherent System Performance in OptiSystem We developed four DFE schemes and tested their performances in a coherent optical communication system in simulation using OptiSystem. These four schemes are shown below in Table 5.4. The structures of the four DFE schemes are shown in Figure 5.3~5.6. Table 5. 4 Four DFE Schemes Tested in OptiSystem Forward Path Feedback Path Referred Below Initial Coefficients FIR Linear Model FIR Linear Model FF [1,0,0,0] &[1,0,0,0] FIR Linear Model 3rd order Volterra Model FV [1,0,0,0] & zeros(1,44) 3rd order Volterra Model FIR Linear Model VF [1,zeros(1,43)]&[0,0,0,0] 3rd order Volterra Model 3rd order Volterra Model VV [1,zeros(1,43)]&zeros(1,44) 29

Figure 5. 3 FF DFE scheme Figure 5. 4 FV DFE scheme 30

Figure 5. 5 VF DFE scheme Figure 5. 6 VV DFE scheme 31

The modulation scheme in our simulation is 16-QAM. The transmitted signal is coded using parity check and DFE modules are tested under three conditions: without error detection, with perfect error detection, and with parity check error detection. Because of utilization of parity check for error detection, only eight out of sixteen 16-QAM symbols are used in transmission. The BER of coherent communication systems are calculated under different laser power. 5.3.1 System Setup for Coherent 16 QAM The major difference between a coherent system and an incoherent system is that a coherent system recovers the signal phase at the receiver side. Therefore, a coherent system receiver is more complicated than a photo-detector used in the incoherent system. The 16-QAM coherent system is shown in Figure 5.7. Corresponding OptiSystem diagram screen shot is shown in Figure 5.8. In order to concentrate on compensation of nonlinearity, DCF is used to fully compensate fiber chromatic dispersion. Other parameters are given in Table 5.5. Figure 5. 7 Simulation diagram of 16-QAM system with coherent detection 32

Figure 5. 8 OptiSystem Simulation diagram screen shot for coherent system Table 5. 5 Simulation setting of OptiSystem coherent 16-QAM system Bit Rate Sequence Length Wavelength of Optical Fiber Dispersion Parameter for Transmitting Fibers Dispersion Parameter for DCF Total Transmitting Distance 100Gb/s 65536 Bits 1550 nm 16.75 ps/nm/km -134 ps/nm/km 480 km 5.2.2 User Define Signal Generator We generated test sequences using even parity check. The relation between the parity bit and transmitted bits are given in Table 5.6. From Table 5.6, it is easy to identify that only eight symbols in the 16-QAM constellation diagram are used in transmission and they are shown in Figure 5.9. 33

Table 5. 6 Relation between parity bit and transmitted bits Original generated bits Parity bit Transmitted bits 000 0 0000 001 1 0011 010 1 0101 100 1 1001 110 0 1100 101 0 1010 011 0 0110 111 1 1111 Figure 5. 9 OptiSystem constellation diagram for 16-QAM with even parity check We sent the user generated sequence through OptiSystem coherent 16-QAM system and the uncompensated received signal constellation of the testing system is shown in Figure 5.10. From the graph, it is clear that the received signals are rotated and corrupted by noise. Our goal is to use equalization techniques to recover the original transmitted signals. With the original signals and distorted signals acquired from OptiSystem, simulations are conducted in MATLAB to test four DFE schemes. 34

Figure 5. 10 Distorted signal constellation diagram from OptiSystem 5.2.3 Simulation Results and Analysis in MATLAB for 3dBm Laser Power We collected 5 sets of input-output data under 3dBm laser power from OptiSystem, used the four different DFE schemes listed in Table 5.4 to compensated received signal, and calculated the mean BER for each of DFE. The 1 st set of data is utilized as training sequence to get coefficients of DFE whose coefficients are fixed. The input power for per channel is around 0.75dBm. Constellation diagrams of original signal and distorted signal under 3dBm laser power from MATLAB are shown in Figure 5.11 and Figure 5.12 respectively. Notice that received signals are attenuated and we need to divide the original signals by a certain value to make it compatible with received signal while determining the transmitted symbol. For 3dBm laser power, the value is 55. 35

3 Constellation Diagram of Original Signal for 3dBm Laser Power 2 1 0-1 -2-3 -3-2 -1 0 1 2 3 Figure 5. 11 MATLAB Constellation diagram of transmitted signals under 3dBm laser power 0.1 Constellation Diagram of Distorted Signal for 3dBm Laser Power 0.08 0.06 0.04 0.02 0-0.02-0.04-0.06-0.08-0.1-0.1-0.08-0.06-0.04-0.02 0 0.02 0.04 0.06 0.08 0.1 Figure 5. 12 Distorted signal constellation diagram under 3dBm laser power from MATLAB 36

A DFE is sensitive to wrong decisions since the DFE coefficients are updated based on the received signal and the determined signal. A decision error will update the equalizer coefficients in a wrong direction. We compensated 5 sets of received signal using 4 DFE diagrams each of which has perfect error detection, parity check, and no error detection separately and calculated the BER in different settings. The results are shown in Table 5.7. We use the first set of data as a training sequence to obtain coefficients of DFE whose coefficients are fixed. Thus, in Table 5.7, the row that is labeled as Fixed Coefficients refer BER acquired using DFEs whose coefficients are determined by the first set of data and fixed. Again, as shown in Table 5.7, a DFE with fixed coefficients can only slightly decrease BER and there is no significant difference between different DFE schemes. If the DFE is updated with the slicer output continuously without error detection (labeled as No Error Detection in Table 5.7), the system BER increases due to the error propagation. With perfect error detection and parity check, system BER is reduced by more than 85% and VF DFE achieve the lowest BER. Perfect error detection and parity check deliver similar performances. Parity check is not perfect and Table 5.8 shows error missed by parity check. The mean difference is 1.4 which is ignorable in a system with 16,384 symbols. Table 5. 7 Mean BERs for four schemes under different error detection capability Scheme Mean BER for FF Mean BER for VF Mean BER for FV Mean BER for VV Uncompensated 0.049063 0.049063 0.049063 0.049063 Fixed Coefficients 0.044559 0.044556 0.044601 0.044604 Perfect Error Detection 0.00545 0.00505 0.006354 0.00665 Parity Check 0.005447 0.00503 0.006332 0.006714 No Error Detection 0.095721 0.086359 0.098404 0.092383 Table 5. 8 The number of errors missed by parity check Set of Data 1 st Set 2 nd Set 3 rd Set 4 th Set 5 th Set # of missed errors 0 1 1 1 4 37

Equalization performance of DFEs with different error detection capability is visualized in Figure 5.13~5.15. Since the DFE based in FIR(forward)- FIR(feedback) and Volterra(forward)- FIR(feedback) structure has the best performance under 3dBm laser power, we only show the equalized symbol constellation diagram of these two DFE. As shown in Figure 5.13, without error detection, the equalized signal constellations of the system with DFE whose coefficients are continuously updated are even worse than the uncompensated signal constellation shown in Figure 5.12. Error propagation severely degrades the DFE performance in our simulation. On the other hand, the equalized constellation diagrams with perfect error detection (Figure 5.14) or parity check (Figure 5.15), are cleaner than uncompensated signal constellation. Figure 5. 13 Signal constellations after equalization with no error detection under FF and VF systems 38

Figure 5. 14 Signal constellations after equalization with perfect error detection under FF and VF systems Figure 5. 15 Signal constellations after equalization with parity check under FF and VF systems 39