MLSE Channel Estimation Parametric or Non-Parametric?

MLSE Channel Estimation Parametric or Non-Parametric? Optical Fiber Communication Conference 28 Parametric versus Non-Parametric Branch Metrics for MLSE based Receivers with ADC and Clock Recover Stefan Langenbach (1), Gabriella Bosco (2), Pierluigi Poggiolini (2), Theo Kupfer (1) 1) CoreOptics, Nordostpark -14, D-9411 Nürnberg, German {stefan,theo}@coreoptics.com 2) Politecnico di Torino, Corso Duca Abruzzi 24, Torino, Ital {gabriella.bosco,pierluigi.poggiolini}@polito.it OFC 28 JThA6 Part of this work was sponsored b the EU Integrated Project NOBEL II, WP5

Non-Parametric: Canonical Histogram Method HM x = a 2 x r d OFE (AGC) ADC MLSE Non-parametric Method Determine detected bit pattern d Ee and ADC thresholds Associate observed quantized amplitude r Count observed amplitudes r for pattern d into a histogram N(d, r) Metrics for given d and r is ~ log N(d, r) t Pros Simple just counting events Robust insensitive to model mismatch d Amplitude distribution and ADC thresholds = d = 111 Histogram based channel model No parameters are estimated. But a full amplitude histogram needs to be measured for each bit pattern. Cons Data collection time ~ ADC resolution Number of counters ~ ADC resolution -1 1 2 3 4 5 6 7 8 Amplitude Histograms for 3-bit ADC r Canonical metrics Metrics for given r is the logarithm of the observed relative frequenc value. Possibl more sensitive to error propagation? (decision errors translate into metrics errors) Non-Parametric 1 2 3 4 5 6 7 OFC 28 JThA6 2 r

Parametric: : Square Root Method SQRT x = a 2 x z = r d OFE (AGC) ADC MLSE Parametric Method : signal dependent noise The electrical signal has more noise on ones than on zeros. Take square-root z of signal Determine detected bit pattern d Determine mean z-amplitude m(d) for pattern d Ee and mean values t z Ee and mean values t z: signal independent noise The sqrt ed electrical signal has roughl Gaussian noise and roughl signal independent noise. Metrics for given d and z is ~ ( z - m(d) ) ² Amplitude distribution and ADC thresholds Amplitude distribution and ADC thresholds Pros Cons Simple just one parameter per PDF Fast mean value can be estimated quickl Robust decision errors do not corrupt PDF shape Model mismatch penalties are possible DC coupling Note: The square root operation can also be applied implicitl b a non-uniform ADC, or explicitl after the uniform ADC, the latter with minor performance degradation Model PDFs OFC 28 JThA6 3 Model PDFs z Mean based channel model Onl the mean value for each bit pattern needs to be estimated when signal independent noise is postulated (i.e. when the red PDFs are used) Euclidean metrics Metrics for given z is then the Euclidean distance from the mean sqrt ed signal. Parametric

Possible Problems of Parametric Estimation Some problems are specific for a parametric approach. All can lead to wrong metrics. In short: ISI Overload wrong PDF shape and parameters Using a model densit instead of the mixture densit..8.7.6 Model Mismatch under ISI overload Model PDF True PDF Means Model Mismatch Resulting Log-Likelihood (Metric) Note: Noise PDF for coarse state is a mixture densit (i.e. a convex combination of several PDFs) Method-Independent Problems ISI Overload Quantization Channel Memor exceeds State Memor Amplitude differences become invisible Unrealistic Noise Model wrong PDF shape When noise model does not sufficientl accuratel model the true noise PDF, metrics errors are introduced. 6 5 4 Model PDF vs. true PDF real model 2 Model metrics vs. true log-likelihoods real model 3 2-5. - Model Metrics True Log-Likelihoods 1 - Quantization wrong PDF parameters Mean quantized true mean 9 8 7 6 5 4 3 2 true mean histogram mean ADC thresholds Histogram mean versus true mean 15 ps/nm.6ui NRZ Neglected Effects wrong PDF shape or parameters E.g. imperfect clock (here: sinusoidal jitter) 1..9.8.7.6. Model Mismatch under Sinusoidal Jitter Model PDF Jitter PDF True PDF - Resulting Log-Likelihood (Metric) Model Metrics True Log-Likelihoods 1 1 2 3 4 5 6 7 8 9 Using PDF values wrong PMF value Computing PDF is eas, but PMF is hard..7.6 PDF vs PMF OFC 28 JThA6. 4 1 2 3 4 5 Note: Jitter impact on PDF depends on slope and is therefore pattern-dependent: Strong impact on edges. Little impact on rails Wrong Metrics?

Channel Estimation Methods for MLSE Metrics Number of states in Trellis impacts performance MLSE needs metrics. OFE AGC ADC MLSE τ Channel observer associates delaed inputs (quantized waveform samples) and outputs (bit sequences, patterns) Quantizer resolution impacts performance Sampling clock jitter impacts performance CR Metrics are computed from the channel model Metrics Computer Channel Estimator Channel Observer Channel estimator uses the channel observations to estimate a channel model Parametric 1..9.8.7.6 Estimate parameters (e.g. μ, σ) of Probabilit Densit Function (PDF) to Gaussian PDFs compute log-likelihood metrics. 1 2 3 4 5 6 7 8 9 μ σ There are two approaches of channel estimation channel observations estimate parameters compute PDF compute PMF metrics histograms An performance difference? 1 2 3 4 5 6 7 8 OFC 28 JThA6 5 1..9.8.7.6. Non-parametric Estimate probabilities, i.e. values of Probablit Gaussian Histograms Mass Function (PMF)

Abstract and Problem Statement Abstract We We compare the the performance of of MLSE-based receivers with withparametric and and nonparametric channel channel estimation methods and and characterize their their sensitivit against against non- quantization, sampling jitter, jitter, and and intersmbol interference (ISI) (ISI) overload (1) (1) MLSE needs branch metrics Branch metrics are log-likelihoods Two approaches to estimate likelihoods from observations: Problem Statement Non-Parametric Parametric Likelihoods are estimated directl (from observed relative frequencies) Likelihoods are estimated indirectl (parameters of a probabilistic model are estimated from observations) (1) ISI Overload: The phsical channel memor exceeds the state memor of the MLSE Do Do parametric models models suffer suffer from from effects effects not not covered covered in in the the model? model? Are Are there there relevant model mismatch penalties? Simulation Approach Histogram Method HM SQRT method SQRT a practice-proven canonical method of non-parametric channel estimation a particularl efficient example of a parametric method Compare ultimatel and and practicall achievable performance of of HM HM and and of of SQRT. SQRT. OFC 28 JThA6 6

Results and Conclusions SQRT Method compared to Histogram Method (@ BER - 3 ) in a Nutshell Ultimate Performance? Practical Performance? Model Mismatch Penalt? Quantization Penalt? Jitter Penalt? identical slightl worse, but... es, but... es, but... no without complexit limitations, i.e. for unlimited ADC resolution and unlimited number of states penalt is not ver relevant achieves the same dispersion limit (e.g. 5 ps/nm at 15 db) onl at low dispersion and for PMD and outside of useable operation range significant onl for 3-bit ADC not for relevant jitter magnitudes Conclusions We We compared compared ultimatel ultimatel and and practicall practicall achievable achievable performance performance We We assumed assumed that that SQRT SQRT suffers suffers more more from from quantization quantization and and model model mismatch mismatch We We found found such such penalties penalties but but the the are are not not ver ver significant significant The The HM HM channel channel estimator estimator has has practical practical performance performance advantage advantage for for 3-bit 3-bit ADC ADC The The SQRT SQRT channel channel estimator estimator has has speed speed & complexit complexit advantages advantages for for N-bit N-bit ADC ADC For For further further stud: stud: Model Model mismatch mismatch penalties penalties at at lower lower BER? BER? OFC 28 JThA6 7

Simulation Setups References Non-parametric Channel Estimation Transmitter Tx Fiber (linear) ASE noise + Optical filter Photo Diode ( )² Electrical filter HM MLSE SQRT MLSE parameters varied suggested earl for MLSE usage in non-linear channel W. Sauer-Greff et al., "Modified Volterra Series and State Model Approach, Proc. IEEE Sig Proc 99, 19-23 H. F. Haunstein et al., Design of near optimum electrical equalizers for optical transmission, OFC 21, WAA 4-1 implemented in real sstems, e.g. A. Färbert et al., Performance of a.7 Gb/s Receiver with Digital Equalizer using, ECOC 24, Th4.1.5 Two setups were used (man) experimental data available, e.g. Data Format Shaping Filter Extinction Ratio Fiber Opt. filter El. Filter AGC / ADC MLSE Setup 1 Bad Tx for CD, PMD, Jitter DeBruijn-15 (2 19 bits, 32 samples/bit) UI rise-time erfc shaped + 1-pole Bessel (.7 GHz) 11.8 db NRZ @.7 Gbit/s for unconstrained complexit infinite SSMF (D=16 ps/nm), linear propagation Flat Top (4GHz) 4-pole Bessel (7.5 GHz) gain optimized (1) roughl, best sampling phase, varied quantizer resolution 2 samples per bit, self-training, varied number of states Setup 2 Good Tx PRBS-18 (2 18 bits, 2 samples/bit) 5-pole Bessel (7.5 GHz) SuperGauss 2 nd Order (35 GHz) 5-pole Bessel (7.5 GHz) (1) For HM, gain was not optimized. Mean rectified value was maintained at a constant level. J.P. Elbers et al., Measurement of the dispersion tolerance of optical duobinar..., OFC 25, OThJ4 S. Chandrasekhar et al., Chirp-managed laser and MLSE-RX..., PTL, Vol. 18, No. 14, 156-1562, 26 S. Chandrasekhar et al., Performance of MLSE Receiver..., PTL, Vol. 18, No. 23, 2448-245, 26 J. M. Gené et al., Joint PMD and Chromatic Dispersion Compensation Using an MLSE, ECOC 26, We2.5.2 I. L. L. Polo et al., Comparison of Maximum Likelihood Sequence Estimation equalizer..., ECOC 26, We2.5.3 J. D. Downie et al., Experimental Measurements of the Effectiveness of MLSE..., OFC 27, OMG4 C. Xie et. al., Performance Evaluation of Electronic Equalizers for Dnamic PMD..., OFC 27, OTuA7 Parametric Channel Estimation studied since long (for perfomance analsis), e.g. P. A. Humblet, M. Azizoglu, On the Bit Error Rate, JLT 9/11 p.1577 (3), 1991 (and predecessors) A. Weiss, On the Performance of Electrical Equalization in Optical Fiber Transmission Sstems, PTL, Vol. 15, 25-27, 23 well covered in recent MLSE literature, e.g. D. E. Crivelli et al., On the Performance of Reduced-State Viterbi Receivers, ECOC 24, We4.P.83 N. Alic et al., Signal statistics and maximum likelihood sequence estimation, Optics Express, Vol.13, No., 4568-4578, 25 G. Bosco et al., Long-Distance Effectiveness of MLSE IMDD Receivers, PTL, vol. 18, pp.37-39, 26 T. Freckmann, J. Speidel, Viterbi Equalizer with Analticall Calculated Branch Metrics PTL, Vol. 18, 277-279, 26 T. Foggi et al., Maximum-likelihood sequence detection with closed-form metrics, JLT, Vol. 24, No. 8, 373-387, 26 P. Poggiolini et al., Branch Metrics for Effective Long-Haul MLSE'', ECOC 26, We2.5 M. R. Hueda et al., Parametric Estimation of IM/DD Optical Channels, JLT, Vol. 25, No. 3, 957-975, 27 (some) experimental data from offline simulations, e.g. P. Poggiolini et al., 1,4 km uncompensated IMDD transmission, ECOC 26, post-deadline Th4.4.6 Setups References OFC 28 JThA6 8

Performance with Unconstrained MLSE and ADC SQRT Method Histogram Method SQRT versus HM penalt Slightl increased quantization penalt OSNR (db) 3 bits 4 bits 5 bits 13 SQRT: 2, 8, 32, 64, 256 states 2 3 4 L / km 13 HM: 2, 8, 32, 64, 256 states Same achievable dispersion performance 11 9 3 bits 4 bits 5 bits unquantized KLSE 11 9 3 bits 4 bits 5 bits Series3 KLSE Exact Metrics (Karhunen Loeve Series Expansion) (b) 2 3 4 L (km) (c) 2 3 4 L (km) Setup: Good Tx db model mismatch penalt (with infinite Extinction Ratio) Achievable Performance? No difference for fine ADC OFC 28 JThA6 9

Dispersion Tolerance with 16-states MLSE and ADC SQRT Method Histogram Method Irrelevant small model mismatch penalt under ISI overload (outside of useable operation range!) OSNR (db) - - - SQRT versus HM penalt 3 bits 4 bits 2 3 4 5 6 Artifact! Penalt remains positive when HM is gain optimized Chromatic Dispersion (ps/nm) 2 5 db larger quantization penalt for 3-bit ADC at medium CD 2 ADC Resolution ADC Resolution 18 16 14 blank 3 bit 4 bit unquantized SQRT (16 States) 18 16 14 2 bit 3 bit 4 bit 6 bit HM (16 States) (a) 2 3 4 5 6 Chromatic Dispersion (ps/nm) (b) 2 3 4 5 6 Chromatic Dispersion (ps/nm) still db back-to-back model mismatch penalt (at db extinction ratio) Setup: Bad Tx Dispersion? Relevant differences are small OFC 28 JThA6

1 st order PMD with 4-states MLSE and ADC 1 st SQRT Method Histogram Method SQRT versus HM penalt Large but irrelevant model mismatch penalt for ISI overload (outside of useable operation range) OSNR (db) 1.8.6 3 bits 4 bits 2 db quantization penalt for 3-bit ADC 1 1.5 2 DGD (UI) 2 ADC Resolution ADC Resolution 18 16 14 blank 3 bit 4 bit unquantized SQRT (4 States) 18 16 14 2 bit 3 bit 4 bit 6 bit HM (4 States) (a) 1 1.5 2 DGD (UI) (b) 1 1.5 2 DGD (UI) Setup: Bad Tx PMD? Relevant differences remain small OFC 28 JThA6 11

Clock Recover with Jitter (16-states and 4-bit4 ADC) Sinusoidal Jitter Gaussian Jitter (Test Signal) Model Mismatch under Sinusoidal Jitter Resulting Log-Likelihood (Metric) Model Mismatch under Gaussian Jitter Resulting Log-Likelihood (Metric) 1..9.8 Model PDF Jitter PDF True PDF 1..9.8 Model PDF Jitter PDF True PDF.7.7.6.6. - Model Metrics True Log-Likelihoods. - Model Metrics True Log-Likelihoods 14 14 13 11 3 ps/nm, HM 3 ps/nm, SQRT blank blank ps/nm, HM ps/nm, SQRT 13 11 3 ps/nm, HM 3 ps/nm, SQRT blank blank ps/nm, HM ps/nm, SQRT (a) HF Jitter Specification Limit:.5 5 Sinusoidal Peak-to-Peak Jitter (UI) (b) 6 σ ~ 5 UI pp.2.4.6.8 Gaussian RMS Jitter (UI) Setup: Bad Tx Jitter?? No relevant difference OFC 28 JThA6