Modeling, Estimation and Optimal Filtering in Signal Processing. Mohamed Najim

Size: px

Start display at page:

Download "Modeling, Estimation and Optimal Filtering in Signal Processing. Mohamed Najim"

Owen Gardner
5 years ago
Views:

1 Modeling, Estimation and Optimal Filtering in Signal Processing Mohamed Najim

2 This page intentionally left blank

3 Modeling, Estimation and Optimal Filtering in Signal Processing

4 This page intentionally left blank

5 Modeling, Estimation and Optimal Filtering in Signal Processing Mohamed Najim

6 First published in France in 2006 by Hermes Science/Lavoisier entitled Modélisation, estimation et filtrage optimal en traitement du signal First published in Great Britain and the United States in 2008 by ISTE Ltd and John Wiley & Sons, Inc. Apart from any fair dealing for the purposes of research or private study, or criticism or review, as permitted under the Copyright, Designs and Patents Act 1988, this publication may only be reproduced, stored or transmitted, in any form or by any means, with the prior permission in writing of the publishers, or in the case of reprographic reproduction in accordance with the terms and licenses issued by the CLA. Enquiries concerning reproduction outside these terms should be sent to the publishers at the undermentioned address: ISTE Ltd John Wiley & Sons, Inc. 6 Fitzroy Square 111 River Street London W1T 5DX Hoboken, NJ UK USA ISTE Ltd, 2008 LAVOISIER, 2006 The rights of Mohamed Najim to be identified as the author of this work have been asserted by him in accordance with the Copyright, Designs and Patents Act Library of Congress Cataloging-in-Publication Data Najim, Mohamed. [Modélisation, estimation et filtrage optimal en traitement du signal. English] Modeling, Estimation and Optimal Filtering in Signal Processing / Mohamed Najim. p. cm. Includes bibliographical references and index. ISBN: Electric filters, Digital. 2. Signal processing--digital techniques. I. Title. TK7872.F5N '2--dc British Library Cataloguing-in-Publication Data A CIP record for this book is available from the British Library ISBN: Printed and bound in Great Britain by Antony Rowe Ltd, Chippenham, Wiltshire.

7 Table of Contents Preface... xi Chapter 1. Parametric Models Introduction Discrete linear models The moving average (MA) model The autoregressive (AR) model Observations on stability, stationarity and invertibility AR model case ARMA model case The AR model or the ARMA model? Sinusoidal models The relevance of the sinusoidal model Sinusoidal models State space representations Definitions State space representations based on differential equation representation Resolution of the state equations State equations for a discrete-time system Some properties of systems described in the state space Introduction Observability Controllability Plurality of the state space representation of the system Case 1: state space representation of AR processes Case 2: state space representation of MA processes Case 3: state space representation of ARMA processes... 36

8 vi Modeling, Estimation and Optimal Filtering in Signal Processing Case 4: state space representation of a noisy process An AR process disturbed by a white noise AR process disturbed by colored noise itself modeled by another AR process AR process disturbed by colored noise itself modeled by a MA process Conclusion References Chapter 2. Least Squares Estimation of Parameters of Linear Models Introduction Least squares estimation of AR parameters Determination or estimation of parameters? Recursive estimation of parameters Implementation of the least squares algorithm The least squares method with weighting factor A recursive weighted least squares estimator Observations on some variants of the least squares method The autocorrelation method Levinson s algorithm The Durbin-Levinson algorithm Lattice filters The covariance method Relation between the covariance method and the least squares method Effect of a white additive noise on the estimation of AR parameters A method for alleviating the bias on the estimation of the AR parameters Generalized least squares method The extended least squares method Selecting the order of the models References Chapter 3. Matched and Wiener Filters Introduction Matched filter Introduction Matched filter for the case of white noise Matched filter for the case of colored noise Formulation of problem Physically unrealizable matched filter...115

9 Table of Contents vii A matched filter solution using whitening techniques The Wiener filter Introduction Formulation of problem The Wiener-Hopf equation Error calculation in a continuous physically non-realizable Wiener filter Physically realizable continuous Wiener filter. Rational spectra case Discrete-time Wiener filter Finite impulse response (FIR) Wiener filter Infinite impulse response (IIR) Wiener filter Application of non-causal discrete Wiener filter to speech enhancement Modified filter expression Experimental results Enhancement using combination of AR model and non-causal Wiener filter References Chapter 4. Adaptive Filtering Introduction Recursive least squares algorithm Exact RLS method Forgetting factor RLS method The least mean squares algorithm Variants of the LMS algorithm Normalized least mean squares (NLMS) Affine projection algorithm (APA) Summary of the properties of the different adaptive filters Application: noise cancellation References Chapter 5. Kalman Filtering Introduction Derivation of the Kalman filter Statement of problem Propagation step: relationship between xˆ ( k 1/ k) and x ˆ( k / k) ; recurrence relationship between the error covariance matrices P(k + 1 / k) and P(k / k) Update step: relationship between x ˆ( k / k) and x ˆ( k / k 1) ; recursive relationship between P(k / k) and P(k / k 1)...189

10 viii Modeling, Estimation and Optimal Filtering in Signal Processing Expression of the Kalman filter gain Implementation of the filter The notion of innovation Derivation of the Kalman filter for correlated processes Relationship between the Kalman filter and the least squares method with forgetting factor Application of the Kalman filter to parameter estimation Estimation of the parameters of an AR model Application to speech analysis Nonlinear estimation Model linearization: linearized Kalman filter The extended Kalman filter (EKF) Applications of the EKF Parameter estimation of a noisy speech signal Application to tracking formant trajectories of speech signals Conclusion References Chapter 6. Application of the Kalman Filter to Signal Enhancement Introduction Enhancement of a speech signal disturbed by a white noise State space representation of the noisy speech signal Speech enhancement procedure State of the art dedicated to the single-channel enhancement methods using Kalman filtering Alternative methods based on projection between subspaces Introduction Preliminary observations Relation between subspace-based identification methods and the Kalman algorithm Signal prediction using the optimal Kalman filter Kalman filtering and/or smoothing combined with subspace identification methods Simulation results Innovation-based approaches Introduction Kalman-filter based enhancement without direct estimation of variances Q and R Kalman-filter based enhancement using a suboptimal gain Alternative approach to Kalman-filter based enhancement, using the estimation of variances Q and R...244

11 Table of Contents ix 6.3. Kalman filter-based enhancement of a signal disturbed by a colored noise Conclusion References Chapter 7. Estimation using the Instrumental Variable Technique Introduction Introduction to the instrumental variable technique Principle Review of existing instrumental variable methods for the estimation of AR parameters Kalman filtering and the instrumental variable method Signal estimation using noisy observations Estimation of AR parameters using the filtered signal Estimation of the variances of the driving process and the observation noise Concluding observations Case study Preliminary observations Comparative study. Case 1: white additive noise Conclusion References Chapter 8. H Estimation: an Alternative to Kalman Filtering? Introduction Introduction to H estimation Definition of the H norm H filtering Riccati equation-based recursive solution of H filtering Review of the use of H filtering in signal processing Estimation of AR parameters using H filtering H filtering for the estimation of AR parameters Dual H estimation of the AR process and its parameters Relevance of H filtering to speech enhancement Conclusion References Chapter 9. Introduction to Particle Filtering Monte Carlo methods Sequential importance sampling filter...321

12 x Modeling, Estimation and Optimal Filtering in Signal Processing 9.3. Review of existing particle filtering techniques References Appendix A. Karhunen Loeve Transform Appendix B. Subspace Decomposition for Spectral Analysis Appendix C. Subspace Decomposition Applied to Speech Enhancement Appendix D. From AR Parameters to Line Spectrum Pair Appendix E. Influence of an Additive White Noise on the Estimation of AR Parameters Appendix F. The Schur-Cohn Algorithm Appendix G. The Gradient Method Appendix H. An Alternative Way of Understanding Kalman Filtering Appendix I. Calculation of the Kalman Gain using the Mehra Approach..373 Appendix J. Calculation of the Kalman Gain (the Carew and Belanger Method) Appendix K. The Unscented Kalman Filter (UKF) Index...391

Preface There is no royal road to science, and only those who do not dread the fatiguing climb of its steep paths have a chance of gaining its luminous summits.

13 Preface There is no royal road to science, and only those who do not dread the fatiguing climb of its steep paths have a chance of gaining its luminous summits. Karl Marx The core of this book is taken from a series of lectures I gave in Shanghai in 1983 and On this occasion, the Chinese Association of Science and Technology (CAST) gave to me a calligraphic version of philosopher Karl Marx s famous quotation. I have always found the above quotation very pertinent and often used it to perk up my students in times of discouragement. I would like to thank Professor Yong Xiang Lu, President of the Chinese Sciences Academy, who encouraged me to use the quotation at the beginning of this book.

14 xii Modeling, Estimation and Optimal Filtering in Signal Processing The performances of computers in general, and signal processors in particular, have been constantly evolving towards smaller and faster versions with higher and higher storage capabilities. This improved performance allows today s engineer to implement algorithms that are ever more complex 1. Through this book, we wish to give the reader the significant results hitherto obtained in the field of parametric signal modeling, and to present some new approaches. To the best of our knowledge, these results are dispersed through various textbooks, and there is no single compendium grouping them all together. Moreover, a large part of these results are only presented in journal articles often inaccessible to the average student. This book attempts to fill this gap. We will mostly consider signal estimation/identification, traditionally grouped in the domain of control engineering. Today, however, this parameter estimation/identification deserves a stature of its own in view of its recent maturity. One example where the deserved importance was given to it is the tri-annual conferences organized by the International Federation of Automatic Control (IFAC) and called the Symposium on Identification and System Parameter Estimation. A recent trend during these conferences has been the increasing importance given to challenges lying in the field of signal processing. Over the past 15 years or so, identification in signal processing has undergone fervent activity, even a renaissance. One of the fundamental differences between control engineering and signal/image processing comes from the nature of the input signal. In the former, the input is known whereas in the latter, it is always unknown. Several problems specific to signal processing have their roots in this difference. For example, the rapid development of digital communications has led to a renewed interest in the identification of Single Input Multiple Output (SIMO) and Multiple Input Multiple Output (MIMO) systems. Additionally, equalization and blind-deconvolution issues are also increasingly important. This rise in interest is best attested by the number and quality of related articles in the IEEE Transactions on Signal Processing and the ICASSP conferences. This book is organized into 9 chapters. Chapter 1 starts with a brief introduction and review of the basic theory of discrete linear models, notably the AR and ARMA models. We then analyze the shortcomings of these models and present an alternative composed of sinusoidal models and ARCOS models to characterize periodic signals. 1 The term complex here includes both the computation cost and the abstraction level involved in the associated mathematical approaches.

15 Preface xiii Once we have chosen the model and its order, the estimation of its parameters has to be addressed. In Chapter 2, we present the least squares method and its variants in signal processing, namely the autocorrelation method and the covariance method for the AR model. For a non-recursive case, we then define the Normal (or Yule-Walker) equations. Thereafter, we take up the recursive forms of the least squares algorithm and consider the lower-complexity algorithms such as the Levinson and Durbin-Levinson methods. This latter topic also serves as the framework in which we introduce the reflection coefficients and the lattice algorithms. However, as we will note in the appropriate place, the least squares method gives biased estimations for correlated measurements. To get unbiased estimates, we introduce the generalized least squares method and the extended least squares method. Finally, we analyze the effect of an additive white measurement noise on the least squares estimation of AR parameters. We also present a review of existing methods used to compensate for the influence of the measurement noise. In Chapters 3 to 5, we take up parameter estimation using optimal filters and adaptive filters such as the LMS, RLS and APA. For the discrete-time case, we present the relation linking the Wiener filter to the least squares method. To put R. E. Kalman s contribution into perspective, we first present N. Wiener s original derivation [1] for continuous signals, leading to the Wiener Hopf integral equation. The non-recursive nature of the Wiener filter led Kalman to propose an alternative recursive solution, the Kalman filter [2]. This alternative solution consisted of the transformation of the integral equation into a differential stochastic equation, for which he then found a recursive solution. In Chapter 5, we derive the Kalman filter using an algebraic approach. Even though this algebraic approach may lack a certain elegance, it is based on fundamental notions of linear algebra. This presentation can form the starting point for the interested reader, leading him to the innovation-based presentation of the Kalman filter presented by Kailath et al. in their book Linear Estimation [3]. We then present the Extended Kalman Filter (EKF) for nonlinear cases. This extended filter is useful when we have to carry out the joint estimation of the desired signal and the corresponding model parameters associated with it. The EKF is, however, not the only possible solution for nonlinear estimation cases. The purpose of the following chapters is to present other solutions, treating a case often seen in signal processing: when the covariance matrices of the driving process Q and the noise R are not known a priori. Thus, in Chapter 6, we restate the classic methods proposed by Carew-Belanger and R. K. Mehra in the domain of control in the early 1970s. The use of sub-space approaches for identification frees us from the constraint of having known covariance matrices of Q and R; this allows us to see the signal enhancement

Time Frequency Domain for Segmentation and Classification of Non-stationary Signals

Time Frequency Domain for Segmentation and Classification of Non-stationary Signals FOCUS SERIES Series Editor Francis Castanié Time Frequency Domain for Segmentation and Classification of Non-stationary