Advanced Textbooks in Control and Signal Processing

Transcription

1 Advanced Textbooks in Control and Signal Processing

2 Series Editors Professor Michael J. Grimble, Professor of Industrial Systems and Director Professor Emeritus Michael A. Johnson, Professor of Control Systems and Deputy Director Industrial Control Centre, Department of Electronic and Electrical Engineering, University of Strathclyde, Graham Hills Building, 50 George Street, Glasgow G1 1QE, U.K. Other titles published in this series: Genetic Algorithms K.F. Man, K.S. Tang and S. Kwong Neural Networks for Modelling and Control of Dynamic Systems M. Nørgaard, O. Ravn, L.K. Hansen and N.K. Poulsen Modelling and Control of Robot Manipulators (2nd Edition) L. Sciavicco and B. Siciliano Fault Detection and Diagnosis in Industrial Systems L.H. Chiang, E.L. Russell and R.D. Braatz Soft Computing L. Fortuna, G. Rizzotto, M. Lavorgna, G. Nunnari, M.G. Xibilia and R. Caponetto Statistical Signal Processing T. Chonavel Discrete-time Stochastic Processes (2nd Edition) T. Söderström Parallel Computing for Real-time Signal Processing and Control M.O. Tokhi, M.A. Hossain and M.H. Shaheed Multivariable Control Systems P. Albertos and A. Sala Control Systems with Input and Output Constraints A.H. Glattfelder and W. Schaufelberger Analysis and Control of Non-linear Process Systems K. Hangos, J. Bokor and G. Szederkényi Model Predictive Control (2nd Edition) E.F. Camacho and C. Bordons Digital Self-tuning Controllers V.Bobál,J.Böhm,J.FesslandJ.Macháček Control of Robot Manipulators in Joint Space R. Kelly, V. Santibáñez and A. Loría Publication due July 2005 Robust Control Design with MATLAB D.-W. Gu, P.Hr. Petkov and M.M. Konstantinov Publication due July 2005 Active Noise and Vibration Control M.O. Tokhi Publication due November 2005

3 A. Zaknich Principles of Adaptive Filters and Self-learning Systems With 95 Figures 123

4 Anthony Zaknich, PhD School of Engineering Science, Rockingham Campus, Murdoch University, South Street, Murdoch, WA 6150, Australia and Centre for Intelligent Information Processing Systems, School of Electrical, Electronic and Computer Engineering, The University of Western Australia, 35 Stirling Highway, Crawley, WA 6009, Australia Instructors Solutions Manual in PDF can be downloaded from the book s page at springeronline.com British Library Cataloguing in Publication Data Zaknich, Anthony Principles of adaptive filters and self-learning systems. (Advanced textbooks in control and signal processing) 1. Adaptive filters 2. Adaptive signal processing 3. System analysis I. Title ISBN-10: Library of Congress Control Number: Apart from any fair dealing for the purposes of research or private study, or criticism or review, as permitted under the Copyright, Designs and Patents Act 1988, this publication may only be reproduced, stored or transmitted, in any form or by any means, with the prior permission in writing of the publishers, or in the case of reprographic reproduction in accordance with the terms of licences issued by the Copyright Licensing Agency. Enquiries concerning reproduction outside those terms should be sent to the publishers. Advanced Textbooks in Control and Signal Processing series ISSN ISBN ISBN Springer Science+Business Media springeronline.com Springer-Verlag London Limited 2005 The use of registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant laws and regulations and therefore free for general use. The publisher makes no representation, express or implied, with regard to the accuracy of the information contained in this book and cannot accept any legal responsibility or liability for any errors or omissions that may be made. Typesetting: Camera ready by author Production: LE-TEXJelonek,Schmidt&VöcklerGbR,Leipzig, Germany Printed in Germany 69/ Printed on acid-free paper SPIN

5 Contents Part I Introduction 1 1 Adaptive Filtering Linear Adaptive Filters Linear Adaptive Filter Algorithms Nonlinear Adaptive Filters Adaptive Volterra Filters Nonclassical Adaptive Systems Artificial Neural Networks Fuzzy Logic Genetic Algorithms A Brief History and Overview of Classical Theories Linear Estimation Theory Linear Adaptive Filters Adaptive Signal Processing Applications Adaptive Control A Brief History and Overview of Nonclassical Theories Artificial Neural Networks Fuzzy Logic Genetic Algorithms Fundamentals of Adaptive Networks Choice of Adaptive Filter Algorithm Linear Systems and Stochastic Processes Basic Concepts of Linear Systems Discrete-time Signals and Systems The Discrete Fourier Transform (DFT) Discrete Linear Convolution using the DFT Digital Sampling Theory Analogue Interpretation Formula The Fast Fourier Transform The z-transform Relationship between Laplace Transform and z-transform Bilateral z-transform...41

6 xvi Contents Unilateral z-transform Region of Convergence (ROC) for the z-transform Region of Convergence (ROC) for General Signals General Properties of the DFT and z-transform Summary of Discrete-Time LSI Systems Special Classes of Filters Phase Response from Frequency Magnitude Response Linear Algebra Summary Vectors Linear Independence, Vector Spaces, and Basic Vectors Matrices Linear Equations Special Matrices Quadratic and Hermitian Forms Eigenvalues and Eigenvectors Introduction to Stochastic Processes Random Signals Basic Descriptive Models of Random Signals The Mean Square Value and Variance The Probability Density Function Jointly Distributed Random Variables The Expectation Operator The Autocorrelation and Related Functions Power Spectral Density Functions Coherence Function Discrete Ergodic Random Signal Statistics Autocovariance and Autocorrelation Matrices Spectrum of a Random Process Filtering of Random Processes Important Examples of Random Processes Gaussian Process White Noise White Sequences Gauss-Markov Processes The Random Telegraph Wave Exercises Problems...82 Part II Modelling 87 3 Optimisation and Least Square Estimation Optimisation Theory Optimisation Methods in Digital Filter Design Least Squares Estimation Least Squares Maximum Likelihood Estimator...97

7 Contents xvii 3.5 Linear Regression Fitting Data to a Line General Linear Least Squares A Ship Positioning Example of LSE Acoustic Positioning System Example Measure of LSE Precision Measure of LSE Reliability Limitations of LSE Advantages of LSE The Singular Value Decomposition The Pseudoinverse Computation of the SVD The Jacobi Algorithm The QR Algorithm Exercises Problems Parametric Signal and System Modelling The Estimation Problem Deterministic Signal and System Modelling The Least Squares Method The Padé Approximation Method Prony s Method All-Pole Modelling using Prony s Method Linear Prediction Digital Wiener Filter Autocorrelation and Covariance Methods Stochastic Signal Modelling Autoregressive Moving Average Models Autoregressive Models Moving Average Models The Levinson-Durbin Recursion and Lattice Filters The Levinson-Durbin Recursion Development Example of the Levinson-Durbin Recursion The Lattice Filter The Cholesky Decomposition The Levinson Recursion Exercises Problems Part III Classical Filters and Spectral Analysis Optimum Wiener Filter Derivation of the Ideal Continuous-time Wiener Filter The Ideal Discrete-time FIR Wiener Filter General Noise FIR Wiener Filtering...164

8 xviii Contents FIR Wiener Linear Prediction Discrete-time Causal IIR Wiener Filter Causal IIR Wiener Filtering Wiener Deconvolution Exercises Problems Optimal Kalman Filter Background to The Kalman Filter The Kalman Filter Kalman Filter Examples Kalman Filter for Ship Motion Kalman Tracking Filter Proper Simple Example of a Dynamic Ship Models Stochastic Models Alternate Solution Models Advantages of Kalman Filtering Disadvantages of Kalman Filtering Extended Kalman Filter Exercises Problems Power Spectral Density Analysis Power Spectral Density Estimation Techniques Nonparametric Spectral Density Estimation Periodogram Power Spectral Density Estimation Modified Periodogram Data Windowing Bartlett s Method Periodogram Averaging Welch s Method Blackman-Tukey Method Performance Comparisons of Nonparametric Models Minimum Variance Method Maximum Entropy (All Poles) Method Parametric Spectral Density Estimation Autoregressive Methods Yule-Walker Approach Covariance, Least Squares and Burg Methods Model Order Selection for the Autoregressive Methods Moving Average Method Autoregressive Moving Average Method Harmonic Methods Eigendecomposition of the Autocorrelation Matrix Pisarenko s Method MUSIC...222

9 Contents xix 7.4 Exercises Problems Part IV Adaptive Filter Theory Adaptive Finite Impulse Response Filters Adaptive Interference Cancelling Least Mean Squares Adaptation Optimum Wiener Solution The Method of Steepest Gradient Descent Solution The LMS Algorithm Solution Stability of the LMS Algorithm The Normalised LMS Algorithm Recursive Least Squares Estimation The Exponentially Weighted Recursive Least Squares Algorithm Recursive Least Squares Algorithm Convergence Convergence of the Filter Coefficients in the Mean Convergence of the Filter Coefficients in the Mean Square Convergence of the RLS Algorithm in the Mean Square The RLS Algorithm as a Kalman Filter Exercises Problems Frequency Domain Adaptive Filters Frequency Domain Processing Time Domain Block Adaptive Filtering Frequency Domain Adaptive Filtering The Overlap-Save Method The Overlap-Add Method The Circular Convolution Method Computational Complexity Exercises Problems Adaptive Volterra Filters Nonlinear Filters The Volterra Series Expansion A LMS Adaptive Second-order Volterra Filter A LMS Adaptive Quadratic Filter A RLS Adaptive Quadratic Filter Exercises...264

10 xx Contents Problems Adaptive Control Systems Main Theoretical Issues Introduction to Model-reference Adaptive Systems The Gradient Approach Least Squares Estimation A General Single-input-single-output MRAS Lyapunov s Stability Theory Introduction to Self-tuning Regulators Indirect Self-tuning Regulators Direct Self-tuning Regulators Relations between MRAS and STR Applications Part V Nonclassical Adaptive Systems Introduction to Neural Networks Artificial Neural Networks Definitions Three Main Types Specific Artificial Neural Network Paradigms Artificial Neural Networks as Black Boxed Implementation of Artificial Neural Networks When to Use an Artificial Neural Network How to Use an Artificial Neural Network Artificial Neural Network General Applications Simple Application Examples Sheep Eating Phase Identification from Jaw Sounds Hydrate Particle Isolation in SEM Images Oxalate Needle Detection in Microscope Images Water Level Determination from Resonant Sound Analysis Nonlinear Signal Filtering A Motor Control Example A Three-layer Multi-layer Perceptron Model MLP Backpropagation-of-error Learning Derivation of Backpropagation-of-error Learning Change in Error due to Output Layer Weights Change in Error due to Hidden Layer Weights The Weight Adjustments Additional Momentum Factor Notes on Classification and Function Mapping MLP Application and Training Issues...308

11 Contents xxi 12.3 Exercises Problems Introduction to Fuzzy Logic Systems Basic Fuzzy Logic Fuzzy Logic Membership Functions Fuzzy Logic Operations Fuzzy Logic Rules Fuzzy Logic Defuzzification Fuzzy Logic Control Design Fuzzy Logic Controllers Control Rule Construction Parameter Tuning Control Rule Revision Fuzzy Artificial Neural Networks Fuzzy Applications Introduction to Genetic Algorithms A General Genetic Algorithm The Common Hypothesis Representation Genetic Algorithm Operators Fitness Functions Hypothesis Searching Genetic Programming Applications of Genetic Programming Filter Circuit Design Applications of GAs and GP Tic-tac-to Game Playing Application of GAs Part VI Adaptive Filter Application Applications of Adaptive Signal Processing Adaptive Prediction Adaptive Modelling Adaptive Telephone Echo Cancelling Adaptive Equalisation of Communication Channels Adaptive Self-tuning Filters Adaptive Noise Cancelling Focused Time Delay Estimation for Ranging Adaptive Array Processing Other Adaptive Filter Applications Adaptive 3-D Sound Systems Microphone Arrays Network and Acoustic Echo Cancellation Real-world Adaptive Filtering Applications...353

12 xxii Contents 16 Generic Adaptive Filter Structures Sub-band Adaptive Filters Sub-space Adaptive Filters MPNN Model Approximately Piecewise Linear Regression Model The Sub-space Adaptive Filter Model Example Applications of the SSAF Model Loudspeaker 3-D Frequency Response Model Velocity of Sound in Water 3-D Model Discussion and Overview of the SSAF References Index...381

13