Advanced Digital Signal Processing and Noise Reduction

Transcription

1 Advanced Digital Signal Processing and Noise Reduction Fourth Edition Professor Saeed V. Vaseghi Professor of Communications and Signal Processing Department of Electronics & Computer Engineering Brunel University, London, UK A John Wiley and Sons, Ltd, Publication

2

3 Advanced Digital Signal Processing and Noise Reduction

4

5 Advanced Digital Signal Processing and Noise Reduction Fourth Edition Professor Saeed V. Vaseghi Professor of Communications and Signal Processing Department of Electronics & Computer Engineering Brunel University, London, UK A John Wiley and Sons, Ltd, Publication

6 This edition first published John Wiley & Sons Ltd. Registered office John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex, PO19 8SQ, United Kingdom For details of our global editorial offices, for customer services and for information about how to apply for permission to reuse the copyright material in this book please see our website at The right of the author to be identified as the author of this work has been asserted in accordance with the Copyright, Designs and Patents Act All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, except as permitted by the UK Copyright, Designs and Patents Act 1988, without the prior permission of the publisher. Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic books. Designations used by companies to distinguish their products are often claimed as trademarks. All brand names and product names used in this book are trade names, service marks, trademarks or registered trademarks of their respective owners. The publisher is not associated with any product or vendor mentioned in this book. This publication is designed to provide accurate and authoritative information in regard to the subject matter covered. It is sold on the understanding that the publisher is not engaged in rendering professional services. If professional advice or other expert assistance is required, the services of a competent professional should be sought. Library of Congress Cataloging-in-Publication Data Vaseghi, Saeed V. Advanced digital signal processing and noise reduction / Saeed Vaseghi. 4th ed. p. cm. Includes bibliographical references and index. ISBN (cloth) 1. Signal processing. 2. Electronic noise. 3. Digital filters (Mathematics) I. Title. TK V dc A catalogue record for this book is available from the British Library ISBN (H/B) Set in 9/11pt Times by Integra Software Services Pvt. Ltd, Pondicherry, India Printed in Singapore by Markono Print Media Pte Ltd.

7 To my Luke

8

9 Contents Preface Acknowledgements Symbols Abbreviations xix xxiii xxv xxix 1 Introduction Signals, Noise and Information Signal Processing Methods Transform-Based Signal Processing Source-Filter Model-Based Signal Processing Bayesian Statistical Model-Based Signal Processing Neural Networks Applications of Digital Signal Processing Digital Watermarking Bio-medical, MIMO, Signal Processing Echo Cancellation Adaptive Noise Cancellation Adaptive Noise Reduction Blind Channel Equalisation Signal Classification and Pattern Recognition Linear Prediction Modelling of Speech Digital Coding of Audio Signals Detection of Signals in Noise Directional Reception of Waves: Beam-forming Space-Time Signal Processing Dolby Noise Reduction Radar Signal Processing: Doppler Frequency Shift A Review of Sampling and Quantisation Advantages of Digital Format Digital Signals Stored and Transmitted in Analogue Format The Effect of Digitisation on Signal Bandwidth Sampling a Continuous-Time Signal Aliasing Distortion Nyquist Sampling Theorem 27

10 viii Contents Quantisation Non-Linear Quantisation, Companding Summary 32 Bibliography 32 2 Noise and Distortion Introduction Different Classes of Noise Sources and Distortions Different Classes and Spectral/Temporal Shapes of Noise White Noise Band-Limited White Noise Coloured Noise; Pink Noise and Brown Noise Impulsive and Click Noise Transient Noise Pulses Thermal Noise Shot Noise Flicker (I/f ) Noise Burst Noise Electromagnetic (Radio) Noise Natural Sources of Radiation of Electromagnetic Noise Man-made Sources of Radiation of Electromagnetic Noise Channel Distortions Echo and Multi-path Reflections Modelling Noise Frequency Analysis and Characterisation of Noise Additive White Gaussian Noise Model (AWGN) Hidden Markov Model and Gaussian Mixture Models for Noise 49 Bibliography 50 3 Information Theory and Probability Models Introduction: Probability and Information Models Random Processes Information-bearing Random Signals vs Deterministic Signals Pseudo-Random Number Generators (PRNG) Stochastic and Random Processes The Space of Variations of a Random Process Probability Models of Random Signals Probability as a Numerical Mapping of Belief The Choice of One and Zero as the Limits of Probability Discrete, Continuous and Finite-State Probability Models Random Variables and Random Processes Probability and Random Variables The Space and Subspaces of a Variable Probability Mass Function Discrete Random Variables Bayes Rule Probability Density Function Continuous Random Variables Probability Density Functions of Continuous Random Processes Histograms Models of Probability Information Models Entropy: A Measure of Information and Uncertainty Mutual Information 68

11 Contents ix Entropy Coding Variable Length Codes Huffman Coding Stationary and Non-Stationary Random Processes Strict-Sense Stationary Processes Wide-Sense Stationary Processes Non-Stationary Processes Statistics (Expected Values) of a Random Process Central Moments Cumulants The Mean (or Average) Value Correlation, Similarity and Dependency Autocovariance Power Spectral Density Joint Statistical Averages of Two Random Processes Cross-Correlation and Cross-Covariance Cross-Power Spectral Density and Coherence Ergodic Processes and Time-Averaged Statistics Mean-Ergodic Processes Correlation-Ergodic Processes Some Useful Practical Classes of Random Processes Gaussian (Normal) Process Multivariate Gaussian Process Gaussian Mixture Process Binary-State Gaussian Process Poisson Process Counting Process Shot Noise Poisson Gaussian Model for Clutters and Impulsive Noise Markov Processes Markov Chain Processes Homogeneous and Inhomogeneous Markov Chains Gamma Probability Distribution Rayleigh Probability Distribution Chi Distribution Laplacian Probability Distribution Transformation of a Random Process Monotonic Transformation of Random Processes Many-to-One Mapping of Random Signals Search Engines: Citation Ranking Citation Ranking in Web Page Rank Calculation Summary 104 Bibliography Bayesian Inference Bayesian Estimation Theory: Basic Definitions Bayes Theorem Elements of Bayesian Inference Dynamic and Probability Models in Estimation Parameter Space and Signal Space Parameter Estimation and Signal Restoration Performance Measures and Desirable Properties of Estimators Prior and Posterior Spaces and Distributions 114

12 x Contents 4.2 Bayesian Estimation Maximum A Posteriori Estimation Maximum-Likelihood (ML) Estimation Minimum Mean Square Error Estimation Minimum Mean Absolute Value of Error Estimation Equivalence of the MAP, ML, MMSE and MAVE Estimates for Gaussian Processes with Uniform Distributed Parameters Influence of the Prior on Estimation Bias and Variance Relative Importance of the Prior and the Observation Expectation-Maximisation (EM) Method Complete and Incomplete Data Maximisation of Expectation of the Likelihood Function Derivation and Convergence of the EM Algorithm Cramer Rao Bound on the Minimum Estimator Variance Cramer Rao Bound for Random Parameters Cramer Rao Bound for a Vector Parameter Design of Gaussian Mixture Models (GMMs) EM Estimation of Gaussian Mixture Model Bayesian Classification Binary Classification Classification Error Bayesian Classification of Discrete-Valued Parameters Maximum A Posteriori Classification Maximum-Likelihood Classification Minimum Mean Square Error Classification Bayesian Classification of Finite State Processes Bayesian Estimation of the Most Likely State Sequence Modelling the Space of a Random Process Vector Quantisation of a Random Process Vector Quantisation using Gaussian Models of Clusters Design of a Vector Quantiser: K-Means Clustering Summary 145 Bibliography Hidden Markov Models Statistical Models for Non-Stationary Processes Hidden Markov Models Comparison of Markov and Hidden Markov Models Observable-State Markov Process Hidden-State Markov Process A Physical Interpretation: HMMs of Speech Hidden Markov Model as a Bayesian Model Parameters of a Hidden Markov Model State Observation Probability Models State Transition Probabilities State Time Trellis Diagram Training Hidden Markov Models Forward Backward Probability Computation Baum Welch Model Re-estimation Training HMMs with Discrete Density Observation Models 158

13 Contents xi HMMs with Continuous Density Observation Models HMMs with Gaussian Mixture pdfs Decoding Signals Using Hidden Markov Models Viterbi Decoding Algorithm Viterbi Algorithm HMMs in DNA and Protein Sequences HMMs for Modelling Speech and Noise Modelling Speech HMM-Based Estimation of Signals in Noise Signal and Noise Model Combination and Decomposition Hidden Markov Model Combination Decomposition of State Sequences of Signal and Noise HMM-Based Wiener Filters Modelling Noise Characteristics Summary 171 Bibliography Least Square Error Wiener-Kolmogorov Filters Least Square Error Estimation: Wiener-Kolmogorov Filter Derivation of Wiener Filter Equation Calculation of Autocorrelation of Input and Cross-Correlation of Input and Desired Signals Block-Data Formulation of the Wiener Filter QR Decomposition of the Least Square Error Equation Interpretation of Wiener Filter as Projection in Vector Space Analysis of the Least Mean Square Error Signal Formulation of Wiener Filters in the Frequency Domain Some Applications of Wiener Filters Wiener Filter for Additive Noise Reduction Wiener Filter and Separability of Signal and Noise The Square-Root Wiener Filter Wiener Channel Equaliser Time-Alignment of Signals in Multi-channel/Multi-sensor Systems Implementation of Wiener Filters Choice of Wiener Filter Order Improvements to Wiener Filters Summary 191 Bibliography Adaptive Filters: Kalman, RLS, LMS Introduction State-Space Kalman Filters Derivation of Kalman Filter Algorithm Recursive Bayesian Formulation of Kalman Filter Markovian Property of Kalman Filter Comparison of Kalman filter and hidden Markov model Comparison of Kalman and Wiener Filters Extended Kalman Filter (EFK) Unscented Kalman Filter (UFK) Sample Adaptive Filters LMS, RLS 211

14 xii Contents 7.6 Recursive Least Square (RLS) Adaptive Filters Matrix Inversion Lemma Recursive Time-update of Filter Coefficients The Steepest-Descent Method Convergence Rate Vector-Valued Adaptation Step Size Least Mean Squared Error (LMS) Filter Leaky LMS Algorithm Normalised LMS Algorithm Derivation of the Normalised LMS Algorithm Steady-State Error in LMS Summary 223 Bibliography Linear Prediction Models Linear Prediction Coding Predictability, Information and Bandwidth Applications of LP Model in Speech Processing Time-Domain Description of LP Models Frequency Response of LP Model and its Poles Calculation of Linear Predictor Coefficients Effect of Estimation of Correlation Function on LP Model Solution The Inverse Filter: Spectral Whitening, De-correlation The Prediction Error Signal Forward, Backward and Lattice Predictors Augmented Equations for Forward and Backward Predictors Levinson Durbin Recursive Solution Levinson Durbin Algorithm Lattice Predictors Alternative Formulations of Least Square Error Prediction Burg s Method Simultaneous Minimisation of the Backward and Forward Prediction Errors Predictor Model Order Selection Short-Term and Long-Term Predictors MAP Estimation of Predictor Coefficients Probability Density Function of Predictor Output Using the Prior pdf of the Predictor Coefficients Formant-Tracking LP Models Sub-Band Linear Prediction Model Signal Restoration Using Linear Prediction Models Frequency-Domain Signal Restoration Using Prediction Models Implementation of Sub-Band Linear Prediction Wiener Filters Summary 254 Bibliography Eigenvalue Analysis and Principal Component Analysis Introduction Linear Systems and Eigen Analysis A Geometric Interpretation of Eigenvalues and Eigenvectors Eigen Vectors and Eigenvalues Matrix Spectral Theorem Computation of Eigenvalues and Eigen Vectors 263

15 Contents xiii 9.3 Principal Component Analysis (PCA) Computation of PCA PCA Analysis of Images: Eigen-Image Representation PCA Analysis of Speech in White Noise Summary 269 Bibliography Power Spectrum Analysis Power Spectrum and Correlation Fourier Series: Representation of Periodic Signals The Properties of Fourier s Sinusoidal Basis Functions The Basis Functions of Fourier Series Fourier Series Coefficients Fourier Transform: Representation of Non-periodic Signals Discrete Fourier Transform Frequency-Time Resolutions: The Uncertainty Principle Energy-Spectral Density and Power-Spectral Density Non-Parametric Power Spectrum Estimation The Mean and Variance of Periodograms Averaging Periodograms (Bartlett Method) Welch Method: Averaging Periodograms from Overlapped and Windowed Segments Blackman Tukey Method Power Spectrum Estimation from Autocorrelation of Overlapped Segments Model-Based Power Spectrum Estimation Maximum Entropy Spectral Estimation Autoregressive Power Spectrum Estimation Moving-Average Power Spectrum Estimation Autoregressive Moving-Average Power Spectrum Estimation High-Resolution Spectral Estimation Based on Subspace Eigen-Analysis Pisarenko Harmonic Decomposition Multiple Signal Classification (MUSIC) Spectral Estimation Estimation of Signal Parameters via Rotational Invariance Techniques (ESPRIT) Summary 293 Bibliography Interpolation Replacement of Lost Samples Introduction Ideal Interpolation of a Sampled Signal Digital Interpolation by a Factor of I Interpolation of a Sequence of Lost Samples The Factors That Affect Interpolation Accuracy Polynomial Interpolation Lagrange Polynomial Interpolation Newton Polynomial Interpolation Hermite Polynomial Interpolation Cubic Spline Interpolation Model-Based Interpolation Maximum A Posteriori Interpolation 307

16 xiv Contents Least Square Error Autoregressive Interpolation Interpolation Based on a Short-Term Prediction Model Interpolation Based on Long-Term and Short-term Correlations LSAR Interpolation Error Interpolation in Frequency Time Domain Interpolation Using Adaptive Code Books Interpolation Through Signal Substitution LP-HNM Model based Interpolation Summary 319 Bibliography Signal Enhancement via Spectral Amplitude Estimation Introduction Spectral Representation of Noisy Signals Vector Representation of Spectrum of Noisy Signals Spectral Subtraction Power Spectrum Subtraction Magnitude Spectrum Subtraction Spectral Subtraction Filter: Relation to Wiener Filters Processing Distortions Effect of Spectral Subtraction on Signal Distribution Reducing the Noise Variance Filtering Out the Processing Distortions Non-Linear Spectral Subtraction Implementation of Spectral Subtraction Bayesian MMSE Spectral Amplitude Estimation Estimation of Signal to Noise Ratios Application to Speech Restoration and Recognition Summary 338 Bibliography Impulsive Noise: Modelling, Detection and Removal Impulsive Noise Definition of a Theoretical Impulse Function The Shape of a Real Impulse in a Communication System The Response of a Communication System to an Impulse The Choice of Time or Frequency Domain for Processing of Signals Degraded by Impulsive Noise Autocorrelation and Power Spectrum of Impulsive Noise Probability Models of Impulsive Noise Bernoulli Gaussian Model of Impulsive Noise Poisson Gaussian Model of Impulsive Noise A Binary-State Model of Impulsive Noise Hidden Markov Model of Impulsive and Burst Noise Impulsive Noise Contamination, Signal to Impulsive Noise Ratio Median Filters for Removal of Impulsive Noise Impulsive Noise Removal Using Linear Prediction Models Impulsive Noise Detection Analysis of Improvement in Noise Detectability Two-Sided Predictor for Impulsive Noise Detection Interpolation of Discarded Samples 355

17 Contents xv 13.7 Robust Parameter Estimation Restoration of Archived Gramophone Records Summary 358 Bibliography Transient Noise Pulses Transient Noise Waveforms Transient Noise Pulse Models Noise Pulse Templates Autoregressive Model of Transient Noise Pulses Hidden Markov Model of a Noise Pulse Process Detection of Noise Pulses Matched Filter for Noise Pulse Detection Noise Detection Based on Inverse Filtering Noise Detection Based on HMM Removal of Noise Pulse Distortions Adaptive Subtraction of Noise Pulses AR-based Restoration of Signals Distorted by Noise Pulses Summary 369 Bibliography Echo Cancellation Introduction: Acoustic and Hybrid Echo Echo Return Time: The Sources of Delay in Communication Networks Transmission link (electromagnetic wave propagation) delay Speech coding/decoding delay Network processing delay De-Jitter delay Acoustic echo delay Telephone Line Hybrid Echo Echo Return Loss Hybrid (Telephone Line) Echo Suppression Adaptive Echo Cancellation Echo Canceller Adaptation Methods Convergence of Line Echo Canceller Echo Cancellation for Digital Data Transmission Acoustic Echo Sub-Band Acoustic Echo Cancellation Echo Cancellation with Linear Prediction Pre-whitening Multi-Input Multi-Output Echo Cancellation Stereophonic Echo Cancellation Systems Non-uniqueness Problem in MIMO Echo Channel Identification MIMO In-Cabin Communication Systems Summary 389 Bibliography Channel Equalisation and Blind Deconvolution Introduction The Ideal Inverse Channel Filter Equalisation Error, Convolutional Noise Blind Equalisation 394

18 xvi Contents Minimum- and Maximum-Phase Channels Wiener Equaliser Blind Equalisation Using Channel Input Power Spectrum Homomorphic Equalisation Homomorphic Equalisation Using a Bank of High-Pass Filters Equalisation Based on Linear Prediction Models Blind Equalisation Through Model Factorisation Bayesian Blind Deconvolution and Equalisation Conditional Mean Channel Estimation Maximum-Likelihood Channel Estimation Maximum A Posteriori Channel Estimation Channel Equalisation Based on Hidden Markov Models MAP Channel Estimate Based on HMMs Implementations of HMM-Based Deconvolution Blind Equalisation for Digital Communication Channels LMS Blind Equalisation Equalisation of a Binary Digital Channel Equalisation Based on Higher-Order Statistics Higher-Order Moments, Cumulants and Spectra Cumulants Higher-Order Spectra Higher-Order Spectra of Linear Time-Invariant Systems Blind Equalisation Based on Higher-Order Cepstra Bi-Cepstrum Tri-Cepstrum Calculation of Equaliser Coefficients from the Tri-cepstrum Summary 420 Bibliography Speech Enhancement: Noise Reduction, Bandwidth Extension and Packet Replacement An Overview of Speech Enhancement in Noise Single-Input Speech Enhancement Methods Elements of Single-Input Speech Enhancement Segmentation and Windowing of Speech Signals Spectral Representation of Speech and Noise Linear Prediction Model Representation of Speech and Noise Inter-Frame and Intra-Frame Correlations Speech Estimation Module Probability Models of Speech and Noise Cost of Error Functions in Speech Estimation Wiener Filter for De-noising Speech Wiener Filter Based on Linear Prediction Models HMM-Based Wiener Filters Spectral Subtraction of Noise Spectral Subtraction Using LP Model Frequency Response Bayesian MMSE Speech Enhancement Kalman Filter for Speech Enhancement Kalman State-Space Equations of Signal and Noise Models 433

19 Contents xvii Speech Enhancement Using LP-HNM Model Overview of LP-HNM Enhancement System Formant Estimation from Noisy Speech Initial-Cleaning of Noisy Speech Formant Tracking Harmonic Plus Noise Model (HNM) of Speech Excitation Fundamental Frequency Estimation Estimation of Amplitudes Harmonics of HNM Estimation of Noise Component of HNM Kalman Smoothing of Trajectories of Formants and Harmonics Speech Bandwidth Extension Spectral Extrapolation LP-HNM Model of Speech Extrapolation of Spectral Envelope of LP Model Phase Estimation Codebook Mapping of the Gain Extrapolation of Spectrum of Excitation of LP Model Sensitivity to Pitch Interpolation of Lost Speech Segments Packet Loss Concealment Phase Prediction Codebook Mapping Evaluation of LP-HNM Interpolation Multi-Input Speech Enhancement Methods Beam-forming with Microphone Arrays Spatial Configuration of Array and The Direction of Reception Directional of Arrival (DoA) and Time of Arrival (ToA) Steering the Array Direction: Equalisation of the ToAs at the Sensors The Frequency Response of a Delay-Sum Beamformer Speech Distortion Measurements Signal-to-Noise Ratio SNR Segmental Signal to Noise Ratio SNR seg Itakura Saito Distance ISD Harmonicity Distance HD Diagnostic Rhyme Test DRT Mean Opinion Score MOS Perceptual Evaluation of Speech Quality PESQ 464 Bibliography Multiple-Input Multiple-Output Systems, Independent Component Analysis Introduction A note on comparison of beam-forming arrays and ICA MIMO Signal Propagation and Mixing Models Instantaneous Mixing Models Anechoic, Delay and Attenuation, Mixing Models Convolutional Mixing Models Independent Component Analysis A Note on Orthogonal, Orthonormal and Independent Statement of ICA Problem Basic Assumptions in Independent Component Analysis The Limitations of Independent Component Analysis 475

20 xviii Contents Why a mixture of two Gaussian signals cannot be separated? The Difference Between Independent and Uncorrelated Independence Measures; Entropy and Mutual Information Differential Entropy Maximum Value of Differential Entropy Mutual Information The Effect of a Linear Transformation on Mutual Information Non-Gaussianity as a Measure of Independence Negentropy: A measure of Non-Gaussianity and Independence Fourth Order Moments Kurtosis Kurtosis-based Contrast Functions Approximations to Entropic Contrast Super-Gaussian and Sub-Gaussian Distributions Fast-ICA Methods Gradient search optimisation method Newton optimisation method Fixed-point Fast ICA Contrast Functions and Influence Functions ICA Based on Kurtosis Maximization Projection Pursuit Gradient Ascent Jade Algorithm Iterative Diagonalisation of Cumulant Matrices Summary 490 Bibliography Signal Processing in Mobile Communication Introduction to Cellular Communication A Brief History of Radio Communication Cellular Mobile Phone Concept Outline of a Cellular Communication System Communication Signal Processing in Mobile Systems Capacity, Noise, and Spectral Efficiency Spectral Efficiency in Mobile Communication Systems Multi-path and Fading in Mobile Communication Multi-path Propagation of Electromagnetic Signals Rake Receivers for Multi-path Signals Signal Fading in Mobile Communication Systems Large-Scale Signal Fading Small-Scale Fast Signal Fading Smart Antennas Space Time Signal Processing Switched and Adaptive Smart Antennas Space Time Signal Processing Diversity Schemes Summary 508 Bibliography 508 Index 509

21 Preface Since the publication of the first edition of this book in 1996, digital signal processing (DSP) in general and noise reduction in particular, have become even more central to the research and development of efficient, adaptive and intelligent mobile communication and information processing systems. The fourth edition of this book has been revised extensively and improved in several ways to take account of the recent advances in theory and application of digital signal processing. The existing chapters have been updated with new materials added. Two new chapters have been introduced; one on eigen analysis and principal component analysis and the other on multiple-input multiple-output (MIMO) systems and independent component analysis. In addition the speech enhancement section has been substantially expanded to include bandwidth extension and packet loss replacement. The applications of DSP are numerous and include multimedia technology, audio signal processing, video signal processing, cellular mobile communication, voice over IP (VoIP), adaptive network management, radar systems, pattern analysis, pattern recognition, medical signal processing, financial data forecasting, artificial intelligence, decision making systems, control systems and information search engines. The theory and application of signal processing is concerned with the identification, modelling and utilisation of patterns and structures in a signal process. The observation signals are often distorted, incomplete and noisy. Hence, noise reduction and the removal of channel distortion and interference are important parts of a signal processing system. The aim of this book is to provide a coherent and structured presentation of the theory and applications of statistical signal processing and noise reduction methods and is organised in 19 chapters. Chapter 1 begins with an introduction to signal processing, and provides a brief review of signal processing methodologies and applications. The basic operations of sampling and quantisation are reviewed in this chapter. Chapter 2 provides an introduction to noise and distortion. Several different types of noise, including thermal noise, shot noise, burst noise, impulsive noise, flicker noise, acoustic noise, electromagnetic noise and channel distortions, are considered. The chapter concludes with an introduction to the modelling of noise processes. Chapter 3 provides an introduction to the theory and applications of probability models and stochastic signal processing. The chapter begins with an introduction to random signals, stochastic processes, probabilistic models and statistical measures. The concepts of stationary, non-stationary and ergodic processes are introduced in this chapter, and some important classes of random processes, such as Gaussian, mixture Gaussian, Markov chains and Poisson processes, are considered. The effects of transformation of a signal on its statistical distribution are considered. Chapter 4 is on Bayesian estimation and classification. In this chapter the estimation problem is formulated within the general framework of Bayesian inference. The chapter includes Bayesian theory, classical estimators, the estimate maximise method, the Cramér Rao bound on the minimum variance estimate, Bayesian classification, and the modelling of the space of a random signal. This chapter provides a number of examples on Bayesian estimation of signals observed in noise.

22 xx Preface Chapter 5 considers hidden Markov models (HMMs) for non-stationary signals. The chapter begins with an introduction to the modelling of non-stationary signals and then concentrates on the theory and applications of hidden Markov models. The hidden Markov model is introduced as a Bayesian model, and methods of training HMMs and using them for decoding and classification are considered. The chapter also includes the application of HMMs in noise reduction. Chapter 6 considers Wiener Filters. The least square error filter is formulated first through minimisation of the expectation of the squared error function over the space of the error signal. Then a block-signal formulation of Wiener filters and a vector space interpretation of Wiener filters are considered. The frequency response of the Wiener filter is derived through minimisation of mean square error in the frequency domain. Some applications of the Wiener filter are considered, and a case study of the Wiener filter for removal of additive noise provides useful insight into the operation of the filter. Chapter 7 considers adaptive filters. The chapter begins with the state-space equation for Kalman filters. The optimal filter coefficients are derived using the principle of orthogonality of the innovation signal. The nonlinear versions of Kalman filter namely extended Kalman and unscented Kalman filters are also considered. The recursive least squared (RLS) filter, which is an exact sample-adaptive implementation of the Wiener filter, is derived in this chapter. Then the steepest descent search method for the optimal filter is introduced. The chapter concludes with a study of the LMS adaptive filters. Chapter 8 considers linear prediction and sub-band linear prediction models. Forward prediction, backward prediction and lattice predictors are studied. This chapter introduces a modified predictor for the modelling of the short term and the pitch period correlation structures. A maximum a posteriori (MAP) estimate of a predictor model that includes the prior probability density function of the predictor is introduced. This chapter concludes with the application of linear prediction in signal restoration. Chapter 9 considers eigen analysis and principal component analysis. Eigen analysis is used in applications such as the diagonalisation of correlation matrices, adaptive filtering, radar signal processing, feature extraction, pattern recognition, signal coding, model order estimation, noise estimation, and separation of mixed biomedical or communication signals. A major application of eigen analysis is in analysis of the covariance matrix of a signal a process known as the principal component analysis (PCA). PCA is widely used for feature extraction and dimension reduction. Chapter 10 considers frequency analysis and power spectrum estimation. The chapter begins with an introduction to the Fourier transform, and the role of the power spectrum in identification of patterns and structures in a signal process. The chapter considers non parametric spectral estimation, model-based spectral estimation, the maximum entropy method, and high resolution spectral estimation based on eigenanalysis. Chapter 11 considers interpolation of a sequence of unknown samples. This chapter begins with a study of the ideal interpolation of a band-limited signal, a simple model for the effects of a number of missing samples, and the factors that affect interpolation. Interpolators are divided into two categories: polynomial and statistical interpolators. A general form of polynomial interpolation as well as its special forms (Lagrange, Newton, Hermite and cubic spline interpolators) is considered. Statistical interpolators in this chapter include maximum a posteriori interpolation, least squared error interpolation based on an autoregressive model, time frequency interpolation, and interpolation through search of an adaptive codebook for the best signal. Chapter 12 considers spectral amplitude estimation. A general form of spectral subtraction is formulated and the processing distortions that result form spectral subtraction are considered. The effects of processing-distortions on the distribution of a signal are illustrated. The chapter considers methods for removal of the distortions and also non-linear methods of spectral subtraction. This chapter also covers the Bayesian minimum mean squared error method of spectral amplitude estimation. This chapter concludes with an implementation of spectral subtraction for signal restoration. Chapters 13 and 14 cover the modelling, detection and removal of impulsive noise and transient noise pulses. In Chapter 12, impulsive noise is modelled as a binary state non-stationary process and several stochastic models for impulsive noise are considered. For removal of impulsive noise, median filters

23 Preface xxi and a method based on a linear prediction model of the signal process are considered. The materials in Chapter 13 closely follow Chapter 12. In Chapter 13, a template-based method, an HMM-based method and an AR model-based method for removal of transient noise are considered. Chapter 15 covers echo cancellation. The chapter begins with an introduction to telephone line echoes, and considers line echo suppression and adaptive line echo cancellation. Then the problem of acoustic echoes and acoustic coupling between loudspeaker and microphone systems are considered. The chapter concludes with a study of a sub-band echo cancellation system. Chapter 16 is on blind deconvolution and channel equalisation. This chapter begins with an introduction to channel distortion models and the ideal channel equaliser. Then the Wiener equaliser, blind equalisation using the channel input power spectrum, blind deconvolution based on linear predictive models, Bayesian channel equalisation, and blind equalisation for digital communication channels are considered. The chapter concludes with equalisation of maximum phase channels using higher-order statistics. Chapter 17 is on speech enhancement methods. Speech enhancement in noisy environments improves the quality and intelligibility of speech for human communication and increases the accuracy of automatic speech recognition systems. Noise reduction systems are increasingly important in a range of applications such as mobile phones, hands-free phones, teleconferencing systems and in-car cabin communication systems. This chapter covers the three main areas of noise reduction, bandwidth extension and replacement of missing speech segments. This chapter concludes with microphone array beam-forming for speech enhancement in noise. Chapter 18 introduces multiple-input multiple-output (MIMO) systems and consider independent component analysis (ICA) for separation of signals in MIMO systems. MIMO signal processing systems are employed in a wide range of applications including multi-sensors biological signal processing systems, phased-array radars, steerable directional antenna arrays for mobile phone systems, microphone arrays for speech enhancement, multichannel audio entertainment systems. Chapter 19 covers the issue of noise in wireless communication. Noise, fading and limited radio bandwidth are the main factors that constrain the capacity and the speed of communication on wireless channels. Research and development of communication systems aim to increase the spectral efficiency defined as the data bits per second per Hz bandwidth of a communication channel. For improved efficiency modern mobile communication systems rely on signal processing methods at almost every stage from source coding to the allocation of time bandwidth and space resources. In this chapter we consider how communication signal processing methods are employed for improving the speed and capacity of communication systems. Saeed V. Vaseghi July 2008

24

25 Acknowledgements I wish to thank Ales Prochazka, Esi Zavarehei, Ben Milner, Qin Yan, Dimitrios Rentzos, Charles Ho and Aimin Chen. Many thanks also to the publishing team at John Wiley, Sarah Hinton, Mark Hammond, Sarah Tilley, and Katharine Unwin.

26

27 Symbols A Matrix of predictor coefficients a k Linear predictor coefficients a Linear predictor coefficients vector a ij Probability of transition from state i to state j in a Markov model α i (t) Forward probability in an HMM b(m) Backward prediction error b(m) Binary state signal β i (t) Backward probability in an HMM c xx (m) Covariance of signal x(m) c XX (k 1, k 2,, k N ) k th order cumulant of x(m) C XX (ω 1, ω 2,, ω k 1 ) k th order cumulant spectra of x(m) D Diagonal matrix e(m) Estimation error E[x] Expectation of x f Frequency variable Fs Sampling frequency f X (x) Probability density function for process X f X,Y (x,y) Joint probability density function of X and Y f X Y (x y ) Probability density function of X conditioned on Y f X;Θ (x; θ) Probability density function of X with θ as a parameter f X S,M (x s,m) Probability density function of X given a state sequence s of an HMM M of the process X Φ(m, m 1) State transition matrix in Kalman filter G Filter gain factor h Filter coefficient vector, Channel response h max Maximum phase channel response h min Minimum phase channel response h inv Inverse channel response H( f ) Channel frequency response H inv ( f ) Inverse channel frequency response H Observation matrix, Distortion matrix I Identity matrix J Fisher s information matrix J Jacobian of a transformation K(m) Kalman gain matrix

28 xxvi Symbols λ Eigenvalue Λ Diagonal matrix of eigenvalues m Discrete time index m k k th order moment M A model, e.g. an HMM μ Adaptation convergence factor μ x Expected mean of vector x n(m) Noise n(m) A noise vector of N samples n i (m) Impulsive noise N( f ) Noise spectrum N ( f ) Complex conjugate of N( f ) N( f ) Time-averaged noise spectrum N (x, μ xx, Σ xx ) A Gaussian pdf with mean vector μ xx and covariance matrix Σ xx O( ) In the order of ( ) P Filter order (length) P X (x i ) Probability mass function of x i P X,Y (x ( i, y j ) ) P X Y xi yj Joint probability mass function of x i and y j Conditional probability mass function of x i given y j P NN ( f ) Power spectrum of noise n(m) P XX ( f ) Power spectrum of the signal x(m) P XY ( f ) Cross power spectrum of signals x(m) and y(m) θ Parameter vector ˆθ Estimate of the parameter vector θ r k Reflection coefficients r xx (m) Autocorrelation function r xx (m) Autocorrelation vector R xx Autocorrelation matrix of signal x(m) R xy Cross correlation matrix s State sequence s ML Maximum likelihood state sequence σ 2 n Variance of noise n(m) Σ nn Covariance matrix of noise n(m) Σ xx Covariance matrix of signal x(m) σ 2 x Variance of signal x(m) σ 2 n Variance of noise n(m) x(m) Clean signal ˆx(m) Estimate of clean signal x(m) Clean signal vector X( f ) Frequency spectrum of signal x(m) X ( f ) Complex conjugate of X( f ) X( f ) Time-averaged frequency spectrum bof the signal x(m) X(f, t) Time-frequency spectrum of the signal x(m) X Clean signal matrix X H Hermitian transpose of X y(m) Noisy signal y(m) Noisy signal vector ŷ (m m i ) Prediction of y(m) based on observations up to time m i Y Noisy signal matrix

29 Symbols xxvii Y H Hermitian transpose of Y Var Variance w k Wiener filter coefficients w(m) Wiener filter coefficients vector W( f ) Wiener filter frequency response z z-transform variable

30