Advanced Digital Signal Processing and Noise Reduction

Transcription

1 Advanced Digital Signal Processing and Noise Reduction Fourth Edition Professor Saeed V. Vaseghi Professor of Communications and Signal Processing Department of Electronics & Computer Engineering Brunei University, London, UK WILEY A John Wiley and Sons, Ltd, Publication

2 Preface Acknowledgements Symbols Abbreviations xix xxiii xxv xxix 1 Introduction Signals, Noise and Information Signal Processing Methods Transform-Based Signal Processing Source-Filter Model-Based Signal Processing Bayesian Statistical Model-Based Signal Processing Neural Networks Applications of Digital Signal Processing Digital Watermarking Bio-medical, M1MO, Signal Processing Echo Cancellation Adaptive Noise Cancellation Adaptive Noise Reduction Blind Channel Equalisation Signal Classification and Pattern Recognition Linear Prediction Modelling of Speech Digital Coding of Audio Signals Detection of Signals in Noise Directional Reception of Waves: Beam-forming Space-Time Signal Processing Dolby Noise Reduction Radar Signal Processing: Doppler Frequency Shift A Review of Sampling and Quantisation Advantages of Digital Format Digital Signals Stored and Transmitted in Analogue Format The Effect of Digitisation on Signal Bandwidth Sampling a Continuous-Time Signal Aliasing Distortion Nyquist Sampling Theorem 27

3 1.4.7 Quantisation Non-Linear Quantisation, Companding Summary 32 Bibliography 32 Noise and Distortion Introduction Different Classes of Noise Sources and Distortions /.2 Different Classes and Spectral/Temporal Shapes of Noise White Noise Band-Limited White Noise Coloured Noise; Pink Noise and Brown Noise Impulsive and Click Noise Transient Noise Pulses Thermal Noise Shot Noise Flicker (I//) Noise Burst Noise Electromagnetic (Radio) Noise Natural Sources of Radiation of Electromagnetic Noise Man-made Sources of Radiation of Electromagnetic Noise Channel Distortions Echo and Multi-path Reflections Modelling Noise Frequency Analysis and Characterisation of Noise Additive White Gaussian Noise Model (AWGN) Hidden Markov Model and Gaussian Mixture Models for Noise 49 Bibliography 50 Information Theory and Probability Models Introduction: Probability and Information Models Random Processes Information-bearing Random Signals vs Deterministic Signals Pseudo-Random Number Generators (PRNG) Stochastic and Random Processes The Space of Variations of a Random Process Probability Models of Random Signals Probability as a Numerical Mapping of Belief The Choice of One and Zero as the Limits of Probability Discrete, Continuous and Finite-State Probability Models Random Variables and Random Processes Probability and Random Variables - The Space and Suhspaces of a Variable Probability Mass Function-Discrete Random Variables Bay es'rule Probability Density Function - Continuous Random Variables Probability Density Functions of Continuous Random Processes Histograms - Models of Probability Information Models Entropy: A Measure of Information and Uncertainty Mutual Information 68

4 3.4.3 Entropy Coding - Variable Length Codes Huffman Coding Stationary and Non-Stationary Random Processes Strict-Sense Stationary Processes Wide-Sense Stationary Processes Non-Stationary Processes Statistics (Expected Values) of a Random Process Central Moments Cumulants The Mean (or Average) Value Correlation, Similarity and Dependency Autocovariance Power Spectral Density Joint Statistical Averages of Two Random Processes Cross-Correlation and Cross-Covariance Cross-Power Spectral Density and Coherence Ergodic Processes and Time-Averaged Statistics Mean-Ergodic Processes Correlation-Ergodic Processes Some Useful Practical Classes of Random Processes Gaussian (Normal) Process Multivariate Gaussian Process Gaussian Mixture Process Binary-State Gaussian Process Poisson Process - Counting Process Shot Noise Poisson-Gaussian Model for Clutters and Impulsive Noise Markov Processes Markov Chain Processes Homogeneous andinhomogeneous Markov Chains Gamma Probability Distribution Rayleigh Probability Distribution Chi Distribution Laplacian Probability Distribution Transformation of a Random Process Monotonie Transformation of Random Processes Many-to-One Mapping of Random Signals Search Engines: Citation Ranking Citation Ranking in Web Page Rank Calculation Summary 104 Bibliography 105 Bayesian Inference Bayesian Estimation Theory: Basic Definitions Bayes' Theorem Elements of Bayesian Inference Dynamic and Probability Models in Estimation Parameter Space and Signal Space Parameter Estimation and Signal Restoration Performance Measures and Desirable Properties of Estimators Prior and Posterior Spaces and Distributions 114

5 4.2 Bayesian Estimation Maximum A Posteriori Estimation Maximum-Likelihood (ML) Estimation Minimum Mean Square Error Estimation Minimum Mean Absolute Value of Error Estimation Equivalence of the MAP, ML, MMSE and MAVE Estimates for Gaussian Processes with Uniform Distributed Parameters Influence of the Prior on Estimation Bias and Variance Relative Importance of the Prior and the Observation Expectation-Maximisation (EM) Method Complete and Incomplete Data Maximisation of Expectation of the Likelihood Function Derivation and Convergence of the EM Algorithm Cramer-Rao Bound on the Minimum Estimator Variance Cramer-Rao Bound for Random Parameters Cramer-Rao Bound for a Vector Parameter Design of Gaussian Mixture Models (GMMs) EM Estimation of Gaussian Mixture Model Bayesian Classification Binary Classification Classification Error Bayesian Classification of Discrete-Valued Parameters Maximum A Posteriori Classification Maximum-Likelihood Classification Minimum Mean Square Error Classification Bayesian Classification of Finite State Processes Bayesian Estimation of the Most Likely State Sequence Modelling the Space of a Random Process Vector Quantisation of a Random Process Vector Quantisation using Gaussian Models of Clusters Design of a Vector Quantiser: K-Means Clustering Summary 145 Bibliography 146 Hidden Markov Models Statistical Models for Non-Stationary Processes Hidden Markov Models Comparison of Markov and Hidden Markov Models Observable-State Markov Process Hidden-State Markov Process A Physical Interpretation: HMMs of Speech Hidden Markov Model as a Bayesian Model Parameters of a Hidden Markov Model State Observation Probability Models State Transition Probabilities State-Time Trellis Diagram Training Hidden Markov Models Forward-Backward Probability Computation Baum-Weich Model Re-estimation Training HMMs with Discrete Density Observation Models 158

6 5.3.4 HMMs with Continuous Density Observation Models HMMs with Gaussian Mixture pdfs Decoding Signals Using Hidden Markov Models I Viterbi Decoding Algorithm Viterbi Algorithm HMMs in DNA and Protein Sequences HMMs for Modelling Speech and Noise Modelling Speech HMM-Based Estimation of Signals in Noise Signal and Noise Model Combination and Decomposition Hidden Markov Model Combination Decomposition of State Sequences of Signal and Noise HMM-Based Wiener Filters Modelling Noise Characteristics Summary 171 Bibliography 171 Least Square Error Wiener-Kolmogorov Filters Least Square Error Estimation: Wiener-Kolmogorov Filter /. 1 Derivation of Wiener Filter Equation Calculation ofautocorrelation of Input and Cross-Correlation of Input and Desired Signals Block-Data Formulation of the Wiener Filter QR Decomposition of the Least Square Error Equation Interpretation of Wiener Filter as Projection in Vector Space Analysis of the Least Mean Square Error Signal Formulation of Wiener Filters in the Frequency Domain Some Applications of Wiener Filters Wiener Filter for Additive Noise Reduction Wiener Filter and Separability of Signal and Noise The Square-Root Wiener Filter Wiener Channel Equaliser Time-Alignment of Signals in Multi-channel/Multi-sensor Systems Implementation of Wiener Filters Choice of Wiener Filter Order Improvements to Wiener Filters Summary 191 Bibliography 191 Adaptive Filters: Kaiman, RLS, LMS Introduction State-Space Kaiman Filters Derivation of Kaiman Filter Algorithm Recursive Bayesian Formulation of Kaiman Filter Markovian Property of Kaiman Filter Comparison of Kaiman filter and hidden Markov model Comparison of Kaiman and Wiener Filters Extended Kaiman Filter (EFK) Unscented Kaiman Filter (UFK) Sample Adaptive Filters - LMS, RLS 211

7 7.6 Recursive Least Square (RLS) Adaptive Filters Matrix Inversion Lemma Recursive Time-update of Filter Coefficients The Steepest-Descent Method Convergence Rate Vector-Valued Adaptation Step Size Least Mean Squared Error (LMS) Filter Leaky LMS Algorithm Normalised LMS Algorithm Derivation of the Normalised LMS Algorithm Steady-State Error in LMS Summary 223 Bibliography 224 Linear Prediction Models Linear Prediction Coding Predictability, Information and Bandwidth Applications of LP Model in Speech Processing Time-Domain Description of LP Models Frequency Response of LP Model and its Poles Calculation of Linear Predictor Coefficients Effect of Estimation of Correlation Function on LP Model Solution The Inverse Filter: Spectral Whitening, De-correlation The Prediction Error Signal Forward, Backward and Lattice Predictors Augmented Equations for Forward and Backward Predictors Levinson-Durbin Recursive Solution Levinson-Durbin Algorithm Lattice Predictors Alternative Formulations of Least Square Error Prediction Burg's Method Simultaneous Minimisation of the Backward and Forward Prediction Errors Predictor Model Order Selection Short-Term and Long-Term Predictors MAP Estimation of Predictor Coefficients Probability Density Function of Predictor Output Using the Prior pdf of the Predictor Coefficients Formant-Tracking LP Models Sub-Band Linear Prediction Model Signal Restoration Using Linear Prediction Models Frequency-Domain Signal Restoration Using Prediction Models Implementation of Sub-Band Linear Prediction Wiener Filters Summary 254 Bibliography 254 Eigenvalue Analysis and Principal Component Analysis Introduction - Linear Systems and Eigen Analysis A Geometric Interpretation of Eigenvalues and Eigenvectors Eigen Vectors and Eigenvalues Matrix Spectral Theorem Computation of Eigenvalues and Eigen Vectors 263

8 9.3 Principal Component Analysis (PCA) Computation of PCA PCA Analysis of Images: Eigen-Image Representation PCA Analysis of Speech in White Noise Summary 269 Bibliography 270 Power Spectrum Analysis Power Spectrum and Correlation Fourier Series: Representation of Periodic Signals The Properties of Fourier's Sinusoidal Basis Functions The Basis Functions of Fourier Series Fourier Series Coefficients 21A 10.3 Fourier Transform: Representation of Non-periodic Signals Discrete Fourier Transform lid Frequency-Time Resolutions: The Uncertainty Principle Energy-Spectral Density and Power-Spectral Density Non-Parametric Power Spectrum Estimation The Mean and Variance of Periodograms Averaging Periodograms (Bartlett Method) Welch Method: Averaging Periodograms from Overlapped and Windowed Segments Blackman-Tukey Method Power Spectrum Estimation from Autocorrelation of Overlapped Segments Model-Based Power Spectrum Estimation Maximum-Entropy Spectral Estimation Autoregressive Power Spectrum Estimation Moving-Average Power Spectrum Estimation Autoregressive Moving-Average Power Spectrum Estimation High-Resolution Spectral Estimation Based on Subspace Eigen-Analysis Pisarenko Harmonic Decomposition Multiple Signed Classification (MUSIC) Spectral Estimation Estimation of Signal Parameters via Rotational Invariance Techniques (ESPRIT) Summary 293 Bibliography 293 Interpolation - Replacement of Lost Samples Introduction Ideal Interpolation of a Sampled Signal Digital Interpolation by a Factor of I Interpolation of a Sequence of Lost Samples The Factors That Affect Interpolation Accuracy Polynomial Interpolation Lagrange Polynomial Interpolation Newton Polynomial Interpolation Hermite Polynomial Interpolation Cubic Spline Interpolation Model-Based Interpolation Maximum A Posteriori Interpolation 307

9 /1.3.2 Least Square Error Autoregressive Interpolation Interpolation Based on a Short-Term Prediction Model Interpolation Based on Long-Term and Short-term Correlations LSAR Interpolation Error Interpolation in Frequency-Time Domain Interpolation Using Adaptive Code Books Interpolation Through Signal Substitution 318 /1.3.9 LP-HNM Model based Interpolation Summary 319 Bibliography Signal Enhancement via Spectral Amplitude Estimation Introduetion Spectral Representation of Noisy Signals Vector Representation of Spectrum of Noisy Signals Spectral Subtraction Power Spectrum Subtraction Magnitude Spectrum Subtraction Spectral Subtraction Filter: Relation to Wiener Filters Processing Distortions Effect of Spectral Subtraction on Signal Distribution Reducing the Noise Variance Filtering Out the Processing Distortions Non-Linear Spectral Subtraction Implementation of Spectral Subtraction Bayesian MMSE Spectral Amplitude Estimation Estimation of Signal to Noise Ratios Application to Speech Restoration and Recognition Summary 338 Bibliography Impulsive Noise: Modelling, Detection and Removal Impulsive Noise Definition of a Theoretical Impulse Function The Shape of a Real Impulse in a Communication System The Response of a Communication System to an Impulse The Choice of Time or Frequency Domain for Processing of Signals Degraded by Impulsive Noise Autocorrelation and Power Spectrum of Impulsive Noise Probability Models of Impulsive Noise Bernoulli-Gaussian Model of Impulsive Noise Poisson Gaussian Model of Impulsive Noise A Binary-State Model of Impulsive Noise Hidden Markov Model of Impulsive and Burst Noise Impulsive Noise Contamination, Signal to Impulsive Noise Ratio Median Filters for Removal of Impulsive Noise Impulsive Noise Removal Using Linear Prediction Models Impulsive Noise Detection Analysis of Improvement in Noise Detectability Two-Sided Predictor for Impulsive Noise Detection Interpolation of Discarded Samples 355

10 13.7 Robust Parameter Estimation Restoration of Archived Gramophone Records Summary 358 Bibliography Transient Noise Pulses Transient Noise Waveforms Transient Noise Pulse Models Noise Pulse Templates Autoregressive Model of Transient Noise Pulses Hidden Markov Model of a Noise Pulse Process Detection of Noise Pulses Matched Filter for Noise Pulse Detection Noise Detection Based on Inverse Filtering Noise Detection Based on HMM Removal of Noise Pulse Distortions Adaptive Subtraction of Noise Pulses AR-based Restoration of Signals Distorted by Noise Pulses Summary 369 Bibliography Echo Cancellation Introduction: Acoustic and Hybrid Echo Echo Return Time: The Sources of Delay in Communication Networks Transmission link (electromagnetic wave propagation) delay Speech coding/decoding delay Network processing delay De-Jitter delay Acoustic echo delay Telephone Line Hybrid Echo Echo Return Loss Hybrid (Telephone Line) Echo Suppression Adaptive Echo Cancellation Echo Canceller Adaptation Methods Convergence of Line Echo Canceller 380 /5.5.3 Echo Cancellation for Digital Data Transmission Acoustic Echo Sub-Band Acoustic Echo Cancellation Echo Cancellation with Linear Prediction Pre-whitening Multi-Input Multi-Output Echo Cancellation Stereophonic Echo Cancellation Systems Non-uniqueness Problem in MIMO Echo Channel Identification MIMO ln-cabin Communication Systems Summary 389 Bibliography Channel Equalisation and Blind Deconvolution Introduction The Ideal Inverse Channel Filter Equalisation Error, Convolutional Noise Blind Equalisation 394

11 Minimum-and Maximum-Phase Channels Wiener Equaliser Blind Equalisation Using Channel Input Power Spectrum Homomorphic Equalisation Homomorphic Equalisation Using a Bank of High-Pass Filters Equalisation Based on Linear Prediction Models Blind Equalisation Through Model Factorisation Bayesian Blind Deconvolution and Equalisation Conditional Mean Channel Estimation Maximum-Likelihood Channel Estimation Maximum A Posteriori Channel Estimation Channel Equalisation Based on Hidden Markov Models MAP Channel Estimate Based on HMMs Implementations of HMM-Based Deconvolution Blind Equalisation for Digital Communication Channels LMS Blind Equalisation Equalisation of a Binary Digital Channel Equalisation Based on Higher-Order Statistics Higher-Order Moments, Cumulants and Spectra Cumulants Higher-Order Spectra Higher-Order Spectra of Linear Time-Invariant Systems Blind Equalisation Based on Higher-Order Cepstra Bi-Cepstrum Tri-Cepstrum Calculation of Equaliser Coefficients from the Tri-cepstrum Summary 420 Bibliography 421 Speech Enhancement: Noise Reduction, Bandwidth Extension and Packet Replacement An Overview of Speech Enhancement in Noise Single-Input Speech Enhancement Methods Elements of Single-Input Speech Enhancement Segmentation and Windowing of Speech Signals Spectral Representation of Speech and Noise Linear Prediction Model Representation of Speech and Noise Inter-Frame and Intra-Frame Correlations Speech Estimation Module Probability Models of Speech and Noise Cost of Error Functions in Speech Estimation Wiener Filter for De-noising Speech Wiener Filter Based on Linear Prediction Models HMM-Based Wiener Filters Spectral Subtraction of Noise Spectral Subtraction Using LP Model Frequency Response Bayesian MMSE Speech Enhancement Kaiman Filter for Speech Enhancement Kaiman State-Space Equations of Signal and Noise Models 433

12 Speech Enhancement Using LP-HNM Model Overview of LP-HNM Enhancement System Formant Estimation from Noisy Speech Initial-Cleaning of Noisy Speech Formant Tracking Harmonic Plus Noise Model (HNM) of Speech Excitation Fundamental Frequency Estimation Estimation of Amplitudes Harmonics of HNM Estimation of Noise Component of HNM Kaiman Smoothing of Trajectories of Formants and Harmonics Speech Bandwidth Extension-Spectral Extrapolation LP-HNM Model of Speech Extrapolation of Spectral Envelope of LP Model AAA Phase Estimation Codebook Mapping of the Gain Extrapolation of Spectrum of Excitation of LP Model Sensitivity to Pitch Interpolation of Lost Speech Segments-Packet Loss Concealment Phase Prediction Codebook Mapping Evaluation of LP-HNM Interpolation Multi-Input Speech Enhancement Methods Beam-forming with Microphone Arrays Spatial Configuration of Array and The Direction of Reception Directional of Arrival (Do A) and Time of Arrival (To A) Steering the Array Direction: Equalisation of the ToAs at the Sensors The Frequency Response of a Delay-Sum Beamformer Speech Distortion Measurements Signal-to-Noise Ratio - SNR Segmental Signal to Noise Ratio - SNR see Itakura-Saito Distance - ISD Harmonicity Distance - HD Diagnostic Rhyme Test - DRT Mean Opinion Score - MOS Perceptual Evaluation of Speech Quality - PESQ 464 Bibliography Multiple-Input Multiple-Output Systems, Independent Component Analysis Introduction A note on comparison of beam-forming arrays and ICA MIMO Signal Propagation and Mixing Models Instantaneous Mixing Models Anechoic, Delay and Attenuation, Mixing Models Convolutional Mixing Models All 18.4 Independent Component Analysis A Note on Orthogonal, Orthonormal and Independent Statement of ICA Problem Al A Basic Assumptions in Independent Component Analysis The Limitations of Independent Component Analysis Al 5

13 xviii Contents Why a mixture of two Gaussian signals cannot be separated? The Difference Between Independent and Uncorrelated Independence Measures; Entropy and Mutual Information All Differential Entropy Maximum Value of Differential Entropy Mutual Information The Effect of a Linear Transformation on Mutual Information Non-Gaussianity as a Measure of Independence Negentropy: A measure of Non-Gaussianity and Independence Fourth Order Moments - Kurtosis Kurtosis-based Contrast Functions - Approximations to Entropie Contrast Super-Gaussian and Sub-Gaussian Distributions Fast-ICA Methods Gradient search optimisation method Newton optimisation method Fixed-point Fast ICA Contrast Functions and Influence Functions ICA Based on Kurtosis Maximization - Projection Pursuit Gradient Ascent Jade Algorithm - Iterative Diagonalisation of Cumulant Matrices Summary 490 Bibliography Signal Processing in Mobile Communication Introduction to Cellular Communication A Brief History of Radio Communication Cellular Mobile Phone Concept Outline of a Cellular Communication System Communication Signal Processing in Mobile Systems Capacity, Noise, and Spectral Efficiency Spectral Efficiency in Mobile Communication Systems Multi-path and Fading in Mobile Communication Multi-path Propagation of Electromagnetic Signals Rake Receivers for Multi-path Signals Signal Fading in Mobile Communication Systems Large-Scale Signal Fading Small-Scale Fast Signal Fading Smart Antennas - Space-Time Signal Processing Switched and Adaptive Smart Antennas Space-Time Signal Processing - Diversity Schemes Summary 508 Bibliography 508 Index 509