Advanced Signal Processing and Digital Noise Reduction

Size: px

Start display at page:

Download "Advanced Signal Processing and Digital Noise Reduction"

Homer Morris Francis
5 years ago
Views:

1 Advanced Signal Processing and Digital Noise Reduction

2 Advanced Signal Processing and Digital Noise Reduction Saeed V. Vaseghi Queen's University of Belfast UK ~ W I lilteubner L E Y A Partnership between John Wiley & Sons and B. G. Teubner Publishers Chichester. New York. Brisbane. Toronto. Singapore. Stuttgart. Leipzig

3 Copyright 1996 jointly by John Wiley & Sons Ltd. and B.O. Teubner Soflcoverreprintofthe hardcover 1st edition 1996 John Wiley & Sons Ltd Baffins Lane Chichester West Sussex P0191UD England B.O. Teubner Industriestra8e Stuttgart (Vaihingen) Postfach Stuttgart Oennany National Chichester International (+44) National Stuttgart (0711) International All rights reserved. No part of this book may be reproduced by any means. or transmitted. or translated into a machine language without the written pennission of the publisher. Other Wiley Editorial Offices John Wiley & Sons. Inc. 605 Third Avenue New York. NY USA Brisbane Toronto Singapore Other Teubner Editorial Offices B.O. Teubner. Vedagsgesellschaft mbh. JohannisgaBe Leipzig. Oennany Die Deutsche Bibliotheck - CIP-Einheitsaufnahme Vaseghi. Saeed V. Advanced signal processing and digital noise reduction I Saeed V. Vaseghi. -Stuttgart; Leipzig; Teubner; Chichester; New York; Brisbane; Toronto; Singapore :Wiley ISBN DOI / ISBN (ebook) WO: DBN fm British Library Cataloguing in Publication Data A catalogue record for this book is available from the British Library Produced from camera-ready copy supplied by the authors using MacWord 5.1. This book is printed on acid-free paper responsibly manufactured from sustainable forestation. for which at least two trees are planted for each one used for paper production.

4 To my Parents with thanks to Peter Rayner and Ben Milner

5 Contents Preface... xvi 1 Introduction Signals and Information Signal Processing Methods Non-parametric Signal Processing Model-based Signal Processing Bayesian Statistical Signal Processing Neural Networks Applications of Digital Signal Processing Adaptive Noise Cancellation and Noise Reduction Blind Channel Equalisation Signal Classification and Pattern Recognition Linear Prediction Modelling of Speech Digital Coding of Audio Signals Detection of Signals in Noise Directional Reception of Waves: Beamforming Sampling and Analog to Digital Conversion Time-Domain Sampling and Reconstruction of Analog Signals Quantisation Bibliography Stochastic Processes Random Signals and Stochastic Processes Stochastic Processes The Space or Ensemble of a Random Process Probabilistic Models of a Random Process Stationary and Nonstationary Random Processes Strict Sense Stationary Processes Wide Sense Stationary Processes Nonstationary Processes Expected Values of a Stochastic Process The Mean Value Autocorrelation Autocovariance Power Spectral Density Joint Statistical Averages of Two Random Processes Cross Correlation and Cross Covariance Cross Power Spectral Density and Coherence Ergodic Processes and Time-averaged Statistics... 42

6 viii Contents Mean-ergodic Processes Correlation-ergodic Processes Some Useful Classes of Random Processes Gaussian (Normal) Process Multi-variate Gaussian Process Mixture Gaussian Process A Binary-state Gaussian Process Poisson Process Shot Noise Poisson-Gaussian Model for Clutters and Impulsive Noise Markov Processes Transformation of a Random Process Monotonic Transformation of Random Signals Many-to-one Mapping of Random Signals Summary Bibliography Bayesian Estimation and Classification Estimation Theory: Basic Definitions Predictive and Statistical Models in Estimation Parameter Space Parameter Estimation and Signal Restoration Performance Measures Prior, and Posterior Spaces and Distributions Bayesian Estimation Maximum a Posterior Estimation Maximum Likelihood Estimation Minimum Mean Squared Error Estimation Minimum Mean Absolute Value of Error Estimation Equivalence of MAP, ML, MMSE and MA VE Influence of the Prior on Estimation Bias and Variance The Relative Importance of the Prior and the Observation Estimate-Maximise (EM) Method Convergence of the EM algorithm Cramer-Rao Bound on the Minimum Estimator Variance Cramer-Rao Bound for Random Parameters Cramer-Rao Bound for a Vector Parameter Bayesian Classification Classification of Discrete-valued Parameters Maximum a Posterior Classification Maximum Likelihood Classification Minimum Mean Squared Error Classification Bayesian Classification of Finite State Processes Bayesian Estimation of the Most Likely State Sequence Modelling the Space of a Random Signal Vector Ouantisation of a Random Process Design of a Vector Ouantiser: K-Means Algorithm

7 Contents ix Design of a Mixture Gaussian Model The EM Algorithm for Estimation of Mixture Gaussian Densities Summary Bibliography Hidden Markov Models Statistical Models for Nonstationary Processes Hidden Markov Models A Physical Interpretation of Hidden Markov Models Hidden Markov Model As a Bayesian Method Parameters of a Hidden Markov Model State Observation Models State Transition Probabilities State-Time Trellis Diagram Training Hidden Markov Models Forward-Backward Probability Computation Baum-Welch Model Re-Estimation Training Discrete Observation Density HMMs HMMs with Continuous Observation PDFs HMMs with Mixture Gaussian pdfs Decoding of Signals Using Hidden Markov Models Viterbi Decoding Algorithm HMM-based Estimation of Signals in Noise HMM-based Wiener Filters Modelling Noise Characteristics Summary Bibliography Wiener Filters Wiener Filters: Least Squared Error Estimation Block-data Formulation of the Wiener Filter Vector Space Interpretation of Wiener Filters Analysis of the Least Mean Squared Error Signal Formulation of Wiener Filter in Frequency Domain Some Applications of Wiener Filters Wiener filter for Additive Noise Reduction Wiener Filter and Separability of Signal and Noise Squared Root Wiener Filter Wiener Channel Equaliser Time-alignment of Signals Implementation of Wiener Filters Summary Bibliography Kalman and Adaptive Least Squared Error Filters State-space Kalman Filters Sample Adaptive Filters Recursive Least Squares (RLS) Adaptive Filters

8 x Contents 6.4 The Steepest Descent Method The LMS Adaptation Method Summary Bibliography Linear Prediction Models Linear Prediction Coding Least Mean Squared Error Predictor The Inverse Filter: Spectral Whitening The Prediction Error Signal Forward, Backward and Lattice Predictors Augmented Equations for Forward and Backward Predictors Levinson-Durbin Recursive Solution Lattice Predictors Alternative Formulations of Least Squared Error Predictors Model Order Selection Short-term and Long-term Predictors MAP Estimation of Predictor Coefficients Signal Restoration Using Linear Prediction Models Frequency Domain Signal Restoration Summary Bibliography Power Spectrum Estimation Fourier Transform, Power Spectrum and Correlation Fourier Transform Discrete Fourier Transform (DFf) Frequency Resolution and Spectral Smoothing Energy Spectral Density and Power Spectral Density Non-parametric Power Spectrum Estimation The Mean and Variance of Periodograms Averaging Periodograms (Bartlett Method) Welch Method :Averaging Periodograms from Overlapped and Windowed Segments Blackman-Tukey Method Power Spectrum Estimation from Autocorrelation of Overlapped Segments Model-based Power Spectrum Estimation Maximum Entropy Spectral Estimation Autoregressive Power Spectrum Estimation Moving Average Power Spectral Estimation Autoregressive Moving Average Power Spectral Estimation High Resolution Spectral Estimation Based on Subspace Eigen Analysis Pisarenko Harmonic Decomposition Multiple Signal Classification (MUSIC) Spectral Estimation Estimation of Signal Parameters via Rotational Invariance

9 Contents xi Techniques (ESPRIT) Summary Bibliography Spectral Subtraction Spectral Subtraction Power Spectrum Subtraction Magnitude Spectrum Subtraction Spectral Subtraction Filter: Relation to Wiener Filters Processing Distortions Effect of Spectral Subtraction on Signal Distribution Reducing the Noise Variance Filtering Out the Processing Distortions Non-linear Spectral Subtraction Implementation of Spectral Subtraction Application to Speech Restoration and Recognition Summary Bibliography Interpolation Introduction Interpolation of a Sampled Signal Digital Interpolation by a Factor of Interpolation of a Sequence of Lost Samples Factors that Affect Interpolation Polynomial Interpolation Lagrange Polynomial Interpolation Newton Interpolation Polynomial Hermite Interpolation Polynomials Cubic Spline Interpolation Statistical Interpolation Maximum a Posterior Interpolation Least Squared Error Autoregressive Interpolation Interpolation Based on a Short-term Prediction Model Interpolation Based on Long-term and Short-term Correlations LSAR Interpolation Error Interpolation in Frequency-Time Domain Interpolation using Adaptive Code Books Interpolation Through Signal Substitution Summary Bibliography Impulsive NOise Impulsive Noise Autocorrelation and Power Spectrum ofimpulsive Noise Stochastic Models for Impulsive Noise Bernoulli-Gaussian Model of Impulsive Noise Poisson-Gaussian Model of Impulsive Noise

10 xii Contents A Binary State Model of Impulsive Noise Signal to Impulsive Noise Ratio Median Filters Impulsive Noise Removal Using Linear Prediction Models Impulsive Noise Detection Analysis of Improvement in Noise Detectability Two-sided Predictor Interpolation of Discarded Samples Robust Parameter Estimation Restoration of Archived Gramophone Records Summary Bibliography Transient Noise Transient Noise Waveforms Transient Noise Pulse Models Noise Pulse Templates Autoregressive Model of Transient Noise Hidden Markov Model of a Noise Pulse Process Detection of Noise Pulses Matched Filter Noise Detection Based on Inverse Filtering Noise Detection Based on HMM Removal of Noise Pulse Distortions Adaptive Subtraction of Noise pulses AR-based Restoration of Signals Distorted by Noise Pulses Summary Bibliography Echo Cancellation Telephone Line Echoes Telephone Line Echo Suppression Adaptive Echo Cancellation Convergence of Line Echo Canceller Echo Cancellation for Digital Data Transmission over Subscriber's Loop Acoustic Feedback Coupling Sub-band Acoustic Echo Cancellation Summary Bibliography Blind Deconvolution and Channel Equalisation Introduction The Ideal Inverse Channel Filter Equalisation Error, Convolutional Noise Blind Equalisation Minimum and Maximum Phase Channels

11 Contents xiii Wiener Equaliser Blind Equalisation Using Channel Input Power Spectrum Homomorphic Equalisation Homomorphic Equalisation using a Bank of High Pass Filters Equalisation Based on Linear Prediction Models Blind Equalisation Through Model Factorisation Bayesian Blind Deconvolution and Equalisation Conditional Mean Channel Estimation Maximum Likelihood Channel Estimation Maximum a Posterior Channel Estimation Channel Equalisation Based on Hidden Markov Models MAP Channel Estimate Based on HMMs Implementations of HMM-Based Deconvolution Blind Equalisation for Digital Communication Channels Equalisation Based on Higher-Order Statistics Higher-Order Moments Higher Order Spectra of Linear Time-Invariant Systems Blind Equalisation Based on Higher Order Cepstrum Summary Bibliography Frequently used Symbols and Abbreviations Index

12 Preface Stochastic signal processing plays a central role in telecommunication and information processing systems, and has a wide range of applications in speech technology, audio signal processing, channel equalisation, radar signal processing, pattern analysis, data forecasting, decision making systems etc. The theory and application of signal processing is concerned with the identification, modelling, and utilisation of patterns and structures in a signal process. The observation signals are often distorted, incomplete and noisy. Hence, noise reduction and the removal of channel distortions is an important part of a signal processing system. The aim of this book is to provide a coherent and structured presentation of the theory and applications of stochastic signal processing and noise reduction methods. This book is organised in fourteen chapters. Chapter 1 begins with an introduction to signal processing, and provides a brief review of the signal processing methodologies and applications. The basic operations of sampling and quantisation are reviewed in this chapter. Chapter 2 provides an introduction to the theory and applications of stochastic signal processing. The chapter begins with an introduction to random signals, stochastic processes, probabilistic models and statistical measures. The concepts of stationary, non-stationary and ergodic processes are introduced in this chapter, and some important classes of random processes such as Gaussian, mixture Gaussian, Markov chains, and Poisson processes are considered. The effects of transformation of a signal on its distribution are considered. Chapter 3 is on Bayesian estimation/classification. In this chapter the estimation and classification problems are formulated within the general framework of the Bayesian inference. The chapter includes Bayesian theory, classical estimators, estimatemaximise method, Cramer-Rao bound on the minimum variance estimate, Bayesian classification, and the modelling of the space of a random signal. This chapter provides a number of examples on Bayesian estimation of signals observed in noise. Chapter 4 considers hidden Markov models for non stationary signals. The chapter begins with an introduction to the modelling of non stationary signals and then concentrates on the theory and applications of hidden Markov models (HMMs). HMM is introduced as a Bayesian model and the methods of training HMMs, and using HMMs for decoding and classification are considered. The chapter also includes the application of HMMs in noise reduction. Chapter 5 considers Wiener Filters. The least squared error filter is formulated first through minimisation of the expectation of the squared error function over the space of the error signal. Then a block-signal formulation of Wiener filters, and a vector space interpretation of Wiener filters, are considered. The frequency response of the Wiener filter is derived through minimisation of mean squared error in the frequency domain. Some applications of the Wiener filter are considered, and a case study of the Wiener filter for removal of additive noise, provides useful insight into the operation of the filter.

13 xvi Preface Chapter 6 considers the state-space Kalman filters and the sample-adaptive least squared error filters. The chapter begins with the state-space equation for Kalman filters. The optimal filter coefficients are derived using the principle of orthogonality of the innovation signal. The recursive least squared (RLS) filter which is an exact sample-adaptive implementation of the Wiener filter is derived in this chapter. Then the steepest descent search method for the optimal filter is introduced. The chapter concludes with a study of the LMS adaptive filters. Chapter 7 considers linear prediction models. Forward prediction, backward prediction and lattice predictors are studied. This chapter introduces a modified predictor for the modelling of the short term and the pitch period correlation structures. A maximum a posterior (MAP) estimate of a predictor model which includes prior probability density function of the predictor is introduced. This chapter concludes with application of linear prediction models in signal restoration. Chapter 8 consider frequency analysis and power spectrum estimation. The chapter begins with an introduction to Fourier transform, and the role of power spectrum in identification of patterns and structures in a signal process. The chapter considers nonparametric spectral estimation, model-based spectral estimation, maximum entropy method, and high resolution spectral estimation based on eigen analysis. Chapter 9 considers spectral subtraction. A general form of spectral subtraction is formulated and the processing distortions that result form spectral subtraction are considered. The effects of processing distortions on the distribution of a signal is illustrated. The chapter considers methods for removal of the distortions and also nonlinear methods of spectral subtraction. This chapter concludes with an implementation of spectral subtraction for signal restoration. Chapter 10 considers interpolation of a sequence of unknown samples. This chapter begins with a study of the ideal interpolation of a band limited signal, a simple model for the effects of a number of missing samples, and the factors that effect interpolation. Interpolators are divided into two categories of polynomial and statistical interpolators. A general form of polynomial interpolation, and its special forms Lagrange, Newton, Hermite, and cubic spline interpolators are considered. Statistical interpolators in this chapter include maximum a posterior interpolation, least squared error interpolation based on an autoregressive model, time-frequency interpolation, and interpolation through search of an adaptive codebook for the best signal. Chapters 11 and 12 cover the modelling detection and removal of impulsive noise and transient noise pulses. In chapter 11 impulsive noise is modelled as a binary state nonstationary processes and several stochastic models for impulsive noise are considered. For removal of impulsive noise, the median filters, and a method based on a linear prediction model of the signal process are considered. The materials in chapter 12 closely follow chapter 11. In this chapter a template-based method, an HMM-based method, and AR model-based method for removal of transient noise are considered. Chapter 13 covers echo cancellation. The chapter begins with introduction to telephone line echoes, and consider line echo suppression and adaptive line echo cancellation. Then the problem of acoustic echoes and acoustic coupling between loudspeaker and microphone systems are considered. The chapter concludes with a study of a sub-band echo cancellation system

14 Preface xvii Chapter 14 is on blind deconvolution and channel equalisation. This chapter begins with an introduction to channel distortion models and the ideal channel equaliser. Then the Wiener equaliser, blind equalisation using the channel input power spectrum, blind deconvolution based on linear predictive models, Bayesian channel equalisation, and blind equalisation for digital communication channels are considered. The chapter concludes with equalisation of maximum phase channels using the higher-order statistics. Saeed Vaseghl December 1995

Advanced Digital Signal Processing and Noise Reduction

Advanced Digital Signal Processing and Noise Reduction Fourth Edition Professor Saeed V. Vaseghi Professor of Communications and Signal Processing Department of Electronics & Computer Engineering Brunei