Speech & Audio Processing / Part-II. Digital Audio Signal Processing DASP. Marc Moonen

Transcription

1 Speech & Audio Processing / Part-II Digital Audio Signal Processing DASP Marc Moonen Dept. E.E./ESAT-STADIUS, KU Leuven marc.moonen@esat.kuleuven.be homes.esat.kuleuven.be/~moonen/ Overview Aims/scope Case study: Hearing instruments Overview Prerequisites Lectures/course material/literature Exercise sessions/project Exam Digital Audio Signal Processing: Introduction Version Lecture-1: Introduction 2 / 40 1

2 Aims/Scope Aim is 2-fold : Speech & audio per se S & A industry in Belgium/Europe/ Develop basic signal processing tools/principles Spatial filter design Adaptive filter algorithms (e.g. filtered-x LMS,..) Kalman filters Time-frequency analysis/processing Etc. which are also used in many other application fields Digital Audio Signal Processing: Introduction Version Lecture-1: Introduction 3 / 40 Case Study: Hearing Instruments 1/20 Course covers a number of ( typical ) DASP topics, e.g. Noise reduction Microphone array processing/beamforming Dereverberation Acoustic feedback cancellation Active noise control All of these (*) appear in the design of modern digital hearing instruments = remarkable/spectacular Hence very relevant DASP case (*) and many more : Dynamic range compression Auditory scene analysis Etc. = remarkable/spectacular Digital Audio Signal Processing: Introduction Version Lecture-1: Introduction 4 / 40 2

3 Case Study: Hearing Instruments 2/20 Hearing Outer ear/middle ear/inner ear Tonotopy of inner ear: spatial arrangement of where sounds of different frequency are processed Low-freq tone = Cochlea High-freq tone Neural activity for low-freq tone Neural activitity for high-freq tone Digital Audio Signal Processing: Introduction Version Lecture-1: Introduction 5 / 40 Case Study: Hearing Instruments 3/20 Hearing loss types: Conductive (~outer/middle ear) Sensorineural (~inner ear) Mixed One in six adults (Europe) suffers from hearing loss and still increasing Typical causes: Aging Exposure to loud sounds [Source: Lapperre] Digital Audio Signal Processing: Introduction Version Lecture-1: Introduction 6 / 40 3

4 Case Study: Hearing Instruments 4/20 è Hearing Aids (HAs) Audio input/audio output (`microphone-processing-loudspeaker ) 1921 Amplifier, but so much more than an amplifier!! History: Horns/trumpets/ `Desktop HAs (1900) Wearable HAs (1930) Digital HAs (1980) State-of-the-art: Digital HAs with.. MHz s clock speed Millions of arithmetic operations/sec, Multiple microphones 2007 (Oticon) Digital Audio Signal Processing: Introduction Version Lecture-1: Introduction 7 / 40 Case Study: Hearing Instruments 5/20 è Cochlear Implants (CIs) Audio input/electrode stimulation output Preprocessing similar to HAs + stimulation strategy + History: Volta s experiment First implants (1960) Commercial CIs ( ) Digital CIs (1980) State-of-the-art: MHz s clock speed, Mops/sec, Multiple microphones Intra-cochlear electrode Alessandro Volta Cochlear Ltd è Other: Bone anchored HAs, middle ear implants, Electrical stimulation Electrical stimulation Digital Audio Signal Processing: Introduction Version for low frequency Lecture-1: Introduction for high frequency 8 / 40 4

5 Case Study: Hearing Instruments 6/20 è Cochlear Implants (CIs) External Processor Analog/digital-conversion Digital processing & filterbank Etc.. Coil Inductive/magnetic coupling Implant Electrode array Cochlear Ltd PS: Number of CI-implantees worldwide approx PS: 3 Main companies (Cochlear LtD, Med-El, Advanced Bionics) Digital Audio Signal Processing: Introduction Version Lecture-1: Introduction 9 / 40 Case Study: Hearing Instruments 7/20 DASP Challenges in hearing instruments (next slides) Dynamic range compression Noise reduction Dereverberation Acoustic feedback cancellation Active noise control Etc. Technology Challenges in hearing instruments Small form factor (cfr. user acceptance) Low power: 1 5mW (cfr. battery lifetime 1 week) Low processing delay: 10msec (cfr. synchronization with lip reading) Digital Audio Signal Processing: Introduction Version Lecture-1: Introduction 10 / 40 5

6 Case Study: Hearing Instruments 8/20 DASP Challenges: Dynamic range compression Dynamic range & audibility Normal hearing subjects Level Hearing impaired subjects 100dB 0dB Digital Audio Signal Processing: Introduction Version Lecture-1: Introduction 11 / 40 Case Study: Hearing Instruments 9/20 DASP Challenges: Dynamic range compression Dynamic range & audibility Need `signal dependent amplification Level 100dB Output Level (db) 100dB 0dB 0dB 0dB 100dB Input Level (db) Design: multiband DRC, attack time, release time, Digital Audio Signal Processing: Introduction Version Lecture-1: Introduction 12 / 40 6

7 Case Study: Hearing Instruments 10/20 Why multiband DRC? requires analysis/synthesis filter bank Digital Audio Signal Processing: Introduction Version Lecture-1: Introduction 13 / 40 Case Study: Hearing Instruments 11/20 However: Audibility does not imply intelligibility Hearing impaired subjects need 5..10dB larger signal-to-noise ratio (SNR) for speech understanding in noisy environments Need for noise reduction (=speech enhancement) algorithms: State-of-the-art: monaural 2-microphone adaptive noise reduction Near future: binaural noise reduction (see below) Not-so-near future: multi-node noise reduction (see below) SNR 20dB 0dB Hearing loss (db, 3-freq-average) Digital Audio Signal Processing: Introduction Version Lecture-1: Introduction 14 / 40 7

8 Case Study: Hearing Instruments 12/20 DASP Challenges: Noise reduction & beamforming Multimicrophone beamforming, typically with 2 microphones e.g. directional front microphone and omnidirectional back microphone filter-and-sum the microphone signals Digital Audio Signal Processing: Introduction Version Lecture-1: Introduction 15 / 40 Case Study: Hearing Instruments 13/20 Binaural hearing based on binaural auditory cues ITD (interaural time difference) ILD (interaural level difference) signal ILD ITD Binaural cues (ITD: f < 1500Hz, ILD: f > 2000Hz) used for Sound localization Noise reduction =`Binaural unmasking ( cocktail party effect) 0-5dB Digital Audio Signal Processing: Introduction Version Lecture-1: Introduction 16 / 40 8

9 Case Study: Hearing Instruments 14/20 Binaural hearing aids Two hearing aids (L&R) with wireless link & cooperation Opportunities: More signals (e.g. 2*2 microphones) Better sensor spacing (17cm i.o. 1cm) Constraints: power/bandwith/delay of wireless link..10kbit/s: coordinate program settings, parameters,..300kbits/s: exchange 1 or more (compressed) audio signals Challenges: Improved localization through cue preservation Improved noise reduction + benefit from binaural unmasking Signal selection/filtering, audio coding, synchronisation, Digital Audio Signal Processing: Introduction Version Lecture-1: Introduction 17 / 40 Case Study: Hearing Instruments 15/20 Future: Multi-node noise reduction in sensor networks/internet-of-things Digital Audio Signal Processing: Introduction Version Lecture-1: Introduction 18 / 40 9

10 Case Study: Hearing Instruments 18/20 DASP Challenges: Dereverberation Reverb = filtering effect ( echo-ing ) of acoustic channel in between speaker and microphone(s) Reverb has an impact on speech understanding Dereverberation = undo filtering by acoustic channel (e.g. inverse filtering ) Digital Audio Signal Processing: Introduction Version Lecture-1: Introduction 19 / 40 Case Study: Hearing Instruments 16/20 DASP Challenges: Acoustic feedback cancellation Problem statement: Loudspeaker signal is fed back into microphone, then amplified and played back again Closed loop system may become unstable (howling) Similar to feedback problem in public address systems (for the musicians amongst you) Model F - Similar to echo cancellation in GSM handsets, Skype, but more difficult due to signal correlation Digital Audio Signal Processing: Introduction Version Lecture-1: Introduction 20 / 40 10

11 Case Study: Hearing Instruments 17/20 S.Doclo/Univ.Oldenburg DASP Challenges: Active noise control (ANC) ANC (see p.27) to counteract noise leakage & occlusion effect, exploiting additional internal reference microphone Digital Audio Signal Processing: Introduction Version Lecture-1: Introduction 21 / 40 Case Study: Hearing Instruments 20/20 DASP Challenges: piecing things together Digital Audio Signal Processing: Introduction Version Lecture-1: Introduction 22 / 40 11

12 Overview : Lecture-2 Noise Reduction-I Single-channel noise reduction `microphone_signal[k] = speech[k] + noise[k] Spectral subtraction methods (spectral filtering) Iterative methods based on speech modeling (Wiener & Kalman Filters) Applications: smartphones, conferencing, hearing aids, Digital Audio Signal Processing: Introduction Version Lecture-1: Introduction 23 / 40 Noise Reduction-II Overview : Lecture-3 Microphone Array Processing/Fixed Beamforming Referred to as spatial filtering (similar to spectral filtering ) Filter-and-sum beamformer Z( ω, θ ) Σ F 1 ( ω) F 2 ( ω) F m (ω) d m Y ( Y ( ω, ) 1 θ Y ( ω, ) 2 θ Y m ( ω, θ ) cosθ d m (ω) (ω) F M Y M ( ω, θ ) Applications: smartphones, conferencing, hearing aids, Digital Audio Signal Processing: Introduction Version Lecture-1: Introduction 24 / 40 12

13 Noise Reduction-III Overview : Lecture-4 Adaptive Beamforming/Multi-channel noise reduction Adaptive Beamforming Known (fixed) speaker position Unknown (time-varying) noise field Multi-channel noise reduction Wiener filtering approach : spectral+spatial filtering Applications: smartphones, conferencing, hearing aids, Digital Audio Signal Processing: Introduction Version Lecture-1: Introduction 25 / 40 Overview : Lecture-5 Reverberation & Dereverberation ` microphone_signal[k] = filter*speech[k] (+ noise[k]) Reverb = effect of acoustic channel in between speaker and microphone(s) Reverb has an impact on coding, speech recognition, etc. Single-microphone de-reverberation Cepstrum techniques Multi-microphone de-reverberation: Estimation of acoustic impulse responses Inverse-filtering method Matched filtering Digital Audio Signal Processing: Introduction Version Lecture-1: Introduction 26 / 40 13

14 Overview : Lecture-6 Acoustic Echo- and Feedback Cancellation Adaptive filtering problem: Non-stationary/wideband/ speech signals Non-stationary/long/ acoustic channels Adaptive filtering algorithms AEC Control AEC Post-processing Stereo AEC Digital Audio Signal Processing: Introduction Version Lecture-1: Introduction 27 / 40 Overview : Lecture-6 Acoustic Echo- en Feedback Cancellation Hearing aids, public address (PA) systems Correlation between filter input (`x ) and near-end signal ( n ) Fixes : noise injection, pitch shifting, notch filtering, amplifier Digital Audio Signal Processing: Introduction Version Lecture-1: Introduction 28 / 40 14

15 Overview : Lecture-7 Active Noise Control Solution based on `filtered-x LMS Application : active headsets/ear defenders Digital Audio Signal Processing: Introduction Version Lecture-1: Introduction 29 / 40 Overview : Lecture-8 Guest Lecture: Dr. Enzo De Sena, University of Surrey, UK Sound Field Recording and Reproduction Digital Audio Signal Processing: Introduction Version Lecture-1: Introduction 30 / 40 15

16 Lectures Lectures: 1 Intro (counts as half) + 7 Lectures PS: Time budget = 7.5*(2hrs)*4 = 60 hrs Course Material: Slides Use version ! Download from DASP webpage homes.esat.kuleuven.be/~dspuser/dasp/ Digital Audio Signal Processing: Introduction Version Lecture-1: Introduction 31 / 40 Prerequisites Signals & Systems (JVDW) Digital Signal Processing (I) (PW) signal transforms, sampling, multi-rate, DFT, DSP-CIS (MM) filter design, filter banks, optimal & adaptive filters Digital Audio Signal Processing: Introduction Version Lecture-1: Introduction 32 / 40 16

17 Literature Literature (general) (available in DSP-CIS library) Simon Haykin `Adaptive Filter Theory (Prentice Hall 1996) P.P. Vaidyanathan `Multirate Systems and Filter Banks (Prentice Hall 1993) Literature (specialized) (available in DSP-CIS library) S.L. Gay & J. Benesty `Acoustic Signal Processing for Telecommunication (Kluwer 2000) M. Kahrs & K. Brandenburg (Eds) `Applications of Digital Signal Processing to Audio and Acoustics (Kluwer1998) B. Gold & N. Morgan `Speech and Audio Signal Processing (Wiley 2000) Digital Audio Signal Processing: Introduction Version Lecture-1: Introduction 33 / 40 Exercise Sessions/Project Direction-of-arrival θ Acoustic source localization Direction-of-arrival estimation Noise reduction Synthesis Simulated set-up Digital Audio Signal Processing: Introduction Version Lecture-1: Introduction 34 / 40 17

18 Acoustic Source Localization Project Runs over 4 weeks (non-consecutive) PS: groups of 2 Each week 1 PC/Matlab session (supervised, 2.5hrs) 2 Homework sesions (unsupervised, 2*2.5hrs) PS: Time budget = 4*(2.5hrs+5hrs) = 30 hrs Deliverables after week 2 & 4 Grading: based on deliverables, evaluated during sessions TAs: guiliano.bernardi@esat (English+Italian) randall.ali@esat (English+Spanish) Digital Audio Signal Processing: Introduction Version Lecture-1: Introduction 35 / 40 Acoustic Source Localization Project Work Plan Week 1: Matlab acoustic simulation environment Week 2: Direction-of-arrival (DoA) estimation based on the MUSIC algorithm *deliverable* Week 3: DoA estimation + noise reduction ( DOA informed beamforming ) Week 4: Binaural synthesis and 3D audio *deliverable*..be there! Digital Audio Signal Processing: Introduction Version Lecture-1: Introduction 36 / 40 18

19 Exam Oral exam, with preparation time Open book Grading 7 for question-1 7 for question-2 +6 for project = 20 Digital Audio Signal Processing: Introduction Version Lecture-1: Introduction 37 / 40 September Retake Exam Oral exam, with preparation time Open book Grading 7 for question-1 7 for question-2 +6 for question-3 (related to project work) = 20 Digital Audio Signal Processing: Introduction Version Lecture-1: Introduction 38 / 40 19

20 Website 1) TOLEDO 2) Contact: Slides (use `version !!) Schedule DSP-library FAQs (send questions to Digital Audio Signal Processing: Introduction Version Lecture-1: Introduction 39 / 40 Questions? 1) Ask teaching assistant (during exercises sessions) 2) questions to teaching assistant or marc.moonen@esat 3) Make appointment marc.moonen@esat ESAT Room B Digital Audio Signal Processing: Introduction Version Lecture-1: Introduction 40 / 40 20