A Machine Learning Framework for Enhancement and Recognition of Microphone Array Speech

Transcription

1 A Machie Learig Framework for Ehacemet ad Recogitio of Microphoe Array Speech Chi-Hui Lee School of ECE, Georgia Tech I collaboratio with GT ad USTC teams 1 Outlie ad Talk Ageda DSP based o learig oliear spectral regressio Paradigm shift: spectral mappig with deep learig & big data Three classical sigle-chael DSP problems (Part 1) DNN-based speech ehacemet (SE) DNN-based source or speech separatio (SS) DNN-based speech dereverberatio Extesio to far-field microphoe array speech (Part 2) Two-stage architecture for SE/SS ad robust speech recogitio Multiple sources of iterfereces i reverberat coditios Speaker-depedet ehacemet (oly five-miute traiig) Black-box LVCSR (already clea- or multi-coditio traied) Comparig multi-chael DNN architectures & performaces Summary, supplemets, refereces ad recet efforts 2 1

2 Speech i Noisy Eviromet 1. Additive oise (mathematical mixig): y t = x t + t STFT Y l, k = X l, k + N(l, k) Focused i first part 2. Covolutioal oise: y t = x t h t 3. Mixed oise: y t = x t h t + t y t = [x t + v t ] h t + t Ofte solved with mixig sigal assumptios by mathematical optimizatio i covetioal approaches!! Focused i secod part 3 Part 1: Sigle-Chael Speech Ehacemet, Separatio ad Dereverberatio -- Speactral Mappig with DNN Regressio 4 2

3 Topic 1: Speech Ehacemet (SE) Speech ehacemet: improvig the itelligibility ad/or overall perceptual quality of degraded speech sigals usig digital sigal processig (DSP) techiques Oe of the most addressed classical SP problems Issues: musical oise ad o-statioary backgrouds Ehacig Noisy speech, Exhibitio, SNR=5dB Clea speech 5 Covetioal Speech Ehacemet Classified by the umber of sigal chaels 1. Sigle chael speech ehacemet Time ad frequecy iformatio Focused i Part 1 2. Array based speech ehacemet Time, frequecy ad spatial iformatio Focused i Part 2 Covetioal Techiques: math ad physics Spectral subtractio, Wieer filterig, maskig ˆ MMSE log spectral amplitude (MMSE-LSA) Optimally modified LSA (OM-LSA) May others for sigle- ad multi-chaels SE X ( l, k) Y ( l, k) Nˆ( l, k) 6 3

4 Learig-Based Speech Ehacemet Early: HMM-based speech estimatio (Erphraim & Malah, 1984) Deep deoisig autoecoder (Lu, Tsao, Matsuda, Hori, 13) Classificatio-based separatio (Wag & Wag, 13) Noliear regressio fuctio F(.) for spectral mappig i 12 most previous DNN efforts were for classificatio-based learig X(l, k) = F Y l, k + E(l, k) What is F(.)? What parameters? How may? How to obtai a lot of the traiig pairs, (Y, X)? Ay special assumptios? Geeralizatio issues? How to estimate the parameters? How to hadle mismatched spectral magitude & phase 7 DNN-Based SE System Overview Traiig Stage Clea/Noisy Samples Feature Extractio DNN Traiig SPL, Jauary 14 (#3 i SPS sice 14) Ehacemet Stage Noisy Samples Y t Feature Extractio l Y DNN Decodig T-ASLP, Jauary 15 (#1 i SPS sice 15) ˆX l Waveform Recostructio ˆX t f Y 1. Feature extractio: log-power spectra (LPS) 2. Waveform recostructio: overlap-add (OLA) algorithm 3. Traiig: RBM pre-traiig + back-propagatio fie-tuig 4. Phase (later) 8 4

5 DNN Based Spectral Mappig: A Paradigm Shift X 1*257=257 or More 48 h3 48 h2 48 h1 11*257=2827 Y (Output with a sigle frame of clea speech features) (Iput with multiple frames of oisy speech features) W 4 W 3 + ε 3 W 2 + ε 2 W 1 + ε 1 Deep Learig Big Data: Simulatio X= F Y + E High-dim Vector-to-vector oliear regressio: millio parameters 9 Noise-Uiversal SE-DNN A Hope DNN to lear the characteristics of may oise types Classificatios: Crowdig machie trasportatio aimal ature huma, etc. alarm cry G. Hu, 100 o-speech evirometal souds,

6 Ehaced Results: No-statioary Noise A utterace with machie gu oise at SNR= -5dB: with 104- oise ehaced (upper left, PESQ=2.78), MMSE ehaced (lower left, PESQ=1.86), 4-oise ehaced (upper right, PESQ=2.14), ad oisy speech (lower right, PESQ=1.85): 104NT-DNN ehaced PESQ=2.78 Log-MMSE ehaced PESQ=1.86 4NT-DNN ehaced PESQ=2.14 oisy,machie gu, SNR=-5dB PESQ=1.85 Eve the 4NT-DNN is much tha Log-MMSE, SE-DNN is capable of suppress highly ostatioary oise. Why? More ehacemet examples ca be foud at: home.ustc.edu.c/~xuyog62/demo/se_dnn.html 11 Ehaced Results: Real-World Speech Spectrograms of a utterace extracted from the movie Forrest Gump: DNN (left), Log-MMSE (right), ad oisy (middle) with usee oise Uiversal SE-DNN ehaced Log-MMSE ehaced Noisy Good geeralizatio capacity to real-world oisy speech Publicly available tool packages GPU C++ versio: Pytho versio:

7 Topic 2: Source & Speech Separatio (SS) Source separatio aims at separatig a target speaker speech from mixed speech with iterferig speakers (oe domiatig speaker) to improve the itelligibility ad overall perceptual quality of separated speech ad possibly for ASR/SID/LID usig acoustic sigal processig techiques Ideal DNN-based speech ehacemet / source separatio More tha oe-hour traiig speech from the target speaker Mixed speech SIR=0dB Separatio Separated Target Log-MMSE based speech ehacemet 13 Comparisos: M-F Mixture (5-mi Target, No-ideal) Mix (M+F -3dB) Supervised GMM (PESQ=1.79) Semi-supervised DNN (PESQ=2.68) Usupervised CASA (PESQ=1.26) Usupervised DNN (PESQ=2.62) Target(M) Sigle-Chael SS Supervised: both speakers kow Semi-supervised: oly target kow Usupervised: both speaker ukow Multi-chael BSS: ot compared T-ASLP, August 16 T-ASLP, July 17 Semi-supervised DNN: better tha supervised GMM Usupervised DNN: better tha state-of-the-art CASA 7

8 Topic 3: Speech Dereverberatio Reverb Ehaced Reverberat speech, No pre-processig, ASR errors Issues: A lot RIR RT60 Mismatched dereverberatio, less ASR errors, worst PESQ T-ASLP, Ja 17 J-STLP, Dec 17 Good DSP will Lead to Accurate ASR Matched dereverberatio ad SE, o ASR errors, best PESQ 15 Summary: So Far 1. Spectral mappig with deep learig & big data: a paradigm shift 2. Large traiig set: learig rich regressio structure Eve simulatio data ca be very useful if properly geerated 3. For DNN-based speech ehacemet, separatio, ad dereverberatio the results are amazigly good so far Multiple sources of iterfereces: ext target Array-based ehacemet (DNN too big?) ad ASR papers, SPL: #3, T-SALP: #1 top cited paper i IEEE SPS 5. Need to combie with covetioal techiques, e.g., IRM, IME 6. A New Hope: with proper pre-processig followed by itegrated post-processig leadig to robust ASR! But for black-box ASR? Lowest errors i CHiME-2, -4, & REVERB Challeges 16 Focused i Part 2 8

9 Part 2: Sigle- ad Multi-Chael Master-Voice Separatio ad Recogitio of Array Speech -- Prelimiaries with Two-Stage Ehacemet (Balacig Temporal ad Spatial Iformatio) 17 Issues: from Sigle-Chael to Multi-Chael Speech Ukow or imprecise array cofiguratios or microphoe types Robustess to room types, target, iterferece ad array positios Room-specific or geeral room coditios: RIR ad RT60 Whither temporal or spatial iformatio: iput vectors could be too log? Backed ASR is a black box, ofte multi-coditio traied. What to optimize? This talk: livig-room or i-vehicle applicatio scearios Coditio-specific but with more complex traiig data geeratio Multiple sources of iterfereces ad additive oises with room reverberatio Speaker-idepedet (SI) pre-processig could ot deliver satisfactory performaces SD master voice pre-processig performs better but how much traiig data? I this talk: less tha 5-miute master traiig voice, ad mostly clea-traied ASR 18 9

10 Acoustic Eviromet for Array Speech Simulatio Room Size: 6m*5.5m*3m Clea speech: WSJ Noise: OSU-100, NOISEX RIR: ISM, RT60: sec Traiig SD: 30 hours (from 40 to K utteraces) Testig SD: 1800 utteraces: usee speakers ad oises Talkig distace: 1-5 meters SINR: 5, 10 ad 15 db at the receivig microphoe Referece: Microphoe #1 A example 19 SINR > 5dB: or WER may exceed 100% DoA assumed kow by wake-up or cameras Sigle-Chael Speaker-Depedet Speech Separatio Baselie DNN cofiguratio ( sigle ) 3 hidde layer: 48 uits each Iput: 257-dim LPS features Temporal cotext size: 11-frame iput The dual outputs: estimated 257-dim LPS ad 257-dim IRM (ideal ratio mask or IRM for post-processig) features Multi-target learig N 1 2 E Xˆ ( Y, W, b ) X r N 1 1 N N , 0.7, 1.25 IRM ( Y, W, b) IRM. IRM 2 e X X e e N 2 2 Y ( d), IRM ( d) ˆ X ( ) ( ( ) ˆ d Y d X ( d)) / 2, IRM ( d) Xˆ ( d ), IRM 10

11 Overall Sigle-Chael ASR Result Summary Task: 230K-word WSJ speaker-idepedet recogitio with trigram LM perplex of 141 AM: CD-DNN-HMM traied with 70 hours of WSJ clea data, 6 hidde layers with 48 uits each, iput is 40-dim FMLLR, 11-frame expasio, output is 3455 shared states Test: Nov92, 8 speakers, 4 males ad 4 females, 300 utteraces, clea WER: ~3% SD Separator traiig: with 8-speaker clea adaptatio data, 40 utteraces each WER (i %) ad WERR (i paretheses i %) at 1m rage SINR = 5dB SINR = 10dB SINR = 15dB Noisy reverberat DNN processed (70.32) (64.23) (40.79) WER (i %) ad WERR (i paretheses i %) at 3m rage SINR = 5dB SINR = 10dB SINR = 15dB Noisy reverberat DNN processed (68.77) (63.45) (36.18) DNN architecture Ehacemet with 1 DNN Traiig: radomly selected 90 out of 240 hours 3 hidde layers, 48 uits i each 8-frame Iput: 8-chael LPS cocateated Temporal cotext size i each chael: 1 Dual outputs: 257-dim LPS dim IRM of chael 1 Proposed Multi-Chael DNN-Based Speech Ehacemet DNN Architecture 1 (DNN1) 0, 0.05, 0.7, 1.25 IRM-based post-processig ( pp ) Y ( d), IRM ( d) ˆ X ( ) ( ( ) ˆ d Y d X ( d)) / 2, IRM ( d) ˆ X ( d ), IRM N 1 2 E Xˆ ( Y, W, b ) X r N 1 1 N N 1 IRM ( Y, W, b) IRM

12 Proposed Two-stage Multi-Chael DNN-Based Speech Ehacemet DNN Architecture 2 (DNN2) DNN architecture it08 Stage 1: pre-ehacemet with 4 DNNs pre pre pre ad pre hidde layers, 48 uits i each Referece chael: chael 1 10-frame Iput : 2-chael LPS cocateated Temporal cotext size i each chael: 5 Outputs: ehaced 257-dim LPS of chael 1 Stage 2: itegratio with 1 DNN 3 hidde layers, 48 uits i each Iput with 4 or 12 frames: 4 ehaced LPS cocateated with 8-chael oisy LPS Dual outputs: 257-dim LPS dim IRM 23 Proposed Two-stage Multi-Chael DNN-Based Speech Ehacemet DNN Architecture 3 (DNN3) DNN architecture it0/8 Stage 1: pre-ehacemet with 2 DNNs pre ad pre hidde layers, 48 uits i each 12-frame Iput : 4-chael LPS cocateated Temporal cotext size i each chael: 3 Outputs: ehaced 257-dim LPS of chael 1 Stage 2: itegratio with 1 DNN 3 hidde layer with 48 uits i each Iput with 2 or 10 frames: 2 ehaced LPS cocateated with 8-chael oisy LPS Dual outputs: 257-dim LPS dim IRM 24 12

13 Prelimiary Results for Architecture Comparisos WER ad PESQ results at 3m for Speaker 447 SINR/dB RT60/s Average WER (%) Average PESQ Baselie DNN1 DNN2 DNN3 Post-processig oisy ch sigle pre it it pre it it it8+pp aechoic Overall PESQ ad ASR Result Summary Task: 230K-word WSJ cotiuous speech recogitio with trigram LM perplex of 141 AM: CD-DNN-HMM traied with 70 hours of WSJ1 clea data, 6 hidde layers with 48 uits each, iput is 40-dim FMLLR trasformed MFCC, 11-frame expasio, output is 3455 seoes Test: Nov92, 8 speakers, 4 males ad 4 females, about 1800 utteraces for each speaker Separator traiig: with 8-speaker clea adaptatio data, about five miutes each Average PESQ, WER ad WERR (i paretheses i %) over all 8 speakers for systems at SINR 5-15dB, RT s, ad 1-5m 1-5m Average PESQ Average WER % (WERR) oisy ch baselie: sigle (63.10%) proposed: it (62.04%) proposed: it8+pp (3.24%) Aechoic

14 Discussio o Clea- vs. Multi-Coditio Traiig Same sigle-chael ad multi-chael speech pre-processig AM: 41-dim fbak features, 11 frames expasio (per-utt cmv), 4 hidde layers, 1024 hidde odes WER(%) Spk 447 3m clea oisy MC RIR iterfere fbak MC RIR fbak 3.69 MC fbak 3.16 Clea fbak % 8-chael ehaced % % Clea ASR: 70 hours clea data MC fbak ASR: add 90 kids of oise, SNR=0dB 5dB 10dB, 280 hours oisy data MC RIR fbak ASR: covolve clea data with the RIR of 80 degree(rt60=0.2&0.3s), add 90 kids of oise, SNR= 0, 5, 10 db; 280 hours (cotai 70 hours aechoic data) MC RIR iterfere ASR: covolve clea data with RIR (RT60=0.2&0.3s), add 74 iterferers, add 90 kids of oise, SINR= 0, 5, 10 db, still 280 h (cotai 70 hours aechoic data) Ehaced speech works better for multi-coditio model. Better ehacemet put clea model o top? 33 A Example Test Utterace 3m 447 SINR=10dB RT60=0.2s post-processig Multi-chael with pp (PESQ:3.12) Multi-chael without post-processig (pp) (PESQ:3.10) without post-processig baselie sigle-chael clea oisy 0 Sigle chael baselie (PESQ:2.45) Clea Noisy ad reverberat (PESQ:2.24)

15 Summary: Two-Stage DNN Architecture (for SD Separatio ad SI Recogitio) Achieve sigificat PESQ improvemet ad WER reductio for multi-chael DNN, also effective i hadlig sigle-chael speech (5-mi SD traiig) Assume little o array cofiguratios; ot sesitive to array geometry Propose a ew two-stage ehacemet strategy: pre-ehacemet ad itegratio combiig both temporal ad spatial iformatio i spectra Need to replace kow with estimated power for real-world applicatios Power equalizatio caused about 10% degradatio: how to reduce it? Explore other techiques, e.g., SE for black-box clea- or multi-coditio ASR? Research further ito DNN architectures for array-based processig Ivestigate robustess issues i varyig rooms, positios ad array coditios 29 Ackowledgmet 15

16 Part 3: Supplemetary Slides Simulatio, Recet Efforts ad Refereces 31 Sigle-Chael SS: Defiitios Assumig mixed speech from oe target speaker ad oe iterferig speaker (more speakers later) Supervised separatio: both speakers kow GMM-based joit ad coditioal distributios Semi-supervised separatio: oly target kow Most reasoable sceario (more speakers later) This talk: DNN-based Usupervised separatio: both speakers ukow Computatioal auditory scee aalysis (CASA) This talk: DNN-based with geder mixture detectio Blid source separatio (BSS) Usually for multi-chael source separatio 32 16

17 Source Separatio: Real vs. Ideal A few miutes versus hours of target speech for traiig: Source Separatio Challege (SSC) target of GMM output target of DNN-2 output Iterferece of DNN-2 output Target: lay white by Y 6 please Iterferece : lay red with P 2 agai 3 3 Acoustic Eviromet for Array Speech Simulatio Room Size: 6m*5.5m*3m Speaker ID (geder) Target Directio(⁰) Iterferece Directio(⁰) 440 (M) (F) (F) (M) (F) (F) (M) (M) A example For sigle-chael, the sigle mic is Ref 1 Horizotal rage to the ceter is 1m, 3 m, 5m DoA assumed kow by wake-up or cameras 34 17

18 Data Simulatio Sigle Chael (1/2) Data Source: WSJ Corpus; OSU 100-type oise set Traiig Data Geeratio (for speaker-depedet master voice separatio) Target: 4 males, 4 females; ID umber from WSJ0 corpus; 40 clea utteraces for each Iterferig speakers: 72 speakers from WSJ0 corpus Noise: radom 90 types from OSU 100-type oise Corpus Oe simulated traiig utterace: 1 target utterace target Room Impulse Respose(RIR) + 1 radom iterferig utterace iterferig RIR + 1 radom oise Aliged utterace: target utterace Direct path of RIR; Room eviromets: rage 1m, 3m, 5m; each with 2 kids RT60s of 0.2s ad 0.3s; SINR cofiguratios (received at the microphoe): SINR = 5dB, SNR = 10, 15dB; SINR = 10dB, SNR = 15dB; SINR = 15dB, SNR = db (whe SNR was too low the ASR WER ofte exceeded 100%) Normalizatio: For each traiig ad testig target utterace, the reverberat utterace power = clea referece power = oisy reverberat power (ca be relaxed later with estimatio of TR60) Traiig data: SINR=5dB : SINR=10dB : SINR=15dB = 1 : 1 : 1; also iclude 40 SINR30dB SNR 31dB utteraces without reverberatio; about 000 simulated utteraces makig a 30-hour traiig set 35 Data Simulatio Sigle Chael (2/2) Testig Data Geeratio Target: the same 8 target with traiig, about 40 usee utteraces for each from WSJ0 Corpus Iterferig speakers: usee 10 speakers from WSJ0 Corpus Noise: usee 10 kids from OSU oise Corpus Oe simulated testig utterace: 1 target utterace target RIR + 1 radom iterferig utterace iterferig RIR + 1 radom oise Same SINR pairs: SINR = 5dB, SNR = 10, 15dB; SINR = 10dB, SNR = 15dB; SINR = 15dB, SNR = db 1m 3m 5m rage, both with RT60s of 0.2s ad 0.3s Testig data: about 1800 simulated utteraces (geerated Nov92 clea test utteraces) 300 utteraces for each situatio (too low SINR ofte caused over 100% WER) SINR/dB RT60/s

19 Data Simulatio Multi-Chael Traiig Data Geeratio (for speaker-depedet master voice separatio) Usig the same data geeratio strategy 8 chaels could have 8 times (240 hours) as much data as baselie (K utteraces) But oly 3 times (90 hours, utteraces) traiig data of all chaels were used Testig Data Geeratio About 1800 simulated utteraces are received by all chaels Data Geeratio Assumptios SINR was measured at the receivig microphoe With SINR at a lower level, the WER could exceed 100% DNN-Based Ehacemet: assumig kow power of desired aechoic outputs Estimated power (from estimated RT60) gave similar PESQ ad ASR results 37 Discussio o Utterace Power Equalizatio PESQ ad WER results w/o power equalizatio at 1-3 m Relaxig kow power assumptio usig data from Speaker 447 ( pest ): givig about 10% degradatio ID 447 1m 3m PESQ WER PESQ WER it it8+pest it8+pp it8+pp+pest

20 Selected Joural Publicatios 1. Y. Xu, J. Du, L.-R. Dai ad C.-H. Lee, A Experimetal Study o Speech Ehacemet Based o Deep Neural Networks, IEEE Sigal Processig Letters, Vol. 21, No. 1, pp , Jauary Y. Xu, J. Du, L.-R. Dai ad C.-H. Lee, A Regressio Approach to Speech Ehacemet Based o Deep Neural Networks, IEEE/ACM T-ASLP., Vol. 23, No. 1, pp. 7-19, Jauary J. Du, Y. Tu, L.-R. Dai, C.-H. Lee, A Regressio Approach to Sigle-Chael Speech Separatio via High- Resolutio Deep Neural Networks, IEEE/ACM T-ASLP., Vol. 24, No. 8, pp , B. Wu, K. Li, M. Yag, C.-H. Lee, A Reverberat-Time-Aware Approach to Speech Dereverberatio Based o Deep Neural Networks, IEEE/ACM T-ASLP., Vol. 25, No. 1, pp , Jauary Y. Wag, J. Du, L.-R. Dai, C.-H. Lee, A Geder Mixture Detectio Approach to Usupervised Sigle- Chael Speech Separatio Based o Deep Neural Networks, IEEE/ACM T-ASLP., Vol. 25, No. 7, pp , July Y.-H. Tu, J. Du, Q. Wag, X. Bao, L.-R. Dai ad C.-H. Lee, A Iformatio Fusio Framework with Multi- Chael Feature Cocateatio ad Multi-Perspective System Combiatio for Deep Learig Based Robust Recogitio of Microphoe Array Speech, Computer Speech & Laguage, Vol. 46, pp , B. Wu, K. Li, F. Ge, Z. Huag, M. Yag, S. M. Siiscalchi, ad C.-H. Lee, A Ed-to-Ed Deep Learig Approach to Simultaeous Dereverberatio ad Acoustic Modelig for Robust Speech Recogitio, IEEE J- STLP, Vol. 11, Issue 8, pp , December T. Gao, J. Du, L.-R. Dai ad C.-H. Lee, A uified DNN approach to speaker-depedet simultaeous speech ehacemet ad speech separatio i low SNR eviromets, Vol. 95, pp , Speech Com., Dec B. Wu, M. Yag, K. Li, Z. Huag, M. Siiscalchi, T. Wag ad C.-H. Lee, A Reverberatio-Time-Aware Approach Leveragig Spatial Ifo for Microphoe Array Dereverberatio, EURASIP J. o Advaces i Sigal Proc, Y.-H. Lai, Y. Tsao, X. Lu, F. Che, Y.-T. Su, J. K.-C. Che, M.-J. Lie, H.-Y. Che, L. P.-H. Li ad C.-H. Lee, A Noise Classificatio Based Deep Learig Noise Reductio Approach to Improvig Speech Itelligibility for Cochlear Implat Recipiets, to appear LISTEN i Ear Workshop, ad Hearig. 07/17/ Recet Joural ad Coferece Efforts Speaker-depedet ehacemet ad separatio (Speech Comm. 12/17) Array-based dereverberatio (EURASIP JASP, 12/17) Joit SS ad AM for multi-talker speech (SPS, 18) Multi-objective learig ad esemblig for Compact SE (T-ASLP, 07/18) Geeralized Gaussia desities for regressio error modelig (T-ASLP ad IS18) Multi-task learig of LPS ad IRM for SE (Iterspeech15) DNN-based VAD: SE followed by speech detectio (Iterspeech15) SNR-progressive learig for SE (Iterspeech16) ML approach to DNN parameter estimatio for SE (Iterspeech17) Geeratig mixig oises with oise basis fuctios for SE (Iterspeech17) Iterative mask estimatio ad post-processig for Array SE (for CHiME-4, ASPSIPA17) Combiig covetioal ad DNN techiques for SE ad ASR (ICASSP18) SE for speaker diarizatio (ICASSP18 ad Iterspeech18) Two-stage ehacemet of microphoe array speech for ASR (ISCSLP18) 40