Real-time Single-channel Dereverberation and Separation with Time-domain Audio Separation Network

Transcription

1 Interspeech September 2018, Hyderabad Real-tme Sngle-channel Dereverberaton and Separaton wth Tme-doman Audo Separaton Networ Y Luo Nma Mesgaran Department of Electrcal Engneerng, Columba Unversty, New Yor, NY yl3364@columba.edu nma@ee.columba.edu Abstract We nvestgate the recently proposed Tme-doman Audo Separaton Networ (TasNet) n the tas of real-tme snglechannel speech dereverberaton. Unle systems that tae tmefrequency representaton of the audo as nput, TasNet learns an adaptve front-end n replacement of the tme-frequency representaton by a tme-doman convolutonal non-negatve autoencoder. We show that by formulatng the dereverberaton problem as a denosng problem where the drect path s separated from the reverberatons, a TasNet denosng autoencoder can outperform a deep LSTM baselne on log-power magntude spectrogram nput n both causal and non-causal settngs. We further show that adjustng the strde sze n the convolutonal autoencoder helps both the dereverberaton and separaton performance. Index Terms: speech dereverberaton, speech separaton, tmedoman, deep learnng 1. Introducton Real-world speech communcaton often taes place n crowded or reverberant condtons where the speech sgnal s corrupted by other speaers, envronmental noses, or room reverberatons. A successful system n such condtons thus requres robust speech separaton or speech dereverberaton functon. Moreover, n such applcatons where real-tme processng s necessary, the latency of the system remans an mportant lmtng ssue. In recent years, deep learnng systems have shown to have better generalzaton ablty and hgher performance n varous condtons for both separaton and dereverberaton [1, 2, 3, 4, 5, 6, 7, 8, 9]. In most of the systems, a tme-frequency (T-F) representaton s calculated from the audo waveform as the nput by short-tme Fourer transform (STFT). In speech separaton tass, a general method s to estmate a T-F mas for each of the speaer n the mxture. In dereverberaton, the anechoc T-F representaton s typcally estmated from the reverberant sgnal. The reconstructon of the waveforms s then done by nverse STFT. However, there are several ssues wth the usage of T-F representatons. Frst, performance of STFT-based systems s related to the choce of the wndow length n STFT, whch drectly affects the frequency resoluton as well as the system latency. In many systems, a wndow sze that s longer than 32 ms s requred to acheve a good performance [1, 2, 10]. Ths lmts the use of such systems n applcatons where a very short latency s requred, such as hearng ads and telecommuncaton devces. Addtonally, most of the systems for separaton and dereverberaton only modfy the magntude spectrogram or the mel-frequency cepstral coeffcent (MFCC) whle the phase spectrogram remans unchanged [3, 5, 6]. Ths lmts the performance upper-bound due to the usage of the nosy phase durng nverse STFT. Although there are methods such as phase-senstve mas for separaton [11] or complex rato mas and teratve reconstructon for dereverberaton [12, 7], the performance s stll lmted and model complexty mght be much hgher. Modelng the sgnals drectly n tme-doman may remedy the ssues mentoned above. A recently proposed neural networ, the Tme-doman Audo Separaton Networ (TasNet [4]), s a deep learnng system that operates n the tme-doman. TasNet models the nput waveform wth a 1-D convolutonal encoder-decoder framewor where the output of the encoder forms a non-negatve adaptve front-end (representaton) to replace the STFT. The target sources are estmated by calculatng mas-le matrces that are appled to the non-negatve representaton of the nput, whch s smlar to the typcal mas estmaton process n STFT-based systems. Because all of the operatons n TasNet are n the tme-doman, there s no upperbound performance due to the nosy phase spectrogram, and the latency of the system can be controlled by the length of the 1-D flters n the convolutonal autoencoder. Comparng wth STFTbased systems, the latency of TasNet can be as low as 5ms [4], whch maes t possble for real-tme low latency applcatons. It was shown that TasNet outperformed the state-of-the-art STFT-based systems on the separaton tas n both causal and non-causal confguratons [4]. However, whether TasNet s effectve n the problem of sngle-channel dereverberaton s unnown. In ths paper, we nvestgate the usage of TasNet as a denosng autoencoder (DAE) n the problem of speech dereverberaton. We formulate the dereverberaton problem as a separaton problem, where the reverberant speech s treated as the summaton of the drect path and the reverberant nose. A smlar mas estmaton process s desgned to extract the drect path from the reverberant nput. Based on the observaton on the dereverberaton problem, we further show that by addng overlap between the wndows (.e. adjustng the strde sze n the convolutonal autoencoder), the performance of dereverberaton and separaton can both be mproved. The rest of the paper s organzed as follows. Secton 2 descrbes the problem formulaton of the dereverberaton tas. Secton 3 consders the TasNet archtecture for dereverberaton. Secton 4 provdes the detals about the experments. Secton 5 concludes the paper. 2. Problem Descrpton A reverberant speech sgnal s composed of the drect sgnal x (d) (t) and the remanng reverberant nose x (e) (t) x(t) = x (d) (t) + x (e) (t) (1) In real-tme applcatons, audo sgnals typcally come n streams or segments. At each tme step, we assume that an au /Interspeech

2 do stream wth length of L samples s receved x = x(t) x (d) = x (d) (t) t [H, H + L), = 1,..., K (2) = x (e) (t) x (e) where x, x (d), x(e) R 1 L and H denotes the hop sze between streams. K stands for the total number of audo streams and vares from utterance to utterance. We drop the notaton where there s no ambguty. The am of dereverberaton s to estmate the drect sgnal x (d) from x. Followng the dea from the orgnal TasNet, a set of tranable bass sgnals B = [b 1, b 2,..., b N ] R N L s used to represent each of the segments wth a set of non-negatve weghts through a deconvolutonal operaton { x = Deconv(w, B) x (d) = Deconv(w (d) (3), B) where the weght vectors w, w (d) R 1 N. Wth the nonnegatvty constrant, a mas-le vector m (d) R 1 N can be estmated for w (d) w (d) = w (w (d) w) (4) := w m (d) (5) where and denotes element-wse multplcaton and dvson. Therefore, the problem of estmatng the drect path s equvalent to estmatng a mas-le vector whch s appled to a representaton of the reverberant speech. 3. TasNet for Dereverberaton Fgure 1: Tme-doman Audo Separaton Networ (TasNet) models the nput sgnal n the tme-doman usng a convolutonal encoder-decoder framewor. The output of the encoder forms the non-negatve representaton for the nput, and a mas for the target drect path s learned from the separator and appled to the encoder output. The decoder then reconstructs the waveform through a deconvolutonal operaton. The TasNet archtecture contans one 1-D convolutonal layer as the non-negatve encoder and several recurrent layers for mas estmaton, and one lnear 1-D deconvolutonal layer as the decoder. The encoder output serves as an adaptve front-end representaton for the tme-doman sgnal to replace the STFT feature. The decoder nverts the convolutonal operaton n the encoder by performng deconvoluton wth a set of tranable bass sgnals (flters) and reconstructs the waveforms. The recurrent layers estmate the mass usng the representaton generated by the encoder. Fgure 1 shows the flowchart of the system D convolutonal encoder The encoder conssts of a 1-D convolutonal layer wth ReLU actvaton for the non-negatvty constrant w = ReLU(LN(x U)) (6) where U R N L s the tranable parameter, x R 1 L s a segment of the nput mxture, and N s the number of channels (.e. the number of the flters). denotes the convoluton operator. LN corresponds to the layer normalzaton operaton [13]. The layer normalzaton operaton s appled here to ensure that the encoder s nvarant to nput rescalng, meanng that changng the energy of the sgnal wll not affect the separaton performance. In [4], t was mentoned that a gated CNN archtecture [14] s helpful for the convergence speed and fnal performance. However, we fnd emprcally that wth proper tranng, usng ReLU as the only actvaton functon does not harm the performance of the networ Deep LSTM separaton module The separaton module contans several staced LSTM layers followed by a fully-connected layer for mas estmaton. The nput to the separaton module s the sequence of K nput weght vectors w 1,... w K R 1 N, and the output s the mas-le vectors for the target sources. For dereverberaton, the output s only one mas m (d) corresponds to the drect path. Sgmod actvaton functon s used n the fully-connected layer. A layer normalzaton style operaton s appled to the nput of the separaton module n order to speed up and stablze the tranng process µ = 1 N w = g (w µ) + b (7) σ N w j σ = 1 N (w j µ) N 2 (8) j=1 j=1 where parameters g R 1 N and b R 1 N are gan and bas vectors that are jontly optmzed wth the networ. We fnd ths mportant for the networ to converge relably. In order to accelerate the tranng process and enhance the gradent flow, an dentty sp connecton [15] s added between every two LSTM layers. A lnear fully-connected layer s appled to the nput to the separaton module for reshapng t to the same sze as the output of the second LSTM layer. After the mas vector m (d) R 1 N for the drect path of each source s generated, the weght vector ŵ (d) each segment s calculated by multplyng m (d) weght vector w, as n equaton D deconvolutonal decoder R 1 N for wth the nput The decoder s a 1-D deconvolutonal layer to nvert the convoluton operaton n the encoder for tme-doman sgnal reconstructon. The waveform for each sgnal n each segment s calculated by the 1-D deconvoluton between the weght vector and a set of tranable 1-D flters B R N L ˆx (d) = Deconv(ŵ (d), B) (9) ˆx = Deconv(ŵ, B) (10) The 1-D flters B are parameters n the deconvolutonal layer and are jontly optmzed wth all the other parts of the 343

3 networ. The entre waveforms are then obtaned by concatenatng all the segments. The reconstructons n the overlapped parts n consecutve segments are summed up to form the fnal output Tranng objectve In [4], t s reported that usng scale-nvarant source-to-nose rato (SI-SNR) as the objectve led to better performance n the separaton tas. However, SI-SNR leads to much slower convergence and worse performance on the dereverberaton tas, possbly due to the auto-correlated structure between the drect path and the reverberant nose. Here, we use the mean-square error (MSE) between the estmated drect path and the real drect path, as well as the estmated recovered nput and the nosy nput as the objectve L = MSE(ˆx (d), x (d) ) + MSE(ˆx, x) (11) Note that the second term n equaton 11 s to ensure that w correctly represent the nput sgnal, whch s necessary because the drect path n dereverberaton problem s part of the nput Dataset Dereverberaton 4. Experments A smulated reverberant speech dataset s generated from the Wall Street Journal (WSJ0) dataset wth three dfferent room reverb characterstcs. Table 1 shows the characterstcs of the rooms. One mcrophone s located at the center of the room. The room mpulse responses (RIRs) are generated wth the mage method [16]. A tranng set of samples (30 hours n total) and a valdaton set of 6000 samples (10 hours n total) are generated from randomly selected utterances from the WSJ0 tranng set s tr s. A test set of 5000 samples (8 hours n total) s generated from randomly selected speaers n WSJ0 s dt 05 and s et 05 datasets. The sample rate for all utterances s set to 8Hz. Durng the generaton of the reverberant speech, a random utterance s frst drawn from the clean speech dataset. A room and ts correspondng T 60 s also randomly selected. The speaer s then randomly placed n the room wth at least 0.5m from the borders. The heght of the speaer s restrcted between 1m to 2m. Table 1: Characterstcs of dfferent rooms for dereverberaton smulaton Separaton Sze (m) T 60 (s) Small Medum Large For the separaton tas, we use the WSJ0-2mx dataset [1, 2, 10], whch contans 30 hours of tranng and 10 hours of valdaton data. The mxtures are generated by randomly selectng utterances from dfferent speaers n WSJ0 tranng set s tr s, and mxng them at random sgnal-to-nose ratos (SNR) between 0 db and 5 db. Fve hours of evaluaton set are generated n the same way usng utterances from 16 unseen speaers from s dt 05 and s et 05 n the WSJ0 dataset. The sample rate s also set to 8Hz Networ confguraton Table 2: Networ confguratons for dfferent tass. Tas Causal (N, H, L) Separator Dereverb (250, 40, 40) Separate (500, 40, 40) The parameters of the system nclude the segment length L, the hop sze H, the number of bass sgnals N, and the confguraton of the separator module. The parameters of the separator nclude the number of (B-)LSTM layers, the number of hdden unts n each (B-)LSTM layer, and the number of hdden unts n the fully-connected layer. Table 2 shows the confguraton of networs for dfferent tass. Note that settng the hop sze H to be equal to the wndow sze L means that there s no overlap between two consecutve segments. In Secton 4.4 we wll dscuss the effect of overlap n both tass. For the dereverberaton tas, we desgn another baselne deep LSTM (DLSTM) DAE model wth log-power magntude spectrogram nput. The wndow sze and hop sze of STFT are 256 samples (32ms) and 64 samples (8ms), respectvely. Ths results n a 129-dmensonal nput feature. The DLSTM DAE contans 4 (B-)LSTM layers wth the same sze as the separator n TasNet, wth a fully-connected layer of 129 hdden unts for estmatng the log-power magntude spectrogram of the drect path. No actvaton functon s appled n the fully-connected layer. Identty sp connectons are added between every two (B-)LSTM layers the same way as n Secton 3.2. We also apply the currculum tranng strategy [17] n a smlar fashon to [4]. We start tranng the networ on 1 secondlong utterances for dereverberaton and 0.5 second-long utterances for separaton, and contnue tranng on 4 second-long utterances afterward. For the DLSTM DAE model, we frst tran on 100 frame-long utterances (0.8s) and contnue on 400 framelong utterances (3.2s) Evaluaton metrcs For the dereverberaton tas, we evaluate the systems usng the perceptual evaluaton of speech qualty (PESQ) [18] and the scale-nvarant sgnal-to-nose rato (SI-SNR) [1, 4]. For the separaton problem, we evaluated the systems wth both SI- SNR mprovement (SI-SNR) and SDR mprovement (SDR) [19] metrcs used n [1, 2, 10] Experment results We frst nvestgate the effect of strde sze n the convolutonal autoencoder of TasNet on the performance. A hop sze of H L corresponds to a strde sze of L H n the 1-D convolutonal and deconvolutonal layers. Table 3 provde the effect of strde on the performance of dereverberaton tas after havng 50% hop sze (.e. addng 50% overlap between segments). We fnd that addng overlap between segments sgnfcantly helps the performance. We then compare TasNet DAE wth DLSTM DAE baselne on the dereverberaton tas. Table 4 presents the results 344

4 Table 3: PESQ and SI-SNR (db) for dfferent hop sze n TasNet DAE. Overlap Causal PESQ SI-SNR 0% % % % of PESQ and SI-SNR of the two systems. We fnd that Tas- Net DAE performs sgnfcantly better on SI-SNR but worse on PESQ. Ths can be explaned by the usage of the MSE-based objectve functon whch favors SNR more. Nevertheless, we can see that a causal TasNet DAE can stll outperform a noncausal DLSTM DAE n terms of SI-SNR. Ths means that Tas- Net DAE s able to learn a better mappng between the waveforms of the anechoc sgnals. Table 5 compares the system latency n causal TasNet and DLSTM DAE. Smlar to [4], the system latency T tot s expressed as the sum of the ntal delay of the system T and the processng tme for a segment T p. T s the length of the segment requred to produce the frst output, and T p s estmated as the average per-segment processng tme across the entre test set. Both models are loaded on a Ttan X Pascal GPU before the processng starts. We observe that the overall latency for TasNet s sgnfcantly smaller than the DLSTM DAE, due to the fact that TasNet decouples the wndow sze and the frequency resoluton n STFT. Ths enables the TasNet model to be deployed to real-tme and low-latency applcatons. Fnally, we examne the effect of strde (H) on speech separaton tas and compared wth the other state-of-the-art systems. Durng the tranng for TasNet wth 50% overlap (TasNet-50%), gradent clppng wth maxmum norm of 3 was appled to allevate the gradent exploson problem. We fnd that ths sgnfcantly mproves the performance. As shown n table 6, although TasNet wthout overlap (TasNet-0%) already has comparable performance wth other systems, TasNet wth 50% overlap sgnfcantly outperforms all the other systems n both causal and non-causal confguratons. Ths performance boost further proves the effcacy of TasNet n both onlne and offlne settngs n comparson wth STFT-based systems. Table 6: SI-SNR (db) and SDR (db) mprovements comparson for dfferent hop sze n TasNet for separaton. Method Causal SI-SNR SDR upit-lstm [2] 7.0 TasNet-0% [4] TasNet-50% DPCL++ [1] 10.8 DANet [10] 10.5 ADANet [3] 10.5 upit-blstm-st [2] 10.0 cupit-grd-rd [20] 10.2 CBLDNN-GAT[21] 11.0 Chmera++ [22] WA-MISI-5 [23] TasNet-0% [4] TasNet-50% Concluson In ths paper, we nvestgated the performance of a recently proposed neural networ for speech separaton, the tme-doman audo separaton networ (TasNet), on the tas of speech dereverberaton. We formulated the dereverberaton problem as a denosng problem where the drect path was separated from the echoc nose. Experments showed that TasNet outperformed a deep LSTM baselne wth spectrogram nput, and adjustng the strde sze n the convolutonal autoencoder further mproved the performance n both separaton and dereverberaton tass. 6. Acnowledgement Ths wor was funded by a grant from Natonal Insttute of Health, NIDCD, DC014279, Natonal Scence Foundaton CA- REER Award, and the Pew Chartable Trusts. Table 4: PESQ and SI-SNR (db) for TasNet DAE and DLSTM DAE baselne. Causal PESQ SI-SNR Mxture TasNet DAE DLSTM DAE TasNet DAE DLSTM DAE Table 5: Mnmum latency (ms) of TasNet and DLSTM DAE n dereverberaton tas. Method T T p T tot TasNet DLSTM DAE

5 7. References [1] Y. Is, J. Le Roux, Z. Chen, S. Watanabe, and J. R. Hershey, Sngle-channel mult-speaer separaton usng deep clusterng, Interspeech 2016, pp , [2] M. Kolbæ, D. Yu, Z.-H. Tan, and J. Jensen, Multtaler speech separaton wth utterance-level permutaton nvarant tranng of deep recurrent neural networs, IEEE/ACM Transactons on Audo, Speech, and Language Processng, vol. 25, no. 10, pp , [3] Y. Luo, Z. Chen, and N. Mesgaran, Speaer-ndependent speech separaton wth deep attractor networ, IEEE/ACM Transactons on Audo, Speech, and Language Processng, vol. 26, no. 4, pp , [4] Y. Luo and N. Mesgaran, Tasnet: tme-doman audo separaton networ for real-tme, sngle-channel speech separaton, n Acoustcs, Speech and Sgnal Processng (ICASSP), 2018 IEEE Internatonal Conference on. IEEE, [5] X. Feng, Y. Zhang, and J. Glass, Speech feature denosng and dereverberaton va deep autoencoders for nosy reverberant speech recognton, n Acoustcs, Speech and Sgnal Processng (ICASSP), 2014 IEEE Internatonal Conference on. IEEE, 2014, pp [6] M. Mmura, S. Saa, and T. Kawahara, Speech dereverberaton usng long short-term memory, n Sxteenth Annual Conference of the Internatonal Speech Communcaton Assocaton, [7] K. Han, Y. Wang, D. Wang, W. S. Woods, I. Mers, and T. Zhang, Learnng spectral mappng for speech dereverberaton and denosng, IEEE/ACM Transactons on Audo, Speech, and Language Processng, vol. 23, no. 6, pp , [8] Y. Ueda, L. Wang, A. Ka, X. Xao, E. S. Chng, and H. L, Snglechannel dereverberaton for dstant-talng speech recognton by combnng denosng autoencoder and temporal structure normalzaton, Journal of Sgnal Processng Systems, vol. 82, no. 2, pp , [9] Y. Zhao, Z.-Q. Wang, and D. Wang, A two-stage algorthm for nosy and reverberant speech enhancement, n Acoustcs, Speech and Sgnal Processng (ICASSP), 2017 IEEE Internatonal Conference on. IEEE, 2017, pp [10] Z. Chen, Y. Luo, and N. Mesgaran, Deep attractor networ for sngle-mcrophone speaer separaton, n Acoustcs, Speech and Sgnal Processng (ICASSP), 2017 IEEE Internatonal Conference on. IEEE, 2017, pp [11] H. Erdogan, J. R. Hershey, S. Watanabe, and J. Le Roux, Phasesenstve and recognton-boosted speech separaton usng deep recurrent neural networs, n Acoustcs, Speech and Sgnal Processng (ICASSP), 2015 IEEE Internatonal Conference on. IEEE, 2015, pp [12] D. S. Wllamson and D. Wang, Speech dereverberaton and denosng usng complex rato mass, n Acoustcs, Speech and Sgnal Processng (ICASSP), 2017 IEEE Internatonal Conference on. IEEE, 2017, pp [13] J. L. Ba, J. R. Kros, and G. E. Hnton, Layer normalzaton, arxv preprnt arxv: , [14] Y. N. Dauphn, A. Fan, M. Aul, and D. Granger, Language modelng wth gated convolutonal networs, n Internatonal Conference on Machne Learnng, 2017, pp [15] K. He, X. Zhang, S. Ren, and J. Sun, Identty mappngs n deep resdual networs, n European Conference on Computer Vson. Sprnger, 2016, pp [16] J. B. Allen and D. A. Berley, Image method for effcently smulatng small-room acoustcs, The Journal of the Acoustcal Socety of Amerca, vol. 65, no. 4, pp , [17] Y. Bengo, J. Louradour, R. Collobert, and J. Weston, Currculum learnng, n Proc. ICML, 2009, pp [18] I.-T. Recommendaton, Perceptual evaluaton of speech qualty (pesq): An objectve method for end-to-end speech qualty assessment of narrow-band telephone networs and speech codecs, Rec. ITU-T P. 862, [19] E. Vncent, R. Grbonval, and C. Févotte, Performance measurement n blnd audo source separaton, IEEE transactons on audo, speech, and language processng, vol. 14, no. 4, pp , [20] C. Xu, X. Xao, and H. L, Sngle channel speech separaton wth constraned utterance level permutaton nvarant tranng usng grd lstm, n Acoustcs, Speech and Sgnal Processng (ICASSP), 2018 IEEE Internatonal Conference on. IEEE, [21] C. L, L. Zhu, S. Xu, P. Gao, and B. Xu, CBLDNN-based speaer-ndependent speech separaton va generatve adversaral tranng, n Acoustcs, Speech and Sgnal Processng (ICASSP), 2018 IEEE Internatonal Conference on. IEEE, [22] Z.-Q. Wang, J. Le Roux, and J. R. Hershey, Alternatve objectve functons for deep clusterng, n Proc. IEEE Internatonal Conference on Acoustcs, Speech and Sgnal Processng (ICASSP), [23] Z.-Q. Wang, J. L. Roux, D. Wang, and J. R. Hershey, End-to-end speech separaton wth unfolded teratve phase reconstructon, arxv preprnt arxv: ,