Speech bandwidth expansion based on Deep Neural Networks

Transcription

1 INTERSPEECH 2015 Speech bandwdth expanson based on Deep Neural Networks Yngxue Wang 1,2, Shenghu hao 1, Wenbo Lu 3,4, Mng L 3,5,Jngmng Kuang 1 1 School of Informaton and Electroncs, Beng Insttute of Technology, Beng , Chna 2 School of Computer Scence, Carnege Mellon Unversty 3 SYSU-CMU Jont Inst. of Eng., Sun Yat-Sen Unversty 4 Department of ECE, Carnege Mellon Unversty 5 SYSU-CMU Shunde Internatonal Jont Research Insttute yxwang.bt@gmal.com, shzhao@bt.edu.cn Abstract Ths paper proposes a new speech bandwdth expanson method, whch uses Deep Neural Networks (DNNs to buld hgh-order egenspaces between the low frequency components and the hgh frequency components of the speech sgnal. A four-layer DNN s traned layer-by-layer from a cascade of Neural Networks (NNs and two Gaussan-Bernoull Restrcted Boltzmann Machnes (GBRBMs. The GBRBMs are adopted to model the dstrbuton of spectral envelopes of the low frequency and the hgh frequency respectvely. The NNs are used to model the ont dstrbuton of hdden varables extracted from the two GBRBMs. The proposed method takes advantage of the strong modelng ablty of GBRBMs n modelng the dstrbuton of the spectral envelopes. And both the obectve and subectve test results show that the proposed method outperforms the conventonal GMM based method. Index Terms: bandwdth extenson, deep neural networks, neural networks, Gaussan-Bernoull Restrcted Boltzmann Machne 1. Introducton Speech bandwdth expanson (BWE s a technque that attempts to mprove the speech qualty by recoverng the mssng hgh frequency components usng the correlaton that exsts between the low and hgh frequency parts of the wde-band speech sgnal. The BWE technques have been appled to varous tasks, such as speech recognton [1], multcast conference [2], etc. Many approaches have been proposed for speech bandwdth extenson durng the last decades. Generally, these methods can be classfed nto two categores: rule-based methods and statstcal methods. The rule based methods drectly regenerate the hgh frequency spectral based on the acoustcal knowledge of the speech sgnal, e.g. smply copyng a porton of the narrowband spectrum onto the desred extenson frequency components [3]. On the other hand, the statstcal methods employ statstcal models to estmate the mappng functon between the low frequency and hgh frequency spectral features [4, 5, 6, 7]. By contrast to rule-based methods, statstcal methods can construct more precse mappng functons usng statstcal models. Therefore, statstcal methods, especally the GMM-based B- WE methods are wdely used [5]. Motvated by the success of Deep Neural Networks (DNN n speech recognton [8], we propose to utlze DNN to estmate a robust mappng functon for speech bandwdth extenson. Dfferent from the conventonal non-lnear or lnear transformaton approaches, the DNN learns both a lnear and a non-lnear relatonshp between the low frequency and hgh frequency spectral envelopes. Thus, DNN can learn a more detaled and precse relatonshp between the low frequency and hgh frequency. In our approach, dfferent from the conventonal feedforward neural networks for regresson tasks, whch are usually traned usng the back-propagaton algorthm under the mnmum mean square error crteron, a four-layer DNN s traned layer-by-layer from a cascade of Neural Networks (NNs and two Gaussan-Bernoull Restrcted Boltzmann Machnes (G- BRBMs. In the tranng phase, we frst tran two exclusve GBRBMs for low frequency and hgh frequency to obtan the deep networks that capture abstractons for each speech. Then, low frequency feature vectors and hgh frequency feature vectors are fed nto ther correspondng GBRBM and hgh-order features produced by GBRBMs are used to tran a concatenatng neural network between the two GBRBMs. In the reconstructon phase, the low frequency sgnal s converted through the traned NNs n the hgh-order space, and brought back to the cepstrum space usng the nverse process of the hgh frequency GBRBM. Ths paper s organzed as follows. Secton 2 gves an overvew of RBM and GBRBM whle secton 3 explans our speech bandwdth extenson method. We show our setup and expermental results n secton 4, and secton 5 s our concluson. 2. Prelmnares Our speech bandwdth extenson method uses GBRBM to capture hgh-order features. We brefly revew the GBRBM and ts fundamental model, Restrcted Boltzmann machne (RBM, n ths secton RBM A RBM s a bpartte undrected graphcal model. It has a twolayer structure wth one vsble layer correspondng to a set of vsble stochastc varables v = [v 1,...v V ] T and one hdden layer correspondng to a set of hdden stochastc varables h = [h 1,...h H] T, where V and H denote the number of unts n the vsble and hdden layers [9]. The ont probablty p(v, h of bnary-valued vsble unts v and bnary-valued hdden unts h s defned as follows: p(v, h = 1 exp(e(v, h (1 E(v, h = av T bh T v T W h (2 Copyrght 2015 ISCA 2593 September 6-10, 2015, Dresden, Germany

2 = exp(e(v, h (3 v,h where W = {w } ɛr I J, aɛr I 1 and bɛr J 1 are the weght parameter matrx between vsble unts and hdden unts, a bas vector of vsble unts, and a bas vector of hdden unts, respectvely. Because there s no connecton between vsble unts or between hdden unts, the condtonal probabltes can be wrtten as: p (v = 1 h = σ (h T W T : + a (4 p (h = 1 v = σ (v T W : + b (5 where W :, W : denote the column vector and the row vector n W respectvely, and ndcates an sgmod functon;.e. σ (x = 1. 1+e x Conventonally, parameters of a RBM are estmated by maxmzng the log-lkelhood L = log n p (vn. Dfferentatng partally wth respect to each parameter, we obtan L W = vh σ 2 data vh σ 2 model (6 L v v = (7 a σ 2 data σ 2 model L = h b data h model (8 where data and model ndcate the expectatons of the nput data and the nner model. Because model s extremely expensve to compute exactly, the contrastve dvergence approxmaton to the gradent s used, where model s replaced by runnng the Gbbs sampler ntalzed at the data for one full step [10] GBRBM GBRBM s an extended verson of RBM and s sutable for contnuous and real-valued data. The unts n the vsble layer of the GBRBM represent Gaussan stochastc varables, whle those n hdden layer represent Bernoull stochastc varables [11]. The dstrbuton of the stochastc varable descrbed by the GBRBM s defned by an energy functon N (v n a n 2 M E (v, h Θ = b mh m n=1 N M n=1 m=1 2σ 2 n v n σ n w nmh m m=1 where Θ = (W, a, b s the parameter set of an GBRBM, W ɛr N M are weghts connectng vsble and hdden neurons, a = [a 1,...a N ] T and b = [b 1,...b M ] T are the bas terms of vsble unts and hdden unts respectvely. s the standard devaton assocated wth a Gaussan vsble neuron v n. The ont dstrbuton p (v, h over v and h s defned by the energy functon as where (9 p (v, h = 1 exp (E (v, h (10 = h exp(e(v, hdv (11 Wx hx x GBRBM x 1 W h NN 2 W h Wy hy y T W y GBRBM y Fgure 1: The structure of the GBRBM BWE system The dstrbuton of the vsble unts s then gven as p (v = 1 exp (E (v, h h = 1 N ( exp (v n a n 2 n=1 2σ 2 n M 1 + exp (b m + v T W :m m=1 (12 The parameters n GBRBMs can be optmzed to maxmze the log-lkelhood functon wth a stochastc gradent. Once the parameters are estmated, the condtonal probablty of h gven v and the condtonal probablty of v gven h are respectvely wrtten as: ( p (v = v h = N v; h w + b, σ 2 (13 p (h = 1 v = σ v w + b (14 where N ( v; µ, σ 2 denotes the probablty densty functon of the Gaussan dstrbuton wth mean µ and varance σ Speech Bandwdth Extenson usng DNNs 3.1. Spectral expanson usng DNN Fgure 1 shows a flow chart of our method. The proposed model s a four-layer feedforward DNN, ncludng an nput layer, two hdden layers and an output layer. In Fgure 1, the dashed arrow ndcates the tranng phase whle the sold arrow ndcates the reconstructon phase. The nput and output layers denote the stochastc varables n the spectral vectors of low frequency and hgh frequency respectvely. In the tranng phase, two GBRBMs are adopted to model the dstrbuton of spectral envelopes for the low frequency and hgh frequency respectvely. Then a NN s employed to model the dstrbuton of the hdden varables extracted from the two GBRBMs. In the reconstructon phase, an nput vector of the low frequency s fed to GBRBM x, NN, GBRBM y n order and then converted to a hgh frequency vector y. 2594

3 To be more specfc, the tranng process and reconstructng process of the proposed DNN based speech bandwdth extenson s conducted as follows: Step 1 : Tran a GBRBM x usng data x of spectral envelopes of the low frequency. Then gven the vsble samples y and estmated parameters, draw ther correspondng hdden samples h x from the condtonal dstrbuton, whch s p (h x, = 1 x = σ h x, = σ x w + b hx, x w + b hx, (15 (16 where b hx are bas vectors of forward nference for low frequency. Step 2: Tran a GBRBM y usng data y of spectral envelopes of the hgh frequency. Then gven the vsble samples y, draw ther correspondng hdden samples h yusng mean-feld approxmaton from the condtonal dstrbuton, whch s p (h y, = 1 y = σ h y, = σ y σ w + b hy, y σ w + b hy, (17 (18 where b hy are bas vectors of forward nference for hgh frequency. Step 3: In the last step, a NN s traned, wth the proected vectors of the low frequency s acoustc feature h x beng the nputs, and the proected vectors of the correspondng hgh frequency s feature h y beng outputs. The weght parameters of the NN are estmated to mnmze the error between the output F (h x and the target vector h y as s typcal for a NN. Once the weght parameters are estmated, an nput vector h y s converted to h y = F (h x = σ ( W 2 h σ ( W 1 h h x + d 1 + d2 (19 where Wh 1, Wh 2 represents the weght matrces of the frst, second layer of the neural network, respectvely. Durng the reconstructon phase, to map the output h y of the NN to the acoustc feature of the hgh frequency, we ust use backward nference of GBRBM y usng Eq. (14, resultng n ( p (y h y = N y; σ ywy T h y + b y, σy 2 (20 When mnmzng the mean square error (MMSE estmaton rule s adopted for parameter generaton, the mappng functon takes the form: } ỹ MMSE = E {y h y = yp (y h y dy Ω y = yn (y; σ yw T h (21 y y + b y dy Ω y = σ yw T y h y + b y 3.2. Exctaton Expanson and power adustment Dfferent exctaton expanson technques have been nvestgated by many researchers and they can be classfed nto two groups. One s reusng the sgnal components of the LF exctaton sgnal by spectral foldng, spectral translaton [12]or nonlnear dstorton whch ncludes half-wave rectfcaton [13], full-wave rectfcaton [14], cubc functon [15], and the other one s generatng new components by nose/snusods generator [16] or non-lnear processng [17]. Utlzng the LF exctaton sgnal as HF exctaton sgnal results n the best BWE performance n terms of sound qualty. Therefore, n ths paper, we use ths method to predct the HF exctaton sgnal. It s necessary to adust the power of the extended exctaton sgnal to the power of the orgnal hgh frequency exctaton sgnal frame by frame. A codebook mappng method s employed to make the adustment. To be more specfc, obtan the energy gan factor g 1between the hgh frequency sgnal s h and the low frequency sgnals l, whch s N1 n=0 g 1 = log s2 h (n 10 (22 N1 n=0 s2 l (n tran and store a codebook C g1 of g 1 usng conventonal LBG algorthm. search an optmal codeword from the codebook C g1, obtanng the optmal codeword g 1 and the correspondng ndex. calculate the energy gan factor g 2 between the low frequency sgnal s l and s e obtaned from low frequency exctaton fltered through hgh frequency synthess flter, whch s N1 n=0 g 2 = log s2 e (n 10 (23 N1 n=0 s2 l (n the fnal gan factor s wrtten as 10 g g = 10 g Setup 4. Experments (24 We conducted speech bandwdth extenson usng one Mandarn Chnese database and an Englsh database. The frst Chnese speech database s from the NTT Advanced Technology Corporaton (NTT-AT [18]. The data s sampled at a 16-kHz samplng rate and dgtzed nto 16-bts resoluton. The Englsh database s the TIMIT corpus, whch also contans 16 khz speech recordngs [19]. A hgh-pass flterng suppled the hgh frequency sgnal. The low frequency sgnal resulted from a 0.3 to 3.4 khz band-pass flterng followed by a down-samplng and up-samplng wth a factor 2. We use the core tranng set defned n TIMIT (462 speakers and 4620 utterances and 64 utterances randomly selected from all speech sound classes n NTT as our tranng set. The test set conssted of the core test set defned n TIMIT and 32 utterances n NTT. The baselne system n our experment was the conversonal GMM based BWE. A GMM wth 128 components was traned for the baselne system. The 16-order lne spectral frequences (LSFs [20] were adopted as the spectral feature for the low frequency and hgh frequency. The frame sze and the frame shft 2595

4 for calculatng spectral envelopes was set to 20ms and 10ms respectvely. As long as learnng of standard devaton s not qute stable, we fxed σ to 1 and normalze the nput spectral feature vectors to zero mean and standard devaton 1. The contrastve dvergence (CD learnng wth 1-step Gbbs samplng was employed to tran GBRBMs. The stochastc batch gradent descent algorthm was adopted to update the model parameters. The sze of each mn-batch was set to 12 and the learnng rate was set to The number of epochs of GBRBMs and NNs were set to 1000 and 300 respectvely. The number of hdden unts of a GBRBM was fxed to 300. We nvestgated on three neural network structures ( a 1- layer NN, a 2-layer NN wth 1 hdden layer whch contans 600 nodes, a 3-layer NN wth 2 hdden layers and each hdden layer contans 600 nodes for the followng experments. Both obectve and subectve measures were used to evaluate the speech bandwdth extenson system. The reconstructed speech was measured obectvely n terms of dstorton between orgnal speech and reconstructed speech. The root mean square log spectral dstorton (RMS-LSD dstance n db and A-B preference tests were used as the obectve and subectve measurement, respectvely Obectve evaluaton We measured the RMS-LSD n the mssng hgh frequency (4-8 khz. The defnton of RMS-LSD [21] s as follows, ( D A,  = 1 ω2  (e ω 2 ω 1 ω 1 20log10 ω 2 dω A (e ω (25 where ω 1 and ω 2 are the cut-off frequences of the mssng band; A ( e ω and  ( e ω denote the power spectrum of orgnal wdeband frame and the power spectrum of correspondng artfcally expanded sgnal respectvely. The smaller the value of RMS-LSD s, the closer the reconstructed hgh frequency to the orgnal hgh frequency, the better the speech qualty s. The RMS-LSD results are shown n Table 1. Table 1: RMS-LSD comparson between GMM based BWE and DNNs based BWE. Method RMS-LSD(dB GMM based method 8.07 DNNs (1-layer NN 7.56 DNNs (2-layer NN 7.29 DNNs (3-layer NN Subectve evaluaton To evaluate the subectve qualty of the proposed DNNs based BWE method, A-B preference tests (A-speech by DNNs based BWE method, B-speech by GMM based BWE method were carred out and a total of 20 subects were asked to partcpate n the preference test. Table 2 shows results of the A-B preference tests. As shown n Table 2, the proposed method s sgnfcantly better than the conventonal GMM based BWE method, snce DNNs can produce more aurally natural speech than GMM. However, n ths paper, only two GBRBMs were studed. In the future, a deeper model whch can better descrbe the nonlnear mappng relatonshp between low frequency and hgh Table 2: Subectve preference scores between GMM based B- WE and DNNs based BWE. Case Propose no GMM-based method preference method DNNs (1-layer NN DNNs (2-layer NN DNNs (3-layer NN frequency wll be used by replacng the GBRBMs wth deeper stochastc neural networks, such as deep belef networks (DB- N. 5. Concluson In ths paper, we proposed a new speech bandwdth extenson method usng a combnaton of a low frequency GBRBM, a hgh frequency GBRBM and concatenatng NNs. In our approach, two exclusve GBRBMs for low frequency and hgh frequency were traned. A NN was then employed to model the ont dstrbuton of the hdden varables extracted from the two GBRBMs. In the reconstructon phase, gven a low frequency feature vector, the condtonal dstrbuton of the hgh frequency feature vector can be derved layer-by-layer. Our expermental results showed the effcacy of the proposed method, n comparson to a conventonal GMM-based method. 6. References [1] P. Bauer, J. Abel, V. Fscher, and T. Fngschedt, Automatc recognton of wdeband telephone speech wth lmted amount of matched tranng data, n Sgnal Processng Conference (EU- SIPCO, 2013 Proceedngs of the 22nd European. IEEE, 2014, pp [2] G. Gandhmath and S. Jayakumar, Speech enhancement usng an artfcal bandwdth extenson algorthm n multcast conferencng through cloud servces. Informaton Technology Journal, vol. 13, no. 12, [3] M. Detz, L. Lleryd, K. Korlng, and O. Kunz, Spectral band replcaton, a novel approach n audo codng, n Audo Engneerng Socety Conventon 112. Audo Engneerng Socety, [4] P. Jax and P. Vary, On artfcal bandwdth extenson of telephone speech, Sgnal Processng, vol. 83, no. 8, pp , [5] K.-Y. Park and H. S. Km, Narrowband to wdeband converson of speech usng gmm based transformaton, n Acoustcs, Speech, and Sgnal Processng, ICASSP 00. Proceedngs IEEE Internatonal Conference on, vol. 3. IEEE, 2000, pp [6] P. Jax and P. Vary, Artfcal bandwdth extenson of speech sgnals usng mmse estmaton based on a hdden markov model, n Acoustcs, Speech, and Sgnal Processng, Proceedngs.(ICASSP IEEE Internatonal Conference on, vol. 1. IEEE, 2003, pp. I 680. [7] B. Iser and G. Schmdt, Neural networks versus codebooks n an applcaton for bandwdth extenson of speech sgnals, n Eghth European Conference on Speech Communcaton and Technology, [8] G. Hnton, L. Deng, D. Yu, G. E. Dahl, A.-r. Mohamed, N. Jatly, A. Senor, V. Vanhoucke, P. Nguyen, T. N. Sanath et al., Deep neural networks for acoustc modelng n speech recognton: The shared vews of four research groups, Sgnal Processng Magazne, IEEE, vol. 29, no. 6, pp , [9] D. H. Ackley, G. E. Hnton, and T. J. Senowsk, A learnng algorthm for boltzmann machnes*, Cogntve scence, vol. 9, no. 1, pp ,

5 [10] G. Hnton, Tranng products of experts by mnmzng contrastve dvergence, Neural computaton, vol. 14, no. 8, pp , [11] A.-r. Mohamed, G. E. Dahl, and G. Hnton, Acoustc modelng usng deep belef networks, Audo, Speech, and Language Processng, IEEE Transactons on, vol. 20, no. 1, pp , [12] N. Enbom and W. B. Klen, Bandwdth expanson of speech based on vector quantzaton of the mel frequency cepstral coeffcents, n Speech Codng Proceedngs, 1999 IEEE Workshop on. IEEE, 1999, pp [13] J. Epps and W. H. Holmes, A new technque for wdeband enhancement of coded narrowband speech, n Speech Codng Proceedngs, 1999 IEEE Workshop on. IEEE, 1999, pp [14] J.-M. Valn and R. Lefebvre, Bandwdth extenson of narrowband speech for low bt-rate wdeband codng, n Speech Codng, Proceedngs IEEE Workshop on. IEEE, 2000, pp [15] P. J. Patrck and C. Xydeas, Speech qualty enhancement by hgh frequency band generaton, Dgtal processng of sgnals n communcatons, pp , [16] S. Vasegh, E. avarehe, and Q. Yan, Speech bandwdth extenson: extrapolatons of spectral envelop and harmoncty qualty of exctaton, n Acoustcs, Speech and Sgnal Processng, ICASSP 2006 Proceedngs IEEE Internatonal Conference on, vol. 3. IEEE, 2006, pp. III III. [17] T. Unno and A. McCree, A robust narrowband to wdeband extenson system featurng enhanced codebook mappng, n Acoustcs, Speech, and Sgnal Processng, Proceedngs.(ICASSP 05. IEEE Internatonal Conference on, vol. 1. IEEE, 2005, pp [18] N. A. T. Corporaton, Mult-lngual speech database for telephonometry, com/products e/speech, [19] J. S. Garofolo, L. F. Lamel, W. M. Fsher, J. G. Fscus, D. S. Pallett, and N. L. Dahlgren, Darpa tmt acoustc phonetc contnuous speech corpus cdrom, [20] S. Chennoukh, A. Gerrts, G. Met, and R. Sluter, Speech enhancement va frequency bandwdth extenson usng lne spectral frequences, n Acoustcs, Speech, and Sgnal Processng, Proceedngs.(ICASSP IEEE Internatonal Conference on, vol. 1. IEEE, 2001, pp [21] R. M. Gray, A. Buzo, A. Gray Jr, and Y. Matsuyama, Dstorton measures for speech processng, Acoustcs, Speech and Sgnal Processng, IEEE Transactons on, vol. 28, no. 4, pp ,