Exploring Convolutional Neural Network Structures and Optimization Techniques for Speech Recognition

Transcription

1 INTERSPEECH 2013 Exploing Convolutional Neual Netwok Stuctues and Optimization Techniques fo Speech Recognition Ossama Abdel-Hamid 1, Li Deng 2, Dong Yu 2 1 Depatment of Compute Science and Engineeing, Yok Univesity, Toonto, Ontaio, Canada 2 Micosoft Reseach, Redmond, WA USA ossama@cse.yoku.ca, {deng, dongyu}@micosoft.com Abstact Recently, convolutional neual netwoks (CNNs) have been shown to outpefom the standad fully connected deep neual netwoks within the hybid deep neual netwok / hidden Makov model (DNN/HMM) famewok on the phone ecognition task. In this pape, we extend the ealie basic fom of the CNN and exploe it in multiple ways. We fist investigate seveal CNN achitectues, including full and limited weight shaing, convolution along fequency and time axes, and stacking of seveal convolution layes. We then develop a novel weighted softmax pooling laye so that the size in the pooling laye can be automatically leaned. Futhe, we evaluate the effect of CNN petaining, which is achieved by using a convolutional vesion of the RBM. We show that all CNN achitectues we have investigated outpefom the ealie basic fom of the DNN on both the phone ecognition and lage vocabulay speech ecognition tasks. The achitectue with limited weight shaing povides additional gains ove the full weight shaing achitectue. The softmax pooling laye pefoms as well as the best CNN with the manually tuned fixed-pooling size, and has a potential fo futhe impovement. Finally, we show that CNN petaining poduces significantly bette esults on a lage vocabulay speech ecognition task. Index Tems: Convolutional Neual Netwok, Hybid Neual Netwok / Hidden Makov Models, Petaining, Convolutional Resticted Boltzmann Machine 1. Intoduction Recently, deep neual netwok hidden Makov model (DNN/HMM) hybid systems achieved emakable pefomance in many lage vocabulay speech ecognition tasks [1, 2, 3, 4, 5, 6, 7]. This is attibuted to the impoved modeling powe of the DNN that enables it to map complex pattens into class labels o posteio pobabilities. This modeling powe stems fom the deep-layeed stuctue and the distibuted epesentation. Moeove, unsupevised petaining of the DNN helps in achieving bette pefomance fo some tasks. This unsupevised petaining is typically done by stacking single layeed geneative models called Resticted Boltzmann Machine (RBM). Moe ecently, Abdel-Hamid et al. [8] showed that a special neual netwok stuctue called the convolutional neual netwok (CNN) can futhe impove the hybid model pefomance on the TIMIT phone ecognition task. The CNN exploits domain knowledge about featue invaiances within its stuctue and has been successfully applied to vaious image analysis and ecognition tasks [9, 10]. Fo speech pocessing, CNN was theoetically poposed (with no expeimental veification) in [11] but with convolution along the time axis to obtain Fequency bands Static,, Fames Convolution laye featue maps Shae same weights max pooling featue maps othe fully connected hidden layes Figue 1: A convolutional neual netwok with full weight shaing applied along the fequency axis. featues obust to small tempoal shifts. In [12], a geneative model named convolutional RBM (CRBM) was used to lean speech featues without supevision but with one dimensional convolution along time. In [8], it has been shown that much gain can be obtained by applying convolution and max-pooling along fequency. Applying convolution along fequency esults in featues invaiant to small fequency shifts of speech featues, which often happen between diffeent speakes and fo the same speake in diffeent moods. Moeove, in [13], a heteogeneous pooling stuctue poved to be beneficial. In this pape, we futhe exploe CNNs in multiple dimensions. We fist investigate diffeent achitectues of the CNN, including full and limited shaing of convolution filtes, convolution along fequency and time axes, and the stacking of multiple convolution layes. We then popose a novel weighted softmax pooling laye to enable automatic leaning of the pooling size. Lastly we evaluate the effect of CNN petaining, which is achieved by stacking tained CRBMs and RBMs [14]. The est of the pape is oganized as follows. In section 2 we intoduce the CNN and its vaious achitectues. In section 3 we descibe the weighted softmax pooling. The CNN petaining is discussed in section 4. We epot the expeimental esults in section 5, and conclude the pape in section The convolutional neual newok 2.1. Basic stuctue The CNN is a neual netwok with a special stuctue. Figue 1 illustates an example CNN with full weight shaing. In this CNN the fist laye, which consists of a numbe of featue maps, Copyight 2013 ISCA August 2013, Lyon, Fance

2 is called a convolution laye. Each neuon in the convolution laye eceives input fom a local eceptive field epesenting featues of a limited fequency ange. Neuons that belong to the same featue map shae the same weights (also called filtes o kenels) but eceive diffeent inputs shifted in fequency. As a esult, the convolution laye does a convolution of the kenels with the lowe laye activations. Suppose the NN input is V R A B, whee A is the numbe of featues epesenting an input fequency band and B is the numbe of the input fequency bands. In the case of filte bank featues, B epesents the size of the filte bank featue vecto. Let s assume that v = [v 1 v 2... v B], whee v b is the featue vecto epesenting band b. The activations of the convolution laye can be computed as: ( s ) h = θ wb,jv T b+k 1 + a j, (1) whee h is the convolution laye s output of the jth featue map on the kth convolution laye band, s is the filte size, w b,j is a weight vecto epesenting the bth band of the jth filte, a j is the bias of the jth featue map, and θ(x) is the activation function, which is set to the sigmoid function in this wok. A pooling laye is added on top of the convolution laye to compute a lowe esolution epesentation of the convolution laye activations though sub-sampling. The pooling function, which computes some statistics of the activations, is typically applied to the neuons along a window of fequency bands and geneated fom the same featue map in the convolution laye. In [8] a max pooling function, which simply computes the maximum value of the featue ove the coesponding fequency bands, was used. The max-pooling activations can be computed as: p j,m = max (h j,(m 1) n+k), (2) k=1 whee p j,m is the pooling laye s output of the jth featue map and mth pooling laye band, n is the sub-sampling facto, and is the pooling size, which is the numbe of bands to be pooled togethe. Both weight shaing and max pooling play a vital ole in achieving invaiance to small fequency shifts. This is a desied popety because, fo example, the fomants of the same phoneme may appea on slightly diffeent fequencies fo diffeent speakes o even fo the same speake in diffeent states. Moeove, weight shaing helps in educing ove-fitting due to the educed numbe of tainable paametes. The convolution-pooling pais can be stacked up to obtain highe level featues, on top of which the standad fully connected layes can be added to combine the featues of diffeent bands Full and limited weight shaing We call the weight shaing scheme in Figue 1 the full weight shaing. This is the standad scheme used fo image pocessing. It is suitable fo image pocessing because the same image patten can appea at any position in an image. In speech, howeve, diffeent pattens appea in diffeent fequency bands. Fo this eason, a limited weight shaing scheme as shown in Figue 2 is moe suitable. The diffeence hee is that a diffeent (not shaed) kenel is used fo the diffeent fequency window in the convolution laye. Each neuon in the pooling laye summaizes convolution laye s activations geneated fom one paticula featue map defined by the kenels. It is as if the convolution and pooling layes ae divided into many sections, each Fequency bands Static,, Fames Convolution laye featue maps max pooling laye nodes Shae same weights Figue 2: A convolution neual netwok with limited weight shaing applied along the fequency axis. of which pocesses only a limited ange of the input bands and geneates only one output band in the pooling laye. In mathematical tems, the convolution and pooling laye activations can be computed as: ( s ) h (m) = θ w (m) b,j vt (m 1) n+b+k 1 + a (m) j, (3) and whee h (m) p j,m = max k=1 (h(m) ), (4) is the value of the kth band of the jth featue map at the mth convolution laye section, and p j,m is the value of the jth featue of the mth pooling laye band. This scheme also helps educe the numbe of neuons in the pooling laye. Because each band uses kenels that conside only the pattens appeaing in the coesponding fequency ange, the numbe of kenels pe band is educed and the esulted epesentation has bette ability to distinguish pattens in diffeent fequency bands. On the othe hand, this limited weight shaing scheme has the disadvantage of peventing the addition of moe convolution layes on top. This is because the featues in diffeent pooling laye bands in this scheme ae unelated and cannot be convolved. An altenative solution, which may lead to bette pefomance, is to apply the limited weight shaing convolution laye on top of a full weight shaing one Convolution along the time axis The CNN achitectue can also be applied along the time axis to educe tempoal vaiability. In the hybid NN/HMM famewok, the HMM is supposed to handle the speech tempoal vaiability. Howeve, since the scoes of each fame ae computed fom a window of consecutive fames, invaiance to small shifts within this context window can be desiable. Note that the pooling and sub-sampling doesn t affect the time esolution seen by the HMM. It only affects the time esolution eceived by highe layes of the CNN. Combining convolution along both fequency and time axes geneates a 2D CNN simila to the ones used fo image analysis, and allows fo futhe pefomance impovement. Diffeent fom that in the image analysis, though, the convolution and pooling paametes along the fequency and time dimensions ae independent in the speech analysis case. This inceases the 3367

3 numbe of model paametes to optimize and equies caeful handling of these paametes. 3. Weighted softmax pooling At the same time when max-pooling intoduces obustness to small fequency shifts, it educes discimination ability to some featues in some bands, esp. when a fixed pooling size is used as in the CNN pesented in the pevious section. To alleviate this poblem, we popose adding leanable weights to the pooling laye. Moe specifically, instead of using a max-pooling function, we use a softmax pooling function which is diffeentiable. The softmax pooling eplaces Eq. 4 by: p j,m = k=1 exp(u j,m,k )exp(ɛh (m) ) i=1 exp(uj,m,i)exp(ɛh(m) j,i ) h(m), (5) whee u j,m,i epesents the logaithm of the pooling weights, and ɛ is a softmax function paamete that contols its smoothness. The pooling weights can be leaned using the backpopagation algoithm by computing the deivative of Eq. 5. By modifying the pooling weights, a simila effect to modifying pooling size can be obtained. 4. Petaining RBM-based petaining impoves DNN pefomance when the taining set is small. Fo convolutional stuctues, a convolutional RBM (CRBM) has been poposed in [14]. Simila to the RBM, the taining of the CRBM maximizes the taining data likelihood using the appoximate contastive divegence algoithm. In the CRBM, the convolutional laye activations ae stochastic. The CRBM model defines the pobability of a cetain configuation of input and hidden units and the conditional pobability of hidden unit activations given the visible units and vice vesa. The CRBM defines a multinomial distibution ove the pooled hidden nodes and at most one node in a pooled set can be active. The conditional pobability of activation of a hidden node h (m) given the CRBM input is defined by the following softmax function: P (h (m) =1 V) = exp(i(h (m) ) k=1 exp(i(h(m) ), (6) whee I(h (m) ) is the summation of weighted signal eaching node h (m) fom the input laye and it is defined as I(h (m) )= s w (m) b,j vt (m 1) n+b+k 1 + a (m) j. (7) The conditional pobability distibution of v i,b, the visible units of the bth band of the ith featue map, given the hidden unit states can be computed by the following Gaussian distibution: P (v i,b H) =N (v i,b ; j,(k,m) C(i,b) h (m) wm i,f(b,k,m),j,σ 2 ) (8) whee the mean of this Gaussian is the summation of the weighted signal aiving fom the hidden units that ae connected to the visible units. C(i, b) epesents this connection as the set of indices of convolution bands and sections that eceive input fom the visible unit v i,b. w m i,f(b,k,m),j is the weight on the link fom the bth band of the ith input featue map to the kth band and the jth featue map of the mth convolution section. σ 2 is the vaiance of the Gaussian distibution and it is a fixed model paamete. The weights of a tained CRBM ae good initial values fo the convolution laye. Afte the fist convolution laye weights ae leaned, they ae used to compute the convolution and pooling laye outputs using equations 3 and 4. The outputs of the pooling laye ae used as inputs to petain the next laye as done in deep belief netwok taining [15]. 5. Expeimental esults 5.1. Expeimental setup Expeiments wee done on two datasets: TIMIT and Micosoftintenal voice seach (VS). The featue extaction is simila. Speech was analyzed using a 25-ms Hamming window with a 10-ms fixed fame ate. The speech featue vecto was geneated by a Fouie-tansfom-based filte-bank, which includes 40 coefficients distibuted on a Mel scale, along with thei fist and second tempoal deivatives. Enegy was added to TIMIT featue vecto only when limited weight shaing is used, while VS has no enegy component. All speech data wee nomalized so that each coefficient has zeo mean and unit vaiance. The oiginal mean and vaiance wee computed once ove the whole taining dataset and wee used fo nomalizing test data as well. In all expeiments a NN with thee hidden layes is used. Each laye is eithe a fully connected laye with 1000 nodes o a convolution and pooling pai The TIMIT phone ecognition task In TIMIT, we used the 462-speake taining set and emoved all SA sentences (they ae two sentences ead by all speakes). A sepaate development set of 50 speakes was used fo tuning the models paametes and contolling the NN taining pogess (the leaning ate and the numbe of iteations). Results ae epoted using the 24-speake coe test set, which has no ovelap with the development set. We used 183 taget class labels (i.e., 3 states fo each one of the 61 phones). Afte decoding, the 61 phone classes wee mapped to a set of 39 classes as in [16] fo scoing. In ou expeiments, a bi-gam language model ove phones, estimated fom the taining set, was used in decoding. To pepae the NN tagets, a mono-phone HMM was tained on the taining dataset, and it was used to geneate state-level foced alignments. Fo neual netwok taining, a leaning ate annealing and an ealy stopping stategies wee utilized as in [15] The voice seach task VS is a lage vocabulay voice seach dataset containing 18 hous of speech data. Initially, a ti-phone HMM was built with state tying. The state labels wee used as the NN tagets. NN taining was done using a fixed ecipe. The fist 15 epochs wee un with a leaning ate of Anothe 10 epochs wee un with a leaning ate of Results on CNN stuctue In this set of expeiments we compae diffeent convolution stuctues on TIMIT. Table 1 shows the esults. With a full weight shaing CNN we got elative eduction in phone eo ate (PER) of moe than 5% compaed to the DNN without convolution. With limited weight shaing the elative eduction 3368

4 Table 1: Compaisons of TIMIT phone ecognition accuacy among diffeent CNN achitectues. LWS: limited weight shaing; FWS: full weight shaing; K: # of featue maps; PS: pooling size; FS: filte size; B: # of bands. Convolution achitectue PER No convolution 22.9 % Feq FWS (K:200, PS:6, FS:8, B:20) 21.6% Feq LWS (K:84, PS:6, FS:8, B:20) 20.5% Time FWS (K:400, PS:2, FS:8, B:7) 22.5% 2D Multi-layes (K:40, PS:2,2, FS:3,3, B:20,7), (K:200, PS:3,1, FS:5,7, B:18,1) 21.5% exceeded 10%. Convolution though time impoved the scoe as well, though, the impovement was much smalle. This indicates that convolution though fequency is moe impotant. Thee ae a numbe of possibilities fo the limited impovement with convolution though time. Fistly, the pooling and sampling may affect label localization though time because it deceases time esolution fo highe CNN layes. Thus it would be moe difficult fo highe layes to attibute the label to the cente fame. Secondly, the numbe of fames within the time window is only 15. This may decease the benefit of convolution along time as compaed with the 40 fequency bands. The last ow shows the esult of an attempt to use a 2D convolution with two convolution layes. Full weight shaing was used so that moe than one convolution layes can be stacked. A small impovement was obtained ove one laye of full weight shaing, but we think that with moe optimization of the paametes we can get a bette pefomance Results on weighted softmax pooling In this expeiment, we tested the pefomance of weighted softmax pooling. An achitectue simila to the limited weight shaing CNN used in the pevious sub-section was used hee. The pooling weights wee initialized with one. Then, the weights wee updated using the back-popagation algoithm. In one expeiments weights wee not tied and we got 20.8% PER. While in anothe expeiment, weights wee tied within thee goups whee each goup has 28 featue maps shaing the same pooling weights. In this case we got 20.4% PER, which is slightly bette than that achieved with max-pooling. Figue 3 shows the leaned weights. We notice that some bands eceive wide inputs than othes. This indicates that diffeent bands and featue maps ae moe suited to diffeent pooling sizes Results on Petaining To test the effects of petaining a CNN, we conducted a set of expeiments on TIMIT and VS datasets. The petaining implementation was based on the technique of [15] with detail descibed in Section 4, whee CRBMs ae used with convolution layes and RBMs ae used with fully connected layes. Table 2 shows the esults on the tasks of TIMIT (in PER) and VS (in wod eo ate (WER)). We obseve that the petaining impoves both the DNN and CNN except fo the CNN on TIMIT whee petaining didn t help. In geneal, the elative impovement of using petaining fo the CNN is less than that on the DNN. This may be attibuted to the bette stuctue of the CNN that educes the benefit of petaining. Figue 3: Leaned softmax pooling weights. Each column epesents a set of pooling weights. Each pooling band has thee sets of weights. The vetical axis epesents the input bands. Table 2: Effects of CRBM petaining (PT) Scoe Function TIMIT PER VS WER DNN, no PT 22.8% 37.1% DNN, PT 21.9% 35.4% CNN no PT 20.5 % 34.2% CNN PT 20.7 % 33.4% 6. Summay and conclusion In this wok we exploed multiple aspects of CNNs. Ou expeimental esults confimed the impoved pefomance of the CNN, peviously obtained on the TIMIT phone ecognition task, ove a DNN, and showed that simila pefomance impovement can be obtained on lage vocabulay tasks as well. The success of the CNN can be attibuted to the leaned featues that ae invaiant to small fequency shifts of speech pattens (e.g. fomants) which inceases the obustness to speake vaiations. Results also demonstated that both full and limited weight shaing achitectues pefom bette than the DNN without convolution. Howeve, the limited weight shaing scheme povides moe gains than the full weight shaing scheme. This is because distinct featue pattens ae expected to appea in each fequency band. Howeve, full weight shaing peseves consistent featues along all fequency bands, which can be futhe convoluted and pooled in highe convolution layes. Ou esults indicate that doing convolution along time, while outpefoming the DNN, pefoms significantly wose than doing it along fequency. We conjectue that the pefomance gap is caused by the implicit label shift and the smalle ange of the time dimension. The expeimental esults suggest that combining convolution along both fequency and time axes in a way that takes into account speech tajectoy confusion may bing moe pefomance gain. In this pape, we also poposed weighted softmax pooling to enable automatic detemination of the pooling size. Expeimental esults showed that diffeent featue-map goups oganized in diffeent bands can effectively lean diffeent pooling weight stuctues, some of which ae wide than othes. This obsevation indicates that sepaate pooling sizes and stuctues ae desied fo diffeent featues and bands. The ecognition pefomance epoted in this wok is only slightly bette than that achieved using the limited weight shaing CNN. Howeve, we believe this is an inteesting diection and bette leaning algoithms may futhe impove the pefomance. Finally, we poposed to use CRBM to petain the convolution layes in the CNN. We evaluated the CNNs with and without petaining and found out that petaining impoved pefomance on a lage vocabulay speech ecognition task. 3369

5 7. Refeences [1] G. Dahl, D. Yu, L. Deng, and A. Aceo, Context-dependent petained deep neual netwoks fo lage-vocabulay speech ecognition, IEEE Tans. Audio, Speech, and Language Poc., vol. 20, no. 1, pp , jan [2] F. Seide, G. Li, and D. Yu, Convesational speech tansciption using context-dependent deep neual netwoks, in Poc. Intespeech, [3] T. N. Sainath, B. Kingsbuy, B. Ramabhadan, P. Fousek, P. Novak, and A. Mohamed, Making deep belief netwoks effective fo lage vocabulay continuous speech ecognition, in ASRU, [4] G. Hinton, L. Deng, D. Yu, G. Dahl, A. Mohamed, N. Jaitly, A. Senio, V. Vanhoucke, P. Nguyen, T. Sainath, and B. Kingsbuy, Deep neual netwoks fo acoustic modeling in speech ecognition: The shaed views of fou eseach goups, IEEE Signal Pocessing Magazine, vol. 29, no. 6, pp , nov [5] L. Deng, J. Li, J.-T. Huang, K. Yao, D. Yu, F. Seide, M. Seltze, G. Zweig, X. He, J. Williams, Y. Gong,, and A. Aceo, Recent advances in deep leaning fo speech eseach at Micosoft, in Acoustics, Speech and Signal Pocessing (ICASSP), 2013 IEEE Intenational Confeence on, May [6] S. M. Siniscalchi, D. Yu, L. Deng, and C. hui Lee, Speech ecognition using long-span tempoal pattens in a deep netwok model, IEEE Signal Pocessing Lettes, Mach [7] D. Yu, L. Deng, and F. Seide, The deep tenso neual netwok with applications to lage vocabulay speech ecognition, IEEE Tansactions on Audio, Speech, and Language Pocessing, vol. 21, no. 2, pp , [8] O. Abdel-Hamid, A. Mohamed, H. Jiang, and G. Penn, Applying convolutional neual netwoks concepts to hybid NN-HMM model fo speech ecognition, in Poc. ICASSP, mach 2012, pp [9] Y. LeCun, F. Huang, and L. Bottou, Leaning methods fo geneic object ecognition with invaiance to pose and lighting, in Poceedings of CVPR 04. IEEE Pess, [10] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffne, Gadient-based leaning applied to document ecognition, in Poceedings of the IEEE, 1998, pp [11] Y. LeCun and Y. Bengio, Convolutional netwoks fo images, speech, and time-seies, in The Handbook of Bain Theoy and Neual Netwoks, M. A. Abib, Ed. MIT Pess, [12] H. Lee, P. Pham, Y. Lagman, and A. Ng, Unsupevised featue leaning fo audio classification using convolutional deep belief netwoks, in Advances in Neual Infomation Pocessing Systems 22, 2009, pp [13] L. Deng, O. Abdel-Hamid, and D. Yu, A deep convolutional neual netwok using heteogeneous pooling fo tading acoustic invaiance with phonetic confusion, in Acoustics, Speech and Signal Pocessing (ICASSP), 2013 IEEE Intenational Confeence on, May [14] H. Lee, R. Gosse, R. Ranganath, and A. Y. Ng, Convolutional deep belief netwoks fo scalable unsupevised leaning of hieachical epesentations, in Poceedings of the 26th Annual Intenational Confeence on Machine Leaning, se. ICML 09. New Yok, NY, USA: ACM, 2009, pp [15] A. Mohamed, G. Dahl, and G. Hinton, Acoustic modeling using deep belief netwoks, IEEE Tans. Audio, Speech, and Language Pocessing,, vol. 20, no. 1, pp , jan [16] K. F. Lee and H. W. Hon, Speake-independent phone ecognition using hidden makov models, IEEE Tansactions on Audio, Speech and Language Poc., vol. 37, no. 11, pp , Novembe