Phoneme Probability Estimation with Dynamic Sparsely Connected Artificial Neural Networks

Transcription

1 The Free Speech Journal, Issue # 5(1997) Publshed 10/22/ All rghts reserved. Phoneme Probablty Estmaton wth Dynamc Sparsely Connected Artfcal Neural Networks Nkko Ström, (nkko@speech.kth.se) Department of Speech, Musc and Hearng, KTH, Stockholm, Sweden Centre for Speech Technology, KTH, Stockholm, Sweden Abstract Ths paper presents new methods for tranng large neural networks for phoneme probablty estmaton. An archtecture combnng tme-delay wndows and recurrent connectons s used to capture the mportant dynamc nformaton of the speech sgnal. Because the number of connectons n a fully connected recurrent network grows super-lnear wth the number of hdden unts, schemes for sparse connecton and connecton prunng are explored. It s found that sparsely connected networks outperform ther fully connected counterparts wth an equal number of connectons. The mplementaton of the combned archtecture and tranng scheme s descrbed n detal. The networks are evaluated n a hybrd HMM/ANN system for phoneme recognton on the TIMIT database, and for word recognton on the WAXHOLM database. The acheved phone error-rate, 27.8%, for the standard 39 phoneme set on the core test-set of the TIMIT database s n the range of the lowest reported. All tranng and smulaton software used s made freely avalable by the author, and detaled nformaton about the software and the tranng process s gven n an Appendx.

2 Table of contents Abstract Introducton Basc theory Feed-forward networks Recurrent connectons and tme-delay Back-propagaton through tme Weght updatng scheme Weght ntalzaton Interpretaton of the output actvaton values The softmax output actvaton functon Phoneme probablty estmaton Input feature representaton Delta coeffcents Input normalzaton Network topology Dynamc decodng Prunng and sparse connecton Connecton Prunng Sparse connecton Recognton results Phoneme recognton results on the TIMIT database Experments wth varyng connectvty Decouplng the output unts Varyng the hdden layer sze Connecton prunng n traned networks Computatonal consderatons Test of the nterpretaton of actvtes as a posteror probabltes Word recognton results on the WAXHOLM human/machne dalog task Conclusons Acknowledgments References...34 Appendx: The NICO neural network toolkt...38 A.1 Structurng the database...38 A.2 Feature extracton...39 A.3 Generaton of phoneme targets...39 A.4 Specfyng the network structure...39 A.5 Tranng...41 A.6 Frame-level evaluaton...41 A.7 Connecton prunng

3 1 Introducton Speech recognton was one of the frst problems that artfcal neural networks (ANNs) were appled to durng the rapd spread of ANN models n the 1980 s. ANNs have been used for recognton of large unts lke words drectly (e.g. Ström, 1992; Englsh and Boggess, 1992; L, Naylor, and Rossen, 1992), but attempts to recognze smaller unts lke phonemes have been more successful. Several dfferent methods exst for combnng the ANN classfcaton of sub-unts nto sequences that consttutes words. A few of the more well-known methods are: Hybrd HMM/ANN Archtecture (Bourlard and Wellekens, 1988), Lnked Predctve Neural Networks (Tebelsks and Wabel, 1990), Hdden Control Neural Archtecture (Levn, 1990), and Stochastc Observaton HMM (Mtchel, Harper, and Jameson, 1996). Of the mentoned methods, the hybrd HMM/ANN archtecture s the most wde-spread today. ANN systems undsputedly solve some problems better than all other methods n automatc speech recognton. The phoneme recognton rate on the TIMIT database reported by Robnson (1994) s several percent hgher than that of all other systems a large dfference for ths type of test (a comparson of dfferent approaches s gven n Table 3, secton 5.1). ANN solutons are also typcally compact,.e., have a small number of parameters, and offer fast decodng compared to standard HMM systems. Stll, for the last years, the research actvtes on ANNs for speech recognton have not by far been as ntensve as for the prevalng HMM paradgm. Preference for a well-establshed technology and the relatvely few equally excellent results reported on the word-level are possble reasons for the mld nterest n ANN solutons. However, the evaluaton of the SQALE-project (Steeneken and van Leeuwen, 1995) s an example of a case where hybrd HMM/ANN technology sgnfcantly outperforms stateof-the-art HMM systems provded by leadng research stes for large vocabulary tasks. A dfferent reason for the lmted spread of ANN solutons can be problems wth the network tranng,.e., effcently and robustly determnng the parameters of large networks. For example, the recurrent network used by Robnson s traned wth specal parallel hardware, and a rather complex tranng heurstc s used wth several ad hoc parameters to be determned emprcally. Recurrent connectons are of course not the only path to good results, but other exstng solutons have dfferent problems. An ANN archtecture wthout recurrent connectons s used wth good results by Bourlard and Morgan (1993). Instead they use tmedelay wndows (Wabel et al., 1987) to capture the temporal cues of the speech sgnal. The lack of recurrent connectons make the tranng algorthm more robust, but very large networks are used to acheve good results, and therefore the avalable computng resources lmt the performance of the system. Contemporary hgh performng ANN solutons typcally requre more computaton for tranng than the wde-spread maxmum lkelhood (ML) tranng for the standard HMM (Rabner and Juang, 1993; Lee, 1989). Comparng standard HMM and hybrd HMM/ANN tranng n more detal, a dfference n computatonal complexty can be noted. The amount of computaton ncreases faster for ANN tranng when the sze of the model s ncreased. When more tranng data s avalable for tranng an HMM system, typcally more context-dependent models are ntroduced. However, n a somewhat smplfed vew 1, the tranng tme s ndependent of the number of models, and proportonal to the amount of tranng data. Ths s because each model s traned only on a partton of the data. In the case of an ANN system, the capacty s determned by the number of hdden unts, and when more tranng data s 1 Ths s not true f the free parameters of the HMM are ncreased by some other means, e.g., ncreasng the number of components per Gaussan mxture. We have also made the approxmaton that the Baum-Welsh algorthm converges n the same number of teratons ndependent of the sze of the tranng data, and on the number of context dependent models. 3

4 avalable, more hdden unts are typcally ntroduced. The computaton for tranng ncreases faster than for HMM tranng, because the entre ANN s traned on all data. Agan n a smplfed vew 2, the tranng tme s proportonal to the product of the amount of tranng data and the sze of the ANN (the number of connectons). Thus, n comparson wth HMM tranng, the amount of computaton for ANN tranng s more dependent on the model sze. In partcular for recurrent networks ths s a problem because the number of recurrent connectons grows quadratcally wth the number of hdden unts. In part, the scalng propertes of ANN tranng orgnate from the nherent dscrmnatve nature of the tranng, but ths s not the whole truth. In ths paper we propose new methods for robust tranng of large, hgh performance ANNs based on sparsely connected networks. In a sparsely connected ANN, the hdden layer can be allowed to grow wthout necessarly ncreasng the number of connectons proportonally. The problem wth the quadratc relaton between unts and connectons n recurrent networks s addressed by ntroducng local connectvty for the recurrent connectons. The local connectvty has the effect that unts close to each other have hgher lkelhood of beng connected. Thus, t promotes the development of groups of unts performng sub-tasks of the problem. The sparse connecton schemes used, and a related concept, connecton prunng, are dscussed n secton 4. In secton 2, a revew of the theory of feed-forward ANNs s gven together wth the defnton of the partcular network archtecture used n ths paper, and detals of the tranng algorthm. Recognton results on the TIMIT and WAXHOLM databases are reported n secton 5. The HMM paradgm has currently the advantage of a large mature body of easly avalable software (e.g., Young et al., 1995). To promote further development n the hybrd HMM/ANN feld, and to make reproducton of our results easer, the software toolkt used for tranng and runnng the neural networks of ths study s made freely avalable. In the appendx, the software s descrbed and nstructons on how to obtan a copy are gven. Informaton about the smulatons of ths study s gven n detal to make reproducton of the results possble. 2 Basc theory Ths secton descrbes the basc theory behnd the dynamc feed-forward artfcal neural networks used n the study. Because we wsh to make replcaton of the results straghtforward, the descrpton s rather detaled. Most of the materal s well-known backgroundknowledge, covered n textbooks on ANN computng (e.g., Bshop, 1995 or Rpley, 1996). Detals of the unfed mplementaton of recurrent unts and tme-delay wndows (Secton 2.2) as well as the ntroducton of multplcaton unts and unts wth several dfferent nonlneartes can be of nterest also for readers wth some experence n the feld. 2.1 Feed-forward networks Feed-forward networks are drected acyclc graphs,.e., the actvtes of all artfcal neurons, hereafter called unts, can be computed n one teraton, and there s no feedback that would make the actvaton of a unt depend recursvely on ts own value. A network conssts of nput unts, hdden unts and output unts and connectons between them. In the popular analogy wth a bologcal nerve-system, the nput unts are sensors whose values are determned by the envronment, the hdden unts corresponds to nternal neurons and the output unts can be thought of as neurons of a motor system, controllng the organsm s responses to the nput from the envronment. Ths analogy can be very nsprng, but n practce feed-forward ANNs are used smply to approxmate complex nput/output maps, wth no known explct formula 2 Ths s true only f the number of tranng epochs needed for convergence of the tranng s ndependent of the ANN s sze, and on the amount of tranng data. Ths matter s dscussed further n secton

5 but a great deal of characterzng data. The strength of ANN models s the weak constrants they put on the mappng; It has been shown that, gven a suffcent number of hdden unts and characterzng data, feed-forward ANNs wth one layer of hdden unts can approxmate any bounded functon on a compact set wth arbtrary accuracy (e.g., Hornk, Stnchcombe and Whte 1989). In practce however, the performance s lmted due to: ) problems fndng the set of connecton weghts that gves the optmal network and ) the sze of the avalable database. In ths paper we concentrate on mappngs defnng a 1-of-N classfcaton of the nput patterns. Each nput pattern of the tranng database s assgned to one of N classes. The target values for the N output unts are -1.0 for all unts except the one correspondng to the correct class whose target s The N classes are the phonemes and the nput patterns are feature representatons of the short-tme spectra of the sound wave. Ths wll be covered more thoroughly n secton 3. Except for some specal unts, actvtes n our framework are computed n the same way as the classc ANNs of (Rumelhart, Hnton and Wllams, 1986), but the sgmod functon s replaced by the computatonally more convenent tanhyp functon. Thus, for the tanhyp unts, the actvaton a of unt s defned by: a ( net ) = tanh (1) where net = w ja j (2) j and w j s the connecton weght from unt j to unt. It s easy to show that besdes the change to an actvaton functon wth a symmetrc range [-1;1], ths s equvalent to a lnear transformaton of the weght space. The tanhyp functon can be expanded as follows: x tanh = e e x x 2 2 e x x e Thus, the transformaton, a w = 2a 1 j = 2w, RHW j, RHW x 2 e + 1 = e + e 1 = sgmod( x) 1 + = e x x x 2 2 where RHW ndcates parameters of the classc network, transforms a classc network based on sgmod unts to a network wth tanhyp non-lnearty. It s sometmes convenent to work also wth unts wthout any non-lnearty and unts wth other non-lneartes. Because these unts are not defned n the classcal networks, the weghts of connectons to them are not affected by the transformaton between our actvtes and the classcal RHW doman of (4). In the experments of ths study we use lnear unts, exponental unts, nverter unts (1/x) and multplcaton unts n addton to the usual tanhyp and nput unts. Moreover, each network has one specal-purpose unt that has a constant actvty of 1.0. Connectons are by default added from ths unt to all tanhyp unts the effect s dentcal to that of a unt-bas. Multplcaton unts are slghtly more complcated than the other types and requre the computaton of (6) nstead of (2). In summary, unt actvtes are computed by (3) (4) 5

6 ( ) a net = tanh f unt s a tanhyp unt a = net f unt s a lnear unt a = 10. f unt s the bas unt a = clamped f unt s an nput unt a ( net ) = exp f unt s an exponental unt a = 1/ net f unt s an nverter unt a = prod f unt s a multplcaton unt where prod = a j j,unt j feeds nto unt (5) (6) n analogy wth (2). The target values for the output unts are 1.0 for the unt correspondng to the correct class and -1.0 for all other unts. The objectve functon for the back-propagaton tranng s based on the cross-entropy dstance n the RHW doman (Solla, Levn, and Flesher, 1988). If the target output actvaton for unt s τ,rwh n the RHW doman, the contrbuton, e, to the cross entropy transformed to the tanhyp doman can be computed from (4): ( 1 τ, ) log( 1, ) τ, log(, ) e = a + a = RWH RWH RWH RWH τ + 1 a + 1 τ + 1 a + 1 = 1 log 1 + log = τ 1 a = log a τ + 1 a + 1 log f τ = log = 2 2 log a + 1 f τ = 1 2 The objectve functon, E, s the sum of e for all unts and nput patterns, so the dervatve wth respect to the actvty s de da 0 = 1 / / 1 ( a ) ( a ) when the unt s not an output unt when the unt corresponds to the correct class otherwse Now we compute the dervatve wth respect to the connecton weghts n the standard way usng the chan rule to get the recursve equatons: (7) (8) 6

7 ( 1 )( 1+ ) a a backnet f s a tanh unt 1 E a backnet 2 f s a nverter unt δ = = net exp( a ) backnet f s a exponental unt f s an nput, lnear, backnet multplcaton or the bas unt where backnet de a = + δ j wj + δ j da a j unt j s not a multplcaton unt j unt j s a multplcaton unt j (9) (10) and the dervatves wth respect to the weghts are: E w j = δ a j The dervatves of (11) can be used for gradent decent type mnmzaton of E. The detals of ths are flled out n secton 2.4. Note that f an output unt has no out-flowng connectons, then de/da cancels one of the factors of δ n (9), and the resultng δ s smply the dfference between a and the target (+1.0 for the correct class and -1.0 otherwse). 2.2 Recurrent connectons and tme-delay Dynamc features of speech such as formant movements, that are known to be of mportance for phoneme classfcaton (e.g., Fant 1969), are not captured by the short tme spectrum representaton used as nput to the network. Therefore, phonetc classfcaton of short-tme spectra can be greatly enhanced by consderng also the context of neghborng spectra. A step n ths drecton was taken by Wabel et al. (1987) when they ntroduced tme-delay neural networks (TDNN). In ths paper we denote by TDNN, all network archtectures where the unts are connected to unts n lower layers wth tme-delayed connectons so that the actvtes depends on the actvtes of lower layer unts n a fnte tme-delay wndow (Fgure 1 (left)). The frst experments wth TDNN successfully showed an mproved classfcaton of stop consonants where t s well known that the dynamc formant patterns are of great mportance. Later the archtecture has been successfully appled to complete phoneme nventores and used n hybrd HMM/ANN speech recognton systems wth good results (e.g., Bourlard and Morgan, 1993; Cohen et al., 1992). A dfferent course to nclude the context n the classfcaton s to connect unts n the same layer wth a delay of one tme-step so called recurrent connectons (Fgure 1 (mddle)). The network stll remans a feed-forward network because the recurrent connectons are delayed. Ths approach dffers from TDNN n that the actvty of a unt at a partcular tme depends recursvely on actvtes n ts layer and lower layer at all prevous tmes. Networks wth recurrent connectons are called recurrent neural networks (RNN) (Rumelhart, Hnton and Wllams, 1986) or dynamc neural networks (Pearlmutter, 1990) and ths s currently the most successful archtecture for phoneme recognton (Robnson and Fallsde, 1991; Robnson, 1994). TDNNs and RNNs have much n common; n partcular, both use tme-delayed connectons to ncorporate context nto the classfcaton. In fact, f the connectons of RNNs (11) 7

8 are allowed to have multple tme-delays nstead of just one tme-step, the resultng network has all the modelng power of both archtectures. Ths unfed archtecture, RTDNN (Recurrent Tme-Delay Neural Network) ntroduced by Ström (1992), s used n ths study (Fgure 1 (rght)). The delayed connectons have the effect that the response of an nput pattern s delayed several tme-ponts n the output unts. One feasble way to tackle ths problem s to delay also the target values for the output unts (Robnsson and Fallsde, 1991; Robnsson 1994). In the RTDNN framework, we have chosen to use look-ahead connectons nstead of delayng the targets. Look-ahead connectons let a unt depend on the actvty of other unts at future tmes (ndcated by a postve superscrpt, e.g., z +1, n Fgure 1). Ths concept s often used n TDNN archtectures, but not possble to mplement n RNN archtectures because recurrent connectons must be delayed n feed-forward networks. Look-ahead connectons force the computaton of some unt actvtes to be delayed, but the network s stll a feed-forward network as long as no unt s actvty at a partcular tme depends on ts own actvty. output output output z -1 z +1 z-1 z +1 z -1 z -1 z -3 z -2 z -1 z +1 z +2 z +3 z -1 z+1 z +2 nput nput nput Fgure 1. Dfferent types of dynamc networks. The z -x operator ndcates that the connecton s delayed x tme-frames. For smplcty, the nput, hdden, and output layers have only one unt each n ths fgure. Left: tme-delay network (TDNN). The unts n hgher layers have access to a tme-delay wndow of the actvtes of unts n lower layers. Note that both tme-delay and look-ahead (z +x ) connectons are used. A consequence of ths s that computaton of actvtes n unts n hgher layers must be delayed untl the actvtes of the look-ahead connectons are known. Mddle: Recurrent network (RNN). The hdden unts are recurrently connected back to the hdden layer. In ths archtecture, the actvtes recursvely depend on the actvtes of all prevous tme-frames. Rght: The combned archtecture (RTDNN) wth both tme-delay wndows, and recurrent connectons. It s not hard to show that any feed-forward network wth look-ahead connectons has an equvalent network wth no look-ahead, but delayed targets nstead. In the pseudo-code of Table 1 we show ths by constructng such an equvalent network, and n the process we also get the order n whch unt actvtes must be computed. In fact, the constructed equvalent network s the one used n the actual computer smulatons. The reason for ntroducng the extra complcaton of workng wth two separate network representatons s that look-ahead connectons are more ntutve and smplfy the network desgn. For example, f one decdes to use a wder tme-delay wndow or change the dynamc 8

9 structure n some other way, t s unnecessary to select or compute a new approprate delay for the target values. Ths s nstead handled automatcally n the converson process outlned n Table 1. In the smulatons, the target values are delayed because that s the computatonally most advantageous representaton, but ths s hdden from the network desgner who can focus on selectng a dynamc structure sutable for the partcular problem. Let all unts of the new, equvalent network have a new property called delay and ntalze t to zero for all unts. Let the delay of unt be d. Let count = 0 do { for each unt { for each connecton w j,d n the orgnal network flowng to unt from unt j wth delay d { f d j - d > d then let d = d j + d } } Let count = count +1 } untl no delay changes durng a whole teraton or count s greater than the total number of unts If the loop was termnated because count grew larger than the number unts, the network cannot be a feed-forward network and thus the desgn s n error. Ths check should be made each tme new connectons are added to a network. Sort the unts n order of ncreasng delay. Ths s the order n whch the unts are computed n the smulatons. Modfy the delays of the equvalent network so that for each connecton w j,d n the orgnal network, the correspondng connecton s w j,d, where d = d - d j + d,.e., the dfference n the two unt s delays s taken nto account. It s easy to see from the constructon of the unt delays that no connectons n the equvalent network can have negatve delay (lookahead). Target values for output unts are delayed by the respectve unt s delays. Table 1 Algorthm for convertng a feed-forward network wth look-ahead connectons to one wth delayed targets. A check s also made that the network s ndeed a feed-forward one; f the varable count grows larger than the number of unts n the network, there must be some loop n the network that allows a unts actvty to depend on ts own value, and the network s therefore not a feed-forward network. The equatons of secton 2.1 must be generalzed to take dynamc connectons nto account. In the RTDNN framework, (2), (5) and (6) are generalzed to 9

10 ( t ) a net, t = tanh f unt s a tanhyp unt, a, t = net, t f unt s a lnear unt a, t = 10. f unt s the bas unt a, t = clamped f unt s an nput unt a ( net ), t = exp f unt s an exponental unt, t a, t = 1 / net, t f unt s an nverter unt a, t = prod, t f unt s a multplcaton unt net, t = w jd a (13) j,( t d ) j (12) prod a, t = j,( t d ) j, unt j feeds nto unt (14) where w jd s the connecton weght for the connecton from unt j to unt wth delay d (possbly negatve for look-ahead) and a,t s the actvty of unt at tme t. 2.3 Back-propagaton through tme In the prevous secton the forward equatons were extended to allow for tme-delayed connectons n a rather straght-forward fashon. In ths secton we focus on the backward equatons,.e., the computaton of the dervatves of the objectve functon, E, wth respect to the connecton weghts. Followng the formalsm of the prevous secton, (9), (10) and (11) can be generalzed as follows: δ, t where ( 1 )( 1+ ) a, t a, t backnet, t f s a tanh unt 1 2 backnet, t f s a nverter unt E a, t = = net, t exp( a, t ) backnet, t f s a exponental unt f s an nput, lnear, backnet, t multplcaton or the bas unt backnet, t de = + δ w, + δ da, t j, d unt j s not a multplcaton unt j + d j d j + d j, d unt j s a multplcaton unt where the two sums together have one term for each connecton flowng out from unt. Further, the dervatves wth respect to the connecton weghts are generalzed to: a j, t + d a, t (15) (16) 10

11 E w j, d t1 = δ + a t = t0 j, t d, t (17) where t s the tme ndex of actvtes and t 0 and t 1 are the boundares of the sequence. Recall that n the network used n the computer smulatons, all delays d are non-negatve. Ths has the consequence that (15) descrbes a recursve set of equatons where the δ s are computed n reverse order of tme, t, hence the name back-propagaton through tme, (Rumelhart, Hnton and Wllams, 1986; Pearlmutter, 1990). A way to vsualze back-propagaton through tme s to draw the spatal dmenson of the network n one dmenson, e.g., lne up all unts n one column. Then unfold the network n the tme dmenson,.e., draw one column of unts for each tme pont. Fgure 2 shows a very smple example of such an unfolded network. The unfolded verson of the network s structurally smlar to a network wth no delays but as many layers as there are tme ponts. The mportant dfference s that the connecton weghts are shared by all connectons that correspond to the same connecton n the orgnal network. Back-propagaton through tme s equvalent to normal back-propagaton wth ths addtonal constrant on the weghts. z -1 b d c d c b d c b d c b d c b d c b z -2 z -1 z a -1 a a a a a tme Fgure 2. A smple dynamc network (left) and the same network unfolded n tme (rght) where the nodes a-d are duplcated for each tme pont. Arcs labeled z -x ndcates that the connecton s delayed x tme ponts. It s easy to see n the rght fgure that the network s feed-forward because all arcs flow from bottom left to upper rght. 2.4 Weght updatng scheme Tranng an ANN usng the back-propagaton paradgm s an optmzaton problem,.e., fndng the set of connecton weghts that mnmzes the objectve functon E. The backward equatons (15) -(17), provde us wth the dervatves of the objectve functon wth respect to the connecton weghts whch makes gradent descent methods feasble. However, gradent descent optmzaton s a very broad class of methods and t s the partcular weght updatng scheme (based on the gradent) that determnes the success or falure of an mplementaton of the algorthm. In our experence, the classc stochastc weght updatng scheme by Rumelhart, Hnton and Wllams (1986), wth some modfcatons, contnues to be a good choce for problems wth a large amount of tranng data. It can be wrtten 11

12 wj = 0 ( n) ( n 1) E wj = η wj + γ wj ( n+ 1) ( n) ( n) w = w + w j ( 0) j j (18) where superscrpt (n) ndcates a parameter after teraton number n and γ and η are the gan and momentum parameters respectvely. There are many well-known methods that utlze curvature nformaton for the optmzaton. General optmzaton methods, e.g., Newton s method or conjugate gradent methods (see for example Luenberger, 1984) can be appled, as well as more or less specalzed methods for ANN-tranng lke QuckProp (Fahlman 1988), and applcaton of Levenberg/Marquardt s method, Levenberg (1944), Marquardt (1963). However, t s nontrval combne these methods wth stochastc approxmaton algorthms, where weghts are updated before the whole tranng data s processed (an epoch). In the smple updatng scheme of (18), networks wthout delayed connectons can be updated after every nput/output pattern so called pattern updatng. Although the updatng s based on an approxmaton of the gradent computed from only one pattern, the algorthm wll stll converge f the gan s small enough (and n many cases much faster than wth epoch updatng). The pcture becomes more complcated n the case of back-propagaton through tme because the dervatves depend not only on the current pattern, but on the whole sequence of patterns. We have adopted the approxmate scheme to update the weghts every N frames,.e., approxmate the dervatve based on sub-sequences of the tranng data. Ths method has also been used by Robnson (1994) and s descrbed n more detal n Table 2. The gradent computed n ths manner s not only an approxmaton because t s based on a small number of tme-ponts t s also approxmate because the δ s of (15) actually depend on the unt actvtes at all followng ponts (not just the ones computed so far). The approxmaton s clearly worse f the weghts are updated more frequently, but on the other hand t s desrable to update the weghts as often as possble to speed up the process. In the smulatons of ths study, weghts are updated every tme ponts. Ths choce was made after some prelmnary experments and s ntutvely reasonable as t corresponds roughly to the length of a syllable. It s also a number smlar to that used by Robnson (1994). The exact number of frames between each update s chosen randomly from a square dstrbuton between 20 and 30, ths has the effect that the ponts of update are dfferent from epoch to epoch. 1) Let t 0 = 1 2) Let step be an nteger randomly chosen from the square dstrbuton [20, 30]. 3) Let t 1 = t 0 + step 4) Compute unt actvtes usng (12) and (13) wth the gven t 0 and t 1 5) Compute dervatves backwards from t 1 to t 0 usng (15) - (17). 6) Update connecton weghts accordng to (18). 7) Let t 0 = t ) Go to step 2 Table 2. Weght updatng scheme for back-propagaton through tme. The random steplength n step 2 makes the update ponts dffer from epoch to epoch. We have stll not dscussed the choce of the parameters γ and η. An oversght n the famous work by Rumelhart, Hnton and Wllams (1986) s that they let γ and η be constants. It s well-known from statstcs theory that back-propagaton tranng wth stochastc updatng, 12

13 converges to a local mnmum of E, only f a few constrants on the decay of the gan parameter are fulflled. An survey of results n the statstcal analyss of ANN learnng schemes s gven by Whte (1989). In secton 4.2 of hs survey, the statstcal propertes of stochastc updatng s dscussed and necessary condtons for convergence s gven. In another study, Juang and Katagr (1992) gve the followng condtons for convergence: n= 0 γ ( n) = ( n) 2 [ γ ] n= 0 < (19) where superscrpt (n) agan ndcates the parameter after teraton number n (e.g Also n practce s t frutful to let the gan parameter decrease durng the optmzaton. We have combned ths feature wth cross-valdaton n a manner smlar to Bourlard and Morgan (1993). The dea s to decrease the gan parameter every tme the objectve functon fals to decrease on the valdaton set. To be more specfc, the tranng data s parttoned nto a tranng set and a smaller valdaton set. The tranng set s used for back-propagaton tranng wth weght updatng accordng to (18), but after each epoch, the objectve functon s computed for the valdaton set too. Ths s done usng only the forward equatons,.e., wthout updatng the weghts. The objectve functon for the valdaton set s recorded for each epoch and whenever t fals to decrease, the gan parameter γ s multpled by a constant factor α < 1. In ths study α s always 0.5. We apprecate that the decay of the gan parameter and the cross valdaton procedure are two separate concepts and that t would therefore be more elegant to control them ndependently. However, n practce the two are closely related, and the descrbed strategy have resulted n fast accurate optmzaton for the phoneme probablty estmaton task. The momentum parameter, η [0, 1], controls the smoothng of the gradent estmates, and can have a consderable effect on the convergence rate. In the smulatons presented below, η s always Weght ntalzaton It s clear that the ntal values of the connecton weghts are mportant for the performance of the back-propagaton tranng. The algorthm fnds one partcular local mnmum of the objectve functon, and the partcular mnmum found depends heavly on the startng pont n the search space,.e., the ntal connecton weghts. It s common practce to ntalze the weghts to small random numbers (e.g., Fahlman, 1988). Ths mples that sgmod and tanhyp unts operate n the lnear regon of the non-lnearty. In our experments, the weghts are ntalzed to square dstrbuted random numbers [-0.1; 0.1] (but see also secton 3.2). 2.6 Interpretaton of the output actvaton values It seems ntutvely clear that reducng the error functon E mproves the classfcaton performance of the ANN. However, t s essental for the understandng of the ANN classfer to formalze ths noton. For our needs, t seems sound to defne the best possble classfer to be the Bayesan dscrmnant functon. Any functon that mplements the classfcaton procedure: assocate the nput observaton wth the class that has the hghest a posteror probablty, consttutes the Bayesan dscrmnant functon. One obvous mplementaton s to accurately estmate the a posteror probabltes for each class and then select the most probable class. In ths case, the degree to whch a classfer succeeds n ts task depends on the 13

14 accuracy n the estmaton of the a posterors. Justfyng nterpretng the output actvtes as a posteror probabltes s fundamental for the theoretcal foundaton of the hybrd ANN/HMM speech recognton paradgm dscussed n secton 3.5. It was proved early on that tranng networks wth the mean square error (MSE) objectve functon s equvalent to mnmzng the MSE of the a posterors. Duda and Hart (1973) formulated the proof for the smple perceptron and t was later extended to mult-layer perceptrons by a number of authors: Baum and Wlczek (1988), Bourlard and Wellekens (1988), Rchard and Lppman (1991), and Gsh (1990). The proofs are vald under the condton that the functonal capacty of the network exceeds the functonal complexty of the a posterors. In our smulatons, the cross-entropy error functon s used. An analogous relatonshp between the a posteror probabltes and the output actvtes of networks traned wth the cross-entropy objectve functon was gven by Hampshre and Pearlmutter (1990). They show that the MSE objectve functon s just a specal case of a class of reasonable error measures that yeld networks wth output unts that converge to the a posteror probabltes P(c o). Here we outlne a smplfed verson of ther proof, showng only that the cross entropy objectve functon s a member of the class of reasonable error measures. An mportant concept of the proof s prototypes. The nput vector space s parttoned nto regons o p, where P(c o o p ) s essentally constant. The regons are called prototypes. Ths modelng of the probablty dstrbutons s consstent wth the lmted resoluton n the modelng of probablty densty functons due to fnte amount of tranng data. To make the dea of prototypes more concrete, we note that the prototype concept s smlar to that of vector quantzaton, often used n speech technology applcatons. In ths analogy, prototypes correspond to entres n a quantzaton code-book. If we consder one partcular class c and let N be the total number of samples n the tranng data, we can wrte the contrbuton of the error from output unt as: E 1 = N N t = 1 e, t where e,t s the contrbuton from unt and sample t. Now, let us sum over prototypes nstead of samples; let P be the number of prototypes, N p the number of samples from prototype number p, and n p the number of samples from prototype number p. belongng to class. Recall that the target τ,rwh s 1 for the n p samples belongng to class, and 0 for the remanng N p - n p samples. Insertng nto (7), we get the followng expresson for the contrbuton of the class s unt to the error: E P N p n p N p n p = log a p + log 1 p= 1 N N p N p ( a p ) where a p = a(o p ) s the output actvty of the class s unt n the RHW doman of prototype p. The asymptotc behavor as N s N P { ( 1 ) ( 1 )} ( ) ( ) ( ) lm E = P o P c o loga + P c o log a p p p p p p= 1 where we have replaced the expressons for relatve frequency n (21) wth probabltes (law of large numbers). A necessary and suffcent condton for local optmzaton s that the gradent of E wth respect to a p s zero for all prototypes. For class we get: (20) (21) (22) 14

15 P c a p = 0 a ( ο p ) 1 P( c ο p ) p 1 a p = 0 a = P c ( ο ) p p Thus, the output actvtes of the network asymptotcally approxmates the a posteror probabltes. The networks of our smulatons are typcally large wth hgh functonal capacty and the tranng database s farly large, so properly traned networks should be able to estmate p(c o) wth good accuracy. However, gven the complexty of the tranng algorthm wth several dfferent approxmatons and ad hoc parameters, an emprcal study s called for. In secton 5 such an experment s presented and t s shown that the output actvtes do ndeed approxmate p(c o) closely. 2.7 The softmax output actvaton functon The nterpretaton of the output actvtes as the a posteror phoneme observaton probabltes establshed n the prevous secton, leads to some concern about the approprateness of the smple actvaton functon of the output unts. If the output unts are the class probabltes and the classes cover the entre observaton space, then ther sum must clearly be exactly one. However, the smple local actvaton functon used does not enforce ths constrant, ndcatng that there s some redundancy n the model. Ths constrant can be enforced by normalzng the actvtes, yeldng a more complex non-local actvaton functon for the output values. A theoretcally appealng normalzaton s the so called softmax actvaton functon (Brdle 1989; Robnson, 1994; Bourlard and Morgan, 1993), a = N e j= 1 net e net j where ndex j runs over the output unts of all classes. Clearly the softmax functon ensures that the output actvtes sum to one, but ths s accomplshed at the cost of some ncrease n complexty. In our framework, the softmax actvaton functon s mplemented usng exponental unts, multplcaton unts and an nverter unt as llustrated n Fgure 3. The redundancy n the model can now be elmnated by removng all connectons to one of the output unts. It s easy to show that ths yelds an equvalent actvaton functon to (24), by showng that addng a constant bas to all net has no effect: net + bas bas net e e e e N = N = N net j + bas bas net j e e e e j= 1 j= 1 j= 1 net net j (23) (24) (25) Because of ts theoretcal appeal, the softmax functon s used n the smulatons, but t should be noted that the performance ncreases only margnally. Networks traned wth softmax tend to converge to globally more optmal local mnma (lower E), probably because low probabltes are modeled better, but we have seen no mprovement n phoneme recognton experments and only a small decrease n word error-rate. 15

16 e 1/x Group of unts, e.g., a layer Multplcaton unt Exponental unt Inverter unt Output unts wth softmax actvaton 1/x e e e e e The hdden unts are connected to these unts Fgure 3. Sub-network mplementng the softmax actvaton functon. The unts n the top layer computes the actvtes of (24). Ths sub-network s regarded as one sngle layer to other layers of the ANN. Connectons from hdden unts to ths output layer are connected to the lower layer only. The nternal weghts of the sub-network have the constant weght 1.0 and are not changed durng tranng. 3 Phoneme probablty estmaton In ths secton we show how the theory and algorthms of secton 2 are appled to the partcular task of estmatng the phoneme probabltes gven acoustc nput. 3.1 Input feature representaton The raw speech waveform s not well suted for nput to an ANN classfer. Therefore, t s common practce to transform the nput speech sgnal to the short-tme frequency doman before feedng t to the phonetc classfer. The transform used n our experments computes a standard varaton of the Mel cepstrum coeffcents. Readers famlar wth the HTK toolkt (Young et al., 1995) wll notce that the features used here are almost dentcal to those of ts feature extracton tool. The procedure s dscussed more elaborately n (Ström, 1996). Here we gve only an outlne of the procedure. The speech sgnal s dvded nto short overlappng frames as shown n Fgure 4. The frame-rate s 100 frames per second. The DC offset n each frame s removed and preemphass s appled to the sgnal. The sgnal of each frame s then Hammng wndowed and padded wth zero-valued samples to the nearest power of two samples, and the FFT transform s appled to get the magntude spectrum. In Fgure 4, the dfferent wndows are shown together wth an example of a speech sgnal. The magntude spectrum of the FFT s a frequency-doman representaton of the speech n the frame as desred, but t s stll not well suted as nput to the ANN. The hgh frequency resoluton of the spectrum gves nput vectors of hgh dmenson, whch requres unnecessarly many weghts to estmate n the ANN. Ths n turn leads to weak estmates and poor performance. For ths reason, the spectrum of each frame s mapped to a more compact representaton by a compressng transform. Ths s done n two steps: frst a flter-bank s appled to the FFT spectrum and then the cosne transform s appled to the vector of flterbank outputs. The flter-bank conssts of a number of overlappng trangular, equdstantly 16

17 spaced flters (see Fgure 4) on the perceptually motvated Mel frequency scale (Schroeder, Atal and Hall, 1979). We use 24 Mel spaced flters coverng the frequency range Hz. The cosne transform s appled to the flter-bank vector, yeldng the cepstrum coeffcents. The frst twelve cepstrum coeffcents are used, and together wth the logarthm of the energy of the frame they consttute the feature representaton that s fed to the ANN classfer as the nput observaton. Hammng wndow, 25 ms weght 1 frame step, 10 ms frames Herz 8000 Hz sample ponts FFT wndow, 512 sample ponts zero paddng ms frame number Fgure 4. Left: Sgnal processng tme constants and a sample speech sgnal. The three dfferent tme axes are 1) sample ponts, the sample frequency s 16k Hz, 2) ms and 3) frames. The features of a frame s computed from the sgnal convoluted wth a Hammng wndow of 25 ms gvng an effectve wndow of about 10 ms. Top rght: Mel scaled flter-bank wth trangular flters. Bottom rght: Illustraton of the resultng tme and frequency resoluton n a spectrogram-lke plot. The uttered (Swedsh) word s Waxholm (YDNVK2OP). 3.2 Delta coeffcents It s common practce to nclude the frst and second tme-dervatves (delta coeffcents) of the cepstrum coeffcents n the nput vector. In the case of the standard HMM-model, ths manly serves the purpose of ntroducng some dynamc nformaton to the classfer. For dynamc ANNs, the reason for ncludng delta coeffcents s less clear. In fact, a TDNN wth a tme-delay wndow of fve frames can learn to extract delta coeffcents from the nput vector durng tranng as they are smply lnear combnatons of the nput. One could therefore argue that, f delta coeffcents are productve for the classfcaton performance, they wll evolve n the network durng tranng, and there s therefore no need for explctly supplyng them. However, ths s true only n the deal stuaton that there s an unlmted amount of tranng data and that the optmzaton algorthm s perfect,.e., fnds the global optmum regardless of the ntal values of the parameters. If we assume that the delta parameters are good features for representng dynamc nformaton, they have two advantages over the brute-force method of wdenng the tmedelay wndow to take dynamc features nto account. Frst, the dervatves are a more compact representaton of the dynamcs than a wndow of nput frames, leadng to fewer parameters to estmate and therefore more robust estmates. Second, the delta coeffcents can be seen as a partcularly good ntalzaton of connecton weghts, leadng to a more well behaved 17

18 optmzaton. The nterpretaton of delta coeffcents as a choce of ntal weghts becomes more clear when we descrbe how delta and delta-delta coeffcents are mplemented n the networks. Because the delta coeffcents are lnear combnatons of the exstng nput actvtes, they can be modeled by lnear unts n the network. Thus, the delta coeffcents are lnear unts wth four n-flowng connectons wth weghts clamped to the values n the followng formula ( ) 1 d = c + c c c t t+ 2 t+ 1 t 1 t 2 where d t s the actvty of the delta coeffcent unt and c t-x are the actvtes of the orgnal nput unt delayed x frames. Equaton (26) can be derved from lnear regresson and s equvalent to the default delta coeffcents of the HTK toolkt (Young et al., 1995). Second order tme dervatves, so called delta-delta parameters, are computed n the same manner by applyng (26) agan to the delta coeffcents. Fgure 5 llustrates the network mplementaton of the delta and delta-delta unts. -2z -2-2z -2 -z -1 -z -1 dd d c z +1 2z +2 z +1 2z +2 dd dd dd dd dd d d d d d c c c c c (26) tme Fgure 5. Implementaton of delta and delta-delta coeffcents n a network. The unt marked c s a cepstrum coeffcent nput unt. The unts marked d and dd are the correspondng delta and delta-delta unts. The weghts of the lnear unts mplementng d and dd are clamped accordng to (26). 3.3 Input normalzaton In secton 2.5 we saw that a well-chosen weght ntalzaton scheme s helpful for fndng good local mnma. For the same reason t s also desrable that the actvtes of the nput unts are of smlar magntude as the other unts of the network. Ths s acheved by applyng a lnear normalzaton to the nput values. The coeffcents of the lnear transform can be determned n dfferent ways, but we have chosen to base the normalzaton on the mean and standard devaton of the nput values n the tranng database. After collectng second order statstcs, t s easy to lnearly transform each nput to a varable wth zero mean and a controlled varance. In the smulatons, we enforced a standard devaton of 1.0 for all nput unts. The normalzaton was appled to the nput unts usng a specal normalzaton step that s performed only for nput/output unts. The delta and delta-delta unts that are not nput unts to the network (but play a smlar role) are also normalzed by addng a connecton from the basunt for the constant offset, and scalng the other connectons approprately. The connecton weghts of connectons flowng nto the normalzed delta and delta-delta unts are then clamped and are not altered n the back-propagaton tranng. After ths normalzaton, the actvtes of the nput unts and the delta and delta-delta unts all have mean zero and standard devaton one. 18

19 3.4 Network topology The topology of the network,.e., the number of hdden unts and the manner n whch they are connected, determnes the functonal capacty of the classfer. It has been proved (e.g., Hornk, Stnchcombe and Whte, 1989) that ANNs wth one layer of hdden unts can approxmate wth arbtrary precson any smooth functon on a compact doman. However, ths s a theoretcal result that requres that the functon s completely known and that the number of hdden unts s unbounded. Nether of these condtons can be fully satsfed n our problem of phoneme probablty estmaton, but t stll gves some gudance. Although t s possble that a network topology wth more than one hdden layer could utlze the connecton parameters n a more effcent manner to estmate the probablty functons, we have lmted the experments to networks wth one hdden layer and vared only the number of hdden unts. The man motvaton for ths decson was to reduce the number of confguratons to evaluate n the computatonally rather costly computer smulatons. The nput unts, the delta and the delta-delta unts are connected to the hdden unts wth a skewed tme-delay wndow wth dynamc connectons rangng from fve frames look-ahead to one frame delay. The motvaton for the skewed wndow s that the hdden unts addtonally have recurrent connectons between each other, wth tme-delay rangng from one to three frames. The recurrent connectons provde the hdden unts wth addtonal nformaton about the state of the network at past frames, and therefore the delayed sde of the tme-delay wndow can be smaller than the look-ahead sde. Fnally, the output unts are connected to the hdden unts wth a symmetrc tme-delay wndow from plus one to mnus one frame. The network topology s llustrated n Fgure 6. The total number of connectons n the network, determnng the computatonal effort needed for computer smulatons, can now be computed. The number of connectons mplementng the delta and delta-delta parameters s small (104) compared to the connectons to and from the hdden layer. There are 13 nput unts and therefore 3 13=39 unts n the lowest layer (see Fgure 6). Further, let there be N unts n the hdden layer and 61 unts n the output layer (the number of phonemes n the TIMIT database). Ths yelds N 7 + N N 3 + N 61 3 = 3N N (27) connectons n total. As an example, the moderate number of 300 hdden unts gves 406,904 connectons a respectable number that mples a substantal computatonal effort for computer smulatons. Ths problem s addressed n secton 4 where we show how t s possble to reduce the number of connectons wthout decreasng the number of hdden unts. 19

20 phoneme probablty output unts hdden unts connectons from all hdden unts to all output unts wth a tme delay wndow from +1 to -1 tme frames group of unts, e.g., a layer tanhyp unt lnear unt connectons from all nput, delta and delta-delta unts, to all hdden unts, wth all look-aheads from +5 tme frames to delays of -1 frames recurrent connectons between all hdden unts wth tme-delays one, two and three delta-delta unts connectons mplementng the delta operaton connectons mplementng the delta operaton delta unts cepstrum nput unts Fgure 6. The network topology of the ANNs. The parallelograms ndcate layers, and arrows between layers represent sets of connectons from the unts n one layer to unts n another. Because of the dynamc nature of the networks, there are typcally several connectons wth dfferent tme-delay or look-ahead between two connected unts. The layer of output unts s sometmes replaced by a group mplementng the softmax actvaton functon (see secton 2.7). 3.5 Dynamc decodng The ANNs descrbed so far classfy 10 ms frames of audo-nput nto one of the phoneme classes. In the dynamc decodng step of the ASR, ths frame-based output from the network s used to fnd the (n some sense) optmal sequence of phonemes or words. The dynamc decodng of the system used n ths study s descrbed n more detal n (Ström, 1996). To summarze ths study, the well-known hybrd HMM/ANN paradgm (Bourlard and Wellekens, 1990) s adopted, where the output actvtes are nterpreted as the a posteror phoneme probabltes, p(c o) (see secton 2.6). The observaton probabltes, p(o c ), are derved from the a posteror phoneme probabltes usng Bayes s rule. In the case of tanhyp output unts, t s necessary to normalze to the RHW range (see secton 2.1, Equaton (3). Thus, the observaton probabltes can be wrtten: 20

21 ( ) p o c ( ) p o c ( ) p c o ac + 1 p( o) = p( o) p c 2 p c ( ) ( ) p c o = p( o) a p c ( ) c p( o) ( ) p c ( ) for tanhyp output unts for the softmax actvaton functon where p(c ) are the a pror class frequences that are estmated off-lne from the tranng data and a c s the actvaton of the output unt for phoneme (wth range [-1, 1]). The uncondtoned observaton probablty, p(o), s constant for all classes and s therefore dropped n the computatons. The Markov model for a phoneme s shown n Fgure 7. The observaton probablty s constant for all transtons of the phoneme model, and the transton probabltes are maxmum lkelhood (ML) estmates from the duratons of the tranng database. In addton to the coarse duraton model mposed by the transton probabltes, a phoneme-dependent mnmum duraton constrant s used. It s mplemented by addng extra nodes to the HMM and puttng a self-loop on the last node only, as shown n Fgure 9. The mnmum duraton, m frames, for each phoneme s selected such that about 5% of the phones n the tranng data are shorter than m frames. The fracton 5% was chosen, after some expermentng, to optmze the recognton performance. However, the mprovement s very small for phoneme recognton. A probablstc word-class b-gram grammar s used n the word-level evaluaton and a phoneme b-gram grammar s used n the phoneme recognton evaluatons. 1 1 d d = mean phoneme duraton measured n 10 ms frames 1 1 d d d d d (28) 1 d mnmum duraton 4 frames Fgure 7. Phoneme HMM. Left: One-state HMM wth transton probabltes expressed n mean duraton. It s easy to show that the ndcated probabltes are the ML estmates of the model parameters. The mean duraton s computed from the phones of the tranng data. Rght: Phoneme HMM wth mnmum duraton constrant. See the man text for detals. 4 Prunng and sparse connecton The number of hdden unts determnes to a large extent a network s ablty to estmate the a posteror probabltes accurately. It s therefore desrable to experment wth networks wth a large hdden layer. Unfortunately, as was llustrated by (27), the number of connectons grows rapdly wth the number of hdden unts, and that number s practcally bounded by the avalable computatonal resources. One could argue that t s the number of free tranable parameters n the network that s the mportant factor. In ths vew, t s not the number of unts, but the number of connectons that s mportant. However, from experence we know that not only the number of parameters s mportant, but also how they are put to use. 21