DEEP NEURAL NETWORKS FOR COCHANNEL SPEAKER IDENTIFICATION

Transcription

1 DEEP NEURAL NETWORKS FOR COCHANNEL SPEAKER IDENTIFICATION Xioji Zho 1, Yuxun Wng 1 nd DeLing Wng 1,2 1 Dertment of Comuter Science nd Engineering, The Ohio Stte University, Columus, OH, USA 2 Center for Cognitive nd Brin Sciences, The Ohio Stte University, Columus, OH, USA {zhox, wngyuxu, dwng}@cse.ohio-stte.edu ABSTRACT Seker identifiction (SID) in cochnnel seech, where two sekers re tlking simultneously over single recording chnnel, is chllenging rolem. Previous studies ddress this rolem in the nechoic environment under the Gussin mixture model (GMM) frmework. On the other hnd, cochnnel SID in reverernt conditions hs not een ddressed. This er studies cochnnel SID in oth nechoic nd reverernt conditions. We exlore dee neurl networks (DNNs) for cochnnel SID nd roose DNN-sed recognition system. Evlution results demonstrte the roosed DNN-sed system outerforms the two stte-of-the-rt cochnnel SID systems in oth nechoic nd reverernt conditions nd vrious trget-to-interferer rtios. Index Terms Cochnnel seker identifiction, reverertion, dee neurl network, Gussin mixture model, trget-to-interferer rtio 1. INTRODUCTION To serte seech signls from multile tlkers, one cn lce microhones t different loctions nd tke dvntge of the time nd intensity differences of the recordings. The tsk, however, ecomes considerly more chllenging with single microhone. Cochnnel seech is such cse where two sekers re recorded in single communiction chnnel. Unlike converstion, the sekers re not wre of ech other, creting lrge mounts of overling seech. Cochnnel seech sertion is chllenging rolem. Suervised methods [13, 16] usully ssume tht the seker identities re ville in order to utilize the seker models. Other work conducts cochnnel seker identifiction (SID) s front-end for sertion, or jointly with sertion. Comred to cochnnel seech recognition, one dvntge of cochnnel SID is tht it only needs suset of homogenous seech segments to infer seker identities. Such segments re clled usle seech [10]. How to grou usle seech cross time into two strems is deemed s sequentil grouing rolem. Sho nd Wng jointly serch ll the grouing hyothesis nd seker cndidtes to get the otiml one [18, 19]. Mowlee et l. roose to tret cochnnel SID nd sertion s n itertive rocess [11]. Lter they imrove the erformnce y fusing dted GMM nd Kullck-Leiler divergence scores [12]. Hershey et l. get the est seech recognition erformnce thnks in rt to excellent erformnce of cochnnel SID nd sertion [7]. Their SID system first cretes short list of most role seker cndidtes. This reserch ws suorted in rt y n AFOSR grnt (FA ). We would like to thnk the Ohio Suercomuter Center for roviding comuting resources. The to seker is then ired with the rest for execttionmximiztion (EM) sed gin estimtion. The outut is the seker ir whose gin dted model mximizes the likelihood of the test utternce. Their system chieves the verge SID ccurcy of etter thn 98%. Li et l. tke very similr SID roch [9]. It dds few constrints to the genertion of the short list. The to seker model is directly comined with ech of the rest nd the comined models re used for SID directly without the EM ste. The refined system yields n ccurcy greter thn 99%. These two my e regrded s the stte-of-the-rt cochnnel SID methods. Due to the excellent erformnce of dee neurl networks (DNNs) in mny tsks, reserchers egin to study how to incororte DNN in seker recognition [2, 4, 17]. However, DNN hs not een utilized in cochnnel SID to our knowledge. Stte-of-the-rt cochnnel SID erformnce is reorted on the seech sertion chllenge (SSC) corus [7, 9]. This corus [3], however, ws tilored for roust seech recognition rther thn seker recognition. The reltive smll voculry nd common words etween trining nd testing reduce the difficulty of the SID tsk [22]. In this study, we emloy seker recognition evlution (SRE) dtset of the Ntionl Institute of Stndrds nd Technology (NIST). We roose the first DNN-sed cochnnel SID system working in oth nechoic nd reverernt conditions. It trins frme level multi-clss DNN clssifier tht oututs the osterior roility of frme eing dominted y ech seker. Frme level decisions re integrted to mke the finl decision. The rest of the er is orgnized s follows. In Sect. 2, we formulte the cochnnel SID rolem nd descrie the roosed system. Sect. 3 descries the currently dominnt GMM-sed roch. Model trining is discussed in Sect. 4, followed y evlution nd comrison in Sect. 5. We conclude this er in Sect DNN-BASED COCHANNEL SID We formulte cochnnel SID s discrimintive lerning rolem, where we directly lern ming from cochnnel oservtions to the corresonding sek identities. Secificlly, we tret cochnnel SID s multi-clss clssifiction rolem nd emloy DNN s the lerning mchine. To our knowledge, this is the first study of DNNsed cochnnel SID.s Figure 1 shows the schemtic digrm of the roosed DNN-sed system. It trins DNN using frme level fetures. The outut lyer hs the sme numer of nodes s sekers. Only the two nodes corresonding to the underlying sekers hve non-zero trining lels. During testing, the frme level outut is ggregted cross time to generte the finl outut /15/$ IEEE 4824 ICASSP 2015

2 We use frme level log-sectrl fetures s inut. To encode temorl context, we slice window of 11 frmes of fetures to trin the DNN. The trining trget of the DNN is the true seker identities. We use soft trining lels where the two underlying sekers ech hve roility of generting the current frme. The sum of their roilities equls one, wheres the other sekers hve zero roilities. We comre frme level energy of two sekers nd use their rtio for the soft lels. More secificlly, we construct the idel inry msk (IBM) [20], nd frme level energy of ech seker is clculted from the mixture cochlegrm ccording to the IBM. The DNN emloyed in our study is dee multilyer ercetron. The DNN uses three hidden lyers, ech hving 1024 sigmoidl hidden units. The stndrd ckrogtion lgorithm couled with droout regulriztion (droout rte 0.2) is used to trin the network. No unsuervised retrining is used, s we hve sufficient leled dt. We use the dtive grdient descent long with momentum term s the otimiztion technique. A momentum rte of 0.5 is used for the first 5 eochs, fter which the rte increses to 0.9. We use softmx outut lyer nd cross-entroy s the loss function. The trining dt is discussed in Section GMM-BASED COCHANNEL SID In this section, we resent the currently dominnt GMM-sed cochnnel SID frmework. This introduction serves to contrst our DNN-sed roch, nd descrie the lgorithms used for lter comrisons. Given n oservtion O, the gol of cochnnel SID is to get the two underlying sekers ˆ nd ˆ tht generte the oservtion. This cn e formulted s serching for the seker ir with the highest osterior roility. ˆ, ˆ rg mx P, O, rg mx, O, P, Seker IDs Frme Score Aggregtion Seker Lels Hidden Lyer Hidden Lyer Frme Level Fetures Inut Seech Figure 1. Schemtic digrm of the roosed DNN sed cochnnel SID system O rg mx O,. (1), We ly the Byes formul to convert the osterior roility to the likelihood of joint distriution of two sekers, with the ssumtion tht ll seker irs re eqully role. (O) is not deendent on sekers nd cn thus e droed from the clcultion. The question now ecomes how to clculte likelihoods of joint distriution. Sho nd Wng hve introduced vrile g, to (1), to ssign ech seech segment to one of the two seker sources [18, 19]. The derivtion is shown s follows. ˆ, ˆ rg mx O,, rg mx, g rg mx mx O, g,, g O, g, rg mx mx X, X. (2), XS Here X denotes seech segment, S the set of ll segments, nd g n ssignment vector of the sme length s S. Ech element of g is inry lel tht ssigns the corresonding segment to seker. The integrtion over ll ssignments is roximted s mx oertion, ssuming tht the otiml ssignment domintes the summtion. By ssuming tht segments re indeendent, the rolem reduces to finding the est ssignment for ech segment nd the likelihood of the utternce is the multiliction of segment likelihoods. The seker ir with the highest likelihood is the SID outut. The corresonding otiml ssignment lso gives solution to the cochnnel sertion rolem y orgnizing segments into two grous. In other words, this roch jointly erforms cochnnel SID nd sertion, so we nme it joint SID & sertion (JSS). Li et l. hve roosed two stge lgorithm tht roduces stteof-the-rt erformnce in the SSC corus [5, 9]. The first stge rnks sekers ccording to their osterior roilities given the oservtion. The osterior roility of ech seker given X is clculted s follows. X P X X (3) m m P m where m is the seker index. P() nd P( m ) re rior roilities. Assuming tht ll the sekers re eqully role, the riors cn e eliminted. Frme level osterior roilities re ggregted cross time to otin utternce level roilities. Sekers re rnked sed on the ggregted scores. The to ten sekers re ket for the second stge where the to seker is comined with ech of the remining nine. The comosite GMMs re used for stndrd seker recognition to get the est seker ir. We oint out tht the comosition oertes on er frme sis. Li et l. s two stge lgorithm is fine-tuned version of Hershey et l. s SID system [7]. Overll, the two systems yield the est erformnce in the SSC corus with Li et l. s verge erformnce round 1% higher. 4. MODEL TRAINING In this study, we del with oth nechoic nd reverernt test conditions. For the nechoic condition, we use nechoic dt to trin GMMs nd DNNs. However, such models do not generlize well to 4825

3 reverernt conditions. Thus, we directly model sekers in the reverernt environments. The degree of reverertion is tyiclly indicted y reverertion time (T 60 ), the time tken for direct sound to ttenute y 60 db [8]. Reverertion is modeled s convolution etween room imulse resonse (RIR) nd direct sound signl. An RIR chrcterizes secific reverernt environment nd is determined y fctors such s the geometry of the room, nd loctions of sound sources nd receivers. Assuming no knowledge of test reverernt conditions, we simulte N reresenttive reverernt trining conditions covering lusile rnge of T 60. Our revious study hs shown tht this technique hs resonle generliztion [23]. We rere trining dt in ech of the N conditions. GMMs re trined using single seker dt while DNNs re trined with cochnnel dt mixed t different TIRs. Detils re given in the next section. 5. EVALUATION AND COMPARISON 5.1. Exerimentl Setu We rndomly select 100 sekers from the 2008 NIST SRE dtset (short2 rt of the trining set). The telehone converstion excert of ech seker is roughly 5 minutes long. Lrge chunks of silence in the excert re removed. Then we divide the recording into 5 s ieces. Two ieces with the highest energy re used for tests in order to rovide sufficient seech informtion. The rest is used for trining. Overll ech seker hs out 20 trining utternces. More detils of the evlution corus cn e found in [23]. A Mtl imlementtion of the imge method of Allen nd Berkley is used to simulte room reverertion [1, 6]. We focus on the T 60 rnge u to 1s tht covers relistic reverernt conditions [8]. Three rooms re simulted to otin 3 trining T 60 s: 300, 600 nd 900 ms. For ech T 60, we generte 5 RIRs y rndomly ositioning the source nd receiver while keeing their distnce fixed t 2 m. Ech trining utternce is convolved with the 5 RIRs of ech room to crete reverernt trining dt. Seven rooms re simulted to otin 7 test T 60 s from 300 ms to 900 ms with ste size of 100 ms. We rndomly generte 3 irs of RIRs t ech T 60 where ech ir rovides one RIR for the trget nd one for the interferer. In totl there re 21 irs of test RIRs. Note tht the RIRs re different etween trining nd testing even when they re generted with the sme T 60. DNNs re trined using cochnnel trining dt. Insted of one DNN er seker, we trin universl DNN for ll the sekers. We include trining dt from every seker ir for comlete coverge. For nechoic conditions, we crete 10 nechoic cochnnel utternces er seker ir t 3 TIRs (-5, 0 nd 5 db). In totl, there re 4950 seker irs nd cochnnel trining utternces er TIR. For reverernt conditions, we crete 10 reverernt cochnnel utternces t ech of the 3 T 60 s nd 3 TIRs. In totl, there re cochnnel trining utternces er TIR nd er T 60. For JSS, we extrct 22-dimensionl MFCC s seker fetures. Seker models re dted from 1024-comonent universl ckground model (UBM) trined y ooling trining dt from ll the sekers [15]. For Li et l., we extrct 64-dimensionl log-sectrl fetures for GMM trining. Secificlly, 64-chnnel gmmtone filternk is emloyed s the front-end. The filter outut is converted to cochlegrm [21]. We tke the log oertion on the cochlegrm to get the fetures. For nechoic conditions, 256-comonent GMM is trined for ech seker [14]. Another 256-comonent GMM is trined using the reverernt trining dt y convolving the nechoic trining dt with the RIRs t 3 T 60 s. Cochnnel test set covers ll ossile seker irs. For ech ir, 4826 we crete two nechoic utternces nd two reverernt utternces t - 5, 0 nd 5 db TIRs. There re totlly 9900 nechoic test utternces nd 9900 reverernt test utternces er TIR. Ech reverernt cochnnel test utternce is creted using rndomly selected RIR ir from the 21 RIR ir lirry Performnce on the SSC Corus The stte-of-the-rt cochnnel SID systems of Hershey et l. nd Li et l. hve reorted erformnce on the SSC corus. This corus consists of trining utternces from 34 sekers. Ech trining utternce is creted following fixed grmmr: commnd, color, reosition, letter, numer, nd dver. Ech of the six ositions hs smll numer of word choices. The cochnnel test set of the SSC corus comrises six TIRs from -9 db to 6 db. There re 600 test utternces for ech TIR. Every test utternce is mixed from clen test utternces of two sekers. Note tht the clen utternces follow the sme grmmr nd shre the sme voculry s the trining utternces. We evlute our roosed system on this dtset in order to mke direct comrison. Tle I gives the SID results of the roosed system nd cometing systems. As cn e seen, our imlementtion of Li et l. s two stge system chieves the sme verge erformnce s their er. The roosed DNN-sed system yields the est results, lthough the erformnce gin is roly not significnt. As the results re nerly erfect, there is not much room to imrove nd we cn conclude tht the roosed system work comrly well Performnce on NIST SRE Dtset with 50 sekers First we test on suset of 50 sekers with 1225 seker irs, to e roughly comrle with the SSC corus in terms of seker numer. We crete two cochnnel utternces for ech ir t ech of 3 TIRs, 5 db, 0 db nd 5 db. In totl, there re 2450 test trils er TIR. The erformnce is given in Tle II. As shown in the tle, there is sustntil dro of erformnce comred to the SSC corus, confirming tht the SSC corus is rther esy for cochnnel SID evlution. For this dtset, JSS outerforms Li et l. y n verge of 4.3%. We lso evlute the DNN-sed cochnnel SID system, which further outerforms the est cometing system y lrge mrgin (lmost 13%). Next we test in the reverernt conditions, nd the results re shown in Tle III. As cn e seen, the erformnces of ll the methods degrde in the reverernt conditions. JSS dros y out 30%. Li et l. s is slightly more roust, ut still dros y more thn 20%. In ddition, the roosed DNN-sed system continues to erform the est, outerforming JSS y more thn 19% nd Li et l. s system y 14% Performnce on NIST SRE Dtset with 100 sekers The SID tsk ecomes more chllenging s the numer of sekers (clsses) increses. To quntify cochnnel SID deendency on the numer of sekers, we hve erformed cochnnel SID evlution y incresing the numer of sekers from 50 to 100, qudruling the numer of clsses to As in the revious results, the defult DNN configurtion (3 hidden lyers with 1024 nodes ech) outerforms the est cometing system. With the increse of seker size s well s trining dt size, we hve lso exlored few different DNN configurtions. As we increse the numer of units from 1024 to 2048 for ech hidden lyer, the SID erformnce imroves y round 4.5%. There is slight imrovement s we exnd the numer of hidden lyers from 3 to 5 without chnging the hidden lyer size, for either 1024 or 2048 hidden units. Further enlrgement of the DNN size is exected to imrove the erformnce even more, ut t the exense of sustntilly incresed comuttionl comlexity.

4 Tle I: SID ccurcy (%) on SSC corus. Method 9 db 6 db 3 db 0 db 3 db 6 db Avg. Reorted Performnce of Hershey et l Reorted Performnce of Li et l JSS Li et l DNN Tle II: SID ccurcy (%) on nechoic NIST SRE dtset with 50 sekers JSS Li et l DNN Tle III: SID ccurcy (%) on reverernt NIST SRE dtset with 50 sekers JSS Li et l DNN Tle IV: SID ccurcy (%) on reverernt NIST SRE dtset with 100 sekers JSS Li et l RELATION TO PRIOR WORK The work resented here hs focused on the cochnnel SID rolem. Previous studies on this toic focus on GMM-sed roches in the nechoic condition. DNN hs not een studied for this rolem, nd there is no revious work on cochnnel SID in reverernt conditions. Our study ddresses this rolem in oth nechoic nd reverernt conditions y introducing DNN-sed roch. DNN (1024 y 3) DNN (1024 y 5) DNN (2048 y 3) DNN (2048 y 5) CONCLUDING REMARKS This er hs numer of novel contriutions. Our first contriution lies in the introduction of DNN for cochnnel SID. Our roosed DNN system sustntilly outerforms the stte-of-the-rt SID methods, which re GMM-sed. Secondly, we ddress cochnnel SID in reverernt conditions, toic tht hs not een studied efore. Since this is the first study of lying DNN to cochnnel SID, therefore there is likely room for future imrovement. For instnce, trining fetures nd lels cn e systemticlly exmined, nd DNN rchitecture cn e otimized. With the excellent erformnce of cochnnel SID, we elieve tht the use of DNN reresents romising direction to ursue noise roust SID, reverertion roust SID, nd seker verifiction tsks REFERENCES [1] J.B. Allen nd D.A. Berkley, "Imge method for efficiently simulting smll-room coustics," Journl of the Acousticl Society of Americ, vol. 65, , [2] K. Chen nd A. Slmn, Lerning seker-secific chrcteristics with dee neurl rchitecture, IEEE Trnsctions on Neurl Networks, vol. 22, no. 11, , [3] M. Cooke nd T. Lee, Seech sertion nd recognition cometition, 2006 [Online]. Aville: htt:// ge.htm [4] S. Grimell nd H. Hermnsky, Fctor nlysis of utossocitive neurl networks with liction in seker verifiction, IEEE Trnsctions on Neurl Networks nd Lerning Systems, vol. 24, no. 4, , [5] Y. Gun nd W. Liu, A two-stge lgorithm for multiseker identifiction system, in Proc. Interntionl Symosium on Chinese Soken Lnguge Processing, 2008, [6] E.A.P. Hets, Room imulse resonse genertor, 2010 [Online]. Aville: htt://home.tiscli.nl/ehets/rir_generto r.html

5 [7] J. Hershey, S. Rennie, P. Olsen, nd T. Kristjnsson, Suer humn multi-tlker seech recognition: A grhicl model roch, Comuter Seech & Lnguge, vol. 24, , 2010 [8] H. Kuttruff, Room Acoustics. New York, NY: Son, [9] P. Li, Y. Gun, S. Wng, B. Xu nd W. Liu, Monurl seech sertion sed on MAXVQ nd CASA for roust seech recognition, Comuter Seech & Lnguge, vol. 24, , [10] J. M. Lovekin, R. E. Yntorno, K. R. Krishnmchri, D. S. Benincs nd S. J. Wenndt, Develoing usle seech criteri for seker identifiction, in Proc. ICASSP, 2001, [11] P. Mowlee, R. Seidi, Z. Tn, M. Christensen, P. Fränti, nd S. Jensen, Joint single-chnnel seech sertion nd seker identifiction, in Proc. ICASSP, 2010, [12] P. Mowlee, R. Seidi, M. Christensen, Z. Tn, T. Kinnunen, P. Fränti nd S. Jensen, A joint roch for single-chnnel seker identifiction nd seech sertion, IEEE Trnsctions on Audio, Seech nd Lnguge Processing, vol. 20, no. 9, , [13] A. Reddy nd B. Rj, Soft msk methods for single-chnnel seker sertion, IEEE Trnsctions on Audio, Seech nd Lnguge Processing, vol. 15, no. 6, , [14] D.A. Reynolds, "Seker identifiction nd verifiction using Gussin mixture seker models," Seech Communiction, vol. 17, , [15] D.A. Reynolds, T.F. Qutieri, nd nd R.B. Dunn, "Seker verifiction using dted Gussin mixture models," Digitl Signl Processing, vol. 10, , [16] S. T. Roweis, "One microhone source sertion," in Proc. NIPS, 2000, [17] M. Senoussoui, N. Dehk, P. Kenny, R. Dehk nd P. Dumouchel, First ttemt of oltzmnn mchines for seker verifiction, in Proc. Odyssey, The Seker nd Lnguge Recognition Worksho, [18] Y. Sho nd D.L. Wng, Co-chnnel seker identifiction using usle seech extrction sed on multi-itch trcking, in Proc. ICASSP, 2003, [19] Y. Sho nd D.L. Wng, Model-sed sequentil orgniztion in cochnnel seech, IEEE Trnsctions on Audio, Seech nd Lnguge Processing, vol. 14, no. 1, , [20] D.L. Wng, "On idel inry msk s the comuttionl gol of uditory scene nlysis," in Seech sertion y humns nd mchines, P. Divenyi, Ed. Norwell, MA: Kluwer Acdemic, 2005, [21] D.L. Wng nd G.J. Brown, Eds., Comuttionl Auditory Scene Anlysis: Princiles, Algorithms, nd Alictions. Hooken, NJ: Wiley-IEEE, [22] X. Zho, Y. Sho, nd D.L. Wng, CASA-sed roust seker identifiction, IEEE Trnsctions on Audio, Seech nd Lnguge Processing, vol. 20, no. 5, , [23] X. Zho, Y. Wng nd D.L. Wng, Roust seker identifiction in noisy nd reverernt conditions, IEEE/ACM Trnsctions on Audio, Seech nd Lnguge Processing, vol. 22, no. 4, ,