OVERCOMPLETE BLIND SOURCE SEPARATION BY COMBINING ICA AND BINARY TIME-FREQUENCY MASKING

Transcription

1 OVERCOMPLETE BLIND SOURCE SEPARATION BY COMBINING ICA AND BINARY TIME-FREQUENCY MASKING Michael Syskid Pederse 1,2, DeLiag Wag 3, Ja Larse 1 ad Ulrik Kjems 2 1 Iformatics ad Mathematical Modellig, Techical Uiversity of Demark Richard Peterses Plads, Buildig 321, DK-28 Kgs. Lygby, Demark 2 Otico A/S, Stradveje 58, DK-29 Hellerup, Demark 3 Departmet of Computer Sciece ad Egieerig & Ceter for Cogitive Sciece, The Ohio State Uiversity, Columbus, OH , USA ABSTRACT A limitatio i may source separatio tasks is that the umber of source sigals has to be kow i advace. Further, i order to achieve good performace, the umber of sources caot exceed the umber of sesors. I may real-world applicatios these limitatios are too strict. We propose a ovel method for overcomplete blid source separatio. Two powerful source separatio techiques have bee combied, idepedet compoet aalysis ad biary time-frequecy maskig. Hereby, it is possible to iteratively extract each speech sigal from the mixture. By usig merely two microphoes we ca separate up to six mixed speech sigals uder aechoic coditios. The umber of source sigals is ot assumed to be kow i advace. It is also possible to maitai the extracted sigals as stereo sigals. 1. INTRODUCTION Blid source separatio (BSS) addresses the problem of recoverig N ukow source sigals s() = [s 1(),..., s N()] T from M recorded mixtures x() = [x 1(),..., x M()] T of the source sigals. The term blid refers to that oly the recorded mixtures are kow. A importat applicatio for BSS is separatio of speech sigals. The recorded mixtures are assumed to be liear superpositios of the source sigals, i.e. x() = As() + ν(), (1) where A is a M N mixig matrix ad deotes the discrete time idex. ν() is additioal oise. A method to retrieve the origial sigals up to a arbitrary permutatio ad scalig is idepedet compoet aalysis (ICA) [1]. I ICA, the mai assumptio is that the source sigals are idepedet. By applyig ICA, a estimate y() of the source sigals ca be obtaied by fidig a (pseudo)iverse W of the mixig matrix so that y() = Wx(). (2) May methods require that the umber of source sigals is kow i advace. Aother drawback of most of these methods is that the umber of source sigals is assumed ot to exceed the umber of microphoes, i.e. M N. Eve if the mixig process A is kow, it is ot ivertible, ad i geeral, the idepedet compoets caot be recovered exactly [1]. I the case of more sources tha sesors, the overcomplete/uderdetermied case, successful separatio ofte relies o the assumptio that the source sigals are sparsely distributed - either i the time domai, i the frequecy domai or i the time-frequecy (T-F) domai [2], [3], [4], [5]. If the source sigals do ot overlap i the time-frequecy domai, high-quality recostructio could be obtaied [4]. However, there is overlap betwee the source sigals. I this case, good separatio ca still be obtaied by applyig a biary time-frequecy mask to the mixture [3], [4]. I computatioal auditory scee aalysis, the techique of T-F maskig has bee commoly used for years (see e.g. [6]). Here, source separatio is based o orgaizatioal cues from auditory scee aalysis [7]. More recetly the techique has also become popular i blid source separatio, where separatio is based o o-overlappig sources i the T-F domai [8]. T-F maskig is applicable to source separatio/ segregatio usig oe microphoe [6], [9] or more tha oe microphoe [3], [4]. T-F maskig ca be applied as a biary mask. For a biary mask, each T-F uit is either weighted by oe or by zero. I order to reduce musical oise, more smooth masks may also be applied [1]. A advatage of usig a biary mask is that oly a biary decisio has to be made [11]. Such a decisio ca be based o, e.g., clusterig [3], [4], [8], or directio-of-arrival [12]. ICA has bee used i differet combiatios with the biary mask. I [12], separatio is performed by removig sigals by maskig N M sigals ad afterwards applyig ICA i order to separate the remaiig M sigals. ICA has also bee used the other way aroud. I [13], it has bee applied to separate two sigals by usig two microphoes. Based o the ICA outputs, T- F masks are estimated ad a mask is applied to each of the ICA outputs i order to improve the sigal to oise ratio. I this paper, a ovel method for separatig a arbitrary umber of speech sigals is proposed. Based o the output of a square (2 2) ICA algorithm ad biary T-F masks, this method iteratively segregates sigals from a mixture util a estimate of each sigal is obtaied. 2. GEOMETRICAL INTERPRETATION OF INSTANTANEOUS ICA We assume that there is a ukow umber of acoustical source sigals but oly two microphoes. It is assumed that each source sigal arrives from a certai directio ad o reflectios occur, i.e. a aechoic eviromet. I order to keep the problem simple, the source sigals are mixed by a istataeous mixig matrix as i eq. (1). Due to delays betwee the microphoes, istataeous ICA with a real-valued mixig matrix usually is ot applica-

2 r 1 (θ) r 2 (θ) FRm RUf 9 CNf 6 ITf dB 1dB db 1dB UKm NLm 18 2dB 1dB db 1dB Fig. 1. The two directioal microphoe resposes are show as fuctio of the directio θ. Table 1. The six speech sigals. All speakers use raised voice as if they were speakig i a oisy eviromet. Abbreviatio CNf NLm FRm ITf UKm RUf Descriptio Female speech i Chiese Male speech i Dutch Male speech i Frech Female speech i Italia Male speech i Eglish Female speech i Russia ble to sigals recorded at a array of microphoes, but if the microphoes are placed at exact same locatio ad the microphoes have differet resposes for differet directios, the separatio of delayed sources ca be approximated by the istataeous model [14]. Hereby, a combiatio of microphoe gais correspod to a certai directioal patter. Therefore, two directioal microphoe resposes are used. The two microphoe resposes are chose as fuctios of the directio θ as r 1(θ) = cos(θ) ad r 2(θ) = 1.5 cos(θ), respectively. The two microphoe resposes are show i figure 1. It is possible to make two such directioal patters by addig ad subtractig omidirectioal sigals from two microphoes placed closely together. Hece, the mixig system is give by A(θ) = r1(θ 1) r 1(θ N) r 2(θ 1) r 2(θ N). (3) Differet speech sigals are used as source sigals. The used sigals are sampled with a samplig frequecy of 1 khz ad the duratio of each sigal is 5 s. The speech sigals are show i table More sources tha sesors Now cosider the case where N (M = 2). Whe there are oly two mixed sigals, a stadard ICA algorithm oly has two Fig. 2. The polar plots show the gai for differet directios. ICA is applied with two sesors ad six sources. The two dots at the periphery show the ull directios. The lies poitig out from the origi deote the true directio of the speech sources. The threeletter abbreviatios (see table 1) idetifies the differet speech sigals which have bee used. As it ca be see from the figure, the ICA solutio teds to place the ull towards sources spatially close to each other. Therefore, each of the two outputs is a group of sigals spatially close to each other. output sigals y() = [y 1(), y 2()] T. Sice the umber of separated sigals obtaied by (2) is smaller tha the umber of source sigals, y does ot cotai the separated sigals. Istead y is aother liear superpositio of each of the source sigals, where the weights are give by G = WA istead of just A as i (1). Hereby, G just correspods to aother weightig depedig o θ. These weights make y 1() ad y 2() as idepedet as possible. This is illustrated i figure 2. A implemetatio of the ifomax ICA algorithm [15] has bee used. The BGFS method has bee used for optimizatio [16] 1. The figure shows the two estimated spatial resposes from G(θ) i the overdetermied case. The respose of the m th output is give by w T ma(θ), where w m is the separatio vector from the m th output ad a(θ) is the mixig vector for the arrival directio θ [17]. By varyig θ over all possible directios, directivity patters ca be created as show i figure 2. The estimated ull placemet is illustrated by the two roud dots placed at the periphery of the polar plot. The lies poitig out from the origi illustrate the correct directio of the source sigals. Here, the sources are uiformly distributed i the iterval [ θ 18 ]. As it ca be see, the ulls do ot cacel sigle sources out. Rather, a ull is placed at a directio poitig towards a group of sources which are spatially close to each other. Here, it ca be see that the first output, y 1(), the sigals NLm ad FRm are domiatig ad i the secod output, y 2(), the sigals UKm, ITf ad CNf are domiatig. The sixth sigal, RUf exists i both outputs. This ew weightig of the sigals ca be used to estimate biary masks. 1 Matlab toolbox available from toolbox/ica/

3 3. BLIND SOURCE EXTRACTION WITH ICA AND BINARY MASKING A flowchart for the algorithm is give i figure 3. As described i the previous sectio, a two-iput-two-output ICA algorithm is applied to the iput mixtures, disregardig the umber of source sigals that actually exist i the mixture. The two output sigals are arbitrarily scaled. The scalig is fixed by usig kowledge about the microphoe resposes. Hereby, the two ull directios ca be foud. The two output sigals are scaled such that where oe directioal respose has a ull, the other respose has a uit gai. The two re-scaled output sigals, ŷ 1() ad ŷ 2() are trasformed ito the frequecy domai e.g. by use of the Short-Time Fourier Trasform STFT so that two spectrograms are obtaied: ŷ 1 Y 1(ω,t) (4) ŷ 2 Y 2(ω,t), (5) where ω deotes the frequecy ad t is the time idex. The biary masks are the determied by for each T-F uit comparig the amplitudes of the two spectrograms: BM1(ω, t) = τ Y 1(ω, t) > Y 2(ω,t) (6) BM2(ω, t) = τ Y 2(ω, t) > Y 1(ω,t), (7) where τ is a threshold. Next, each of the the two biary masks is applied to the origial mixtures i the T-F domai, ad by this oliear processig, some of the speech sigals are removed by oe of the masks while other speakers are removed by the other mask. After the masks have bee applied to the sigals, they are recostructed i the time domai by the iverse STFT. If there is oly a sigle sigal left i the masked output, defied by the selectio criteria i sectio 3.1, i.e. all but oe speech sigal have bee masked, this sigal has bee extracted from the mixture ad it is saved. If there are more tha oe sigal left i the masked outputs, ICA is applied to the two masked sigals agai ad a ew set of masks are created based o (6), (7) ad the previous masks. The use of the previous mask esures that T-F uits that have bee removed from the mixture are ot reitroduced by the ext mask. This is doe by a elemet-wise multiplicatio betwee the previous mask ad the ew mask. This iterative procedure is followed util all masked outputs cosist of oly a sigle speech sigal. Notice, the output sigals are maitaied as two sigals. Stereo sigals created with directioal microphoes placed at the same locatio with a agle betwee the directioal patters of 9 (here 18 ) are termed XY-stereo Selectio criterio Further processig o a pair of masked sigals should be avoided i two cases. If all but oe sigal have bee removed or if too much has bee removed so that there is o sigal left after applyig the mask. The decisios are based o the eigevalues of the covariace matrix betwee the masked sesor sigals. The covariace matrix is calculated as R = T, (8) where deotes the expectatio with respect to the whole sigal, ad is the two time domai sigals of which the biary mask has bee applied. If oly cotais oe sigal, the covariace matrix is sigular, ad the smallest eigevalue λ mi is approximately equal to zero [18]. Sice parts of the other sigals may remai after maskig, the smallest eigevalue is equal to the oise variace Cotiue x Iitializatio 1 x2 xˆ 1x ˆy 1 ICA + scalig Estimatio of the two biary masks BM 1 BM 2 Apply to origial microphoe sigals Selectio criteria Fial stereo sigal Iput sigal buffer xˆ 2 x x1 x2 x1 x2 Apply to origial microphoe sigals Stop Stop ˆy 2 Selectio criteria Fial stereo sigal Cotiue Fig. 3. Flowchart showig the mai steps of the proposed algorithm. From the output of the ICA algorithm, biary masks are estimated. The biary masks are applied to the origial sigals which agai are processed through the ICA step. Every time the output from oe of the biary masks is detected as a sigle sigal, the sigal is stored. The iterative procedure stops whe all outputs oly cosist of a sigle sigal. of these remaiig sigals. Therefore, if λ mi is smaller tha a certai oise threshold τ λmi, it is assumed that there is less tha two sigals ad o further processig is ecessary. I order to discrimiate betwee zero or oe sigal, the largest eigevalue λ max is cosidered. If λ max is smaller tha a certai threshold τ λmax, the output is cosidered of such a bad quality that the sigal should be throw away Fidig the remaiig sigals Sice some sigals may have bee removed by both masks, all T-F uits that have ot bee assiged the value 1 are used to create a remaiig mask, ad the procedure is applied to the mixture sigal of which the remaiig mask is applied, to esure that all sigals are estimated. Notice, this step has bee omitted from figure EVALUATION The algorithm described above has bee implemeted ad evaluated with mixtures of the six sigals from table 1. For the STFT,

4 a FFT legth of 248 has bee used. This gives a frequecy resolutio of 125 frequecy uits. A Haig widow with a legth of 512 samples has bee applied to the FFT sigal ad the frame shift is 256 samples. A high frequecy resolutio is foud to be ecessary i order to obtai good performace. The samplig frequecy of the speech sigals is 1 khz. The three thresholds τ, τ λmi ad τ λmax have bee foud from iitial experimets. I the ICA step, the separatio matrix is iitialized by the idetity matrix, i.e. W = I. I order to test robustess, W was also iitialized with a radom matrix with values uiformly distributed over the iterval [,1]. The differet iitializatio did ot affect the result. Whe usig a biary mask, it is ot possible to recostruct the speech sigal as if it was recorded i the absece of the iterferig sigals, because the sigals partly overlap. Therefore, as a computatioal goal for source separatio, the ideal biary mask has bee suggested [11]. The ideal biary mask for a sigal is foud for each T-F uit by comparig the eergy of the desired sigal to the eergy of all the iterferig sigals. Wheever the sigal eergy is highest, the T-F uit is assiged the value 1 ad wheever the iterferig sigals have more eergy, the T-F uit is assiged the value. As i [9], for each of the separated sigals, the percetage of eergy loss P EL ad the percetage of oise residue P NR are calculated: P EL = P NR = e 2 1() I 2 () e 2 2() (9) O 2 (), (1) where O() is the estimated sigal, ad I() is the recorded mixture resythesized after applyig the ideal biary mask. e 1() deotes the sigal preset i I() but abset i O() ad e 2() deotes the sigal preset i O() but abset i I(). Also the sigal to oise ratio (SNR) is foud. Here the SNR is defied usig the resythesized speech from the ideal biary mask as the groud truth I 2 () SNR = 1log 1. (11) (I() O()) 2 The algorithm has bee applied to mixtures cosistig of up to six sigals. I all mixig situatios, the sigals have bee uiformly distributed i the iterval [ θ 18 ]. The separatio results are show i figure 4 ad i table 2. Two ideal biary masks have bee foud oe for each microphoe sigal. I all cases, all the sigals have bee segregated from the mixture. I most cases also the correct umber of sigals is estimated. Oly i the case of three mixtures, oe of the source sigals is estimated twice. The double extractio is caused by the selectio criteria. Based o the chose thresholds, the selectio criteria i some cases allows a sigal to be extracted more tha oce. I the case of the six mixtures from figure 2, the six estimated biary masks are show i figure 5 alog with the estimated ideal biary masks from each of the two microphoe sigals. The iput SNR (SNR i) is show i figure 4 too. The SNR i is the ratio betwee the desired sigal ad the oise i the recorded mixtures. The separatio quality decreases whe the umber of sigals is Average SNR [db] Number of sources SNR i SNR Fig. 4. The sigal to oise ratio as fuctio of the umber of source sigals. The average SNR for the mixtures before separatio (SNR i) is show as well as the average SNR after separatio calculated by eq. (11). I the case of three sigals, the icorrectly estimated sigal is igored (see table 2). icreased. This is expected because whe the umber of mixed sigals is icreased, the mixtures become less sparse. Radom distributios of the source directios as well as more tha six sigals have also bee examied. Here, i geeral, ot all the sources are separated from each other. If the arrival agles betwee sigals are too arrow, these sigals may be detected as a sigle sigal, ad they are ot separated. Listeig tests validate the separatio results. This method differs from previous methods which use a biary mask ad two microphoes [3], [4]. I [3], biaural cues have bee applied for separatio, i.e iteraural time ad itesity differeces. I [4], the separatio is likewise based o amplitude ad time differece of each source. Here separatio is based o clusterig of T-F uits that have similar amplitude ad phase properties. I our approach too, separatio ca oly be achieved if the source sigals have differet spatial positios, but the separatio criterio is based o idepedece betwee the source sigals. 5. CONCLUDING REMARKS A ovel method of blid source separatio of has bee described. Based o sparseess ad idepedece, the method iteratively extracts all the speech sigals without kowig the sigals i advace. A advatage of this method is that stereo sigals are maitaied through the processig. So far, the method has bee applied to successful separatio of up to six speech sigals uder aechoic coditios by use of two microphoes. Future work will iclude separatio of mixtures i reverberat eviromet, a more blid solutio of the scalig problem, ad improved techiques for the stoppig criteria based o detectio of a sigle sigal. Alterative to usig a liear frequecy scale, a frequecy scale that models the auditory system more accurately could be used, because a auditory-based frot-ed is reported to be more robust tha a Fourier-based aalysis i the presece of backgroud iterferece [9]. The use of more tha two sesors could also be ivestigated. By usig more tha two sesors, a better resolutio ca be obtaied

5 (a) (c) (e) Microphoe 1 ideal biary mask for CNf Microphoe 2 ideal biary mask for CNf Estimated biary mask for CNf Microphoe 1 ideal biary mask for FRm Microphoe 2 ideal biary mask for FRm Estimated biary mask for FRm Microphoe 1 ideal biary mask for UKm Microphoe 2 ideal biary mask for UKm Estimated biary mask for UKm (b) Microphoe 1 ideal biary mask for NLm Microphoe 2 ideal biary mask for NLm Estimated biary mask for NLm (d) Microphoe 1 ideal biary mask for ITf (f) Microphoe 2 ideal biary mask for ITf Estimated biary mask for ITf Microphoe 1 ideal biary mask for RUf Microphoe 2 ideal biary mask for RUf Estimated biary mask for RUf Fig. 5. For a mixture of 6 mixed speech sigals, biary masks have bee estimated for each of the 6 speech sigals. The black areas correspod to the mask value 1 ad the white areas correspod to the mask value. The results are show together with the calculated ideal biary masks of each of the two microphoe sigals. The sigals (a) (f) appear i the order which they were extracted from the mixture. The first three sigals (a) (c) were extracted after two iteratios, the ext two sigals (d), (e) were extracted after three iteratios. The last sigal (f) was extracted from the remaiig mask as described i sectio 3.2.

6 Table 2. Separatio results. Mixtures cosistig from two up to six sigals have bee separated from each other successfully. I most cases, the correct umber of sources has bee extracted. Oly i the case of three source sigals, oe of the sigals has bee estimated twice. Here the average performace has bee calculated with( ) ad without the extra sigal. The sigals appear i the order which they were extracted from the mixture. Separated Microphoe 1 Microphoe 2 Sigal P EL(%) P NR(%) P EL(%) P NR(%) UKm FRm Average NLm CNf CNf RUf Average Average CNf RUf FRm UKm Average RUf NLm FRm ITf CNf Average CNf NLm FRm ITf UKm RUf Average ad ambiguous arrival agles may be avoided. Also applicatios for other types of sparse sigals could be examied. 6. ACKNOWLEDGEMENTS The work was performed while M.S.P. was a visitig scholar at The Ohio State Uiversity Departmet of Computer Sciece ad Egieerig. M.S.P was supported by the Otico Foudatio. M.S.P ad J.L are partly also supported by the Europea Commissio through the sixth framework IST Network of Excellece: Patter Aalysis, Statistical Modellig ad Computatioal Learig (PASCAL), cotract o D.L.W was supported i part by a AFOSR grat (FA ) ad a AFRL grat (FA ). 7. REFERENCES [1] A. Hyvärie, J. Karhue, ad E. Oja, Idepedet Compoet Aalysis, Wiley, 21. [2] P. Bofill ad M. Zibulevsky, Blid separatio of more sources tha mixtures usig sparsity of their short-time fourier trasform, i Proc. ICA 2, 2, pp [3] N. Roma, D. L. Wag, ad G. J. Brow, Speech segregatio based o soud localizatio, J. Acoust. Soc. Amer., vol. 114, o. 4, pp , October 23. [4] Ö. Yilmaz ad S. Rickard, Blid separatio of speech mixtures via time-frequecy maskig, IEEE Tras. Sigal Processig, vol. 52, o. 7, pp , July 24. [5] S. Witer, H. Sawada, S. Araki, ad S. Makio, Overcomplete bss for covolutive mixtures based o hierarchical clusterig, i Proc. ICA 24, Graada, Spai, September , pp [6] D. L. Wag ad G. J. Brow, Separatio of speech from iterferig souds based o oscillatory correlatio, IEEE Tras. Neural Networks, vol. 1, o. 3, pp , May [7] A. S. Bregma, Auditory Scee Aalysis, MIT Press, 2 editio, 199. [8] A. Jourjie, S. Richard, ad Ö. Yilmaz, Blid separatio of disjoit orthogoal sigals: Demixig sources from 2 mixtures, i Proc. ICASSP 2, Istabul, Turkey, Jue 2, vol. V, pp [9] G. Hu ad D. L. Wag, Moaural speech segregatio based o pitch trackig ad amplitude modulatio, IEEE Tras. Neural Networks, vol. 15, o. 5, pp , September 24. [1] S. Araki, S. Makio, H. Sawada, ad R. Mukai, Reducig musical oise by a fie-shift overlap-add method applied to source separatio usig a time-frequecy mask, i Proc. ICASSP25, March , vol. III, pp [11] D. L. Wag, O ideal biary mask as the computatioal goal of auditory scee aalysis, i Speech Separatio by Humas ad Machies, Pierre Diveyi, Ed., pp Kluwer, Norwell, MA, 25. [12] S. Araki, S. Makio, H. Sawada, ad R. Mukai, Uderdetermied blid separatio of covolutive mixtures of speech with directivity patter based mask ad ica, i Proc. ICA 24, September , pp [13] D. Kolossa ad R. Orglmeister, Noliear postprocessig for blid speech separatio, i Proc. ICA 24, Graada, Spai, September , pp [14] M. Ito, Y. Takeuchi, T. Matsumoto, H. Kudo, M. Kawamoto, T. Mukai, ad N. Ohishi, Movig-source separatio usig directioal microphoes, i Proc. ISSPIT 22, December 22, pp [15] A. J. Bell ad T. J. Sejowski, A iformatiomaximizatio approach to blid separatio ad blid decovolutio, Neural Computatio, vol. 7, o. 6, pp , [16] H.B. Nielse, Ucmif - a algorithm for ucostraied, oliear optimizatio, Tech. Rep. IMM-TEC-19, IMM, Techical Uiversity of Demark, 21. [17] M. S. Bradstei ad D. B. Ward, Eds., Microphoe Arrays, Digital Sigal Processig. Spriger, 21. [18] M. Wax ad T. Kailath, Detectio of sigals by iformatio theoretic criteria, IEEE Tras. Acous., Speech ad Sigal Processig, vol. ASSP-33, o. 2, pp , April 1985.