Monaural Speech Separation

Transcription

1 In: Proceedings of Neural Informaion Processing Sysems (NIPS 0), 00 Monaural Speech Separaion Guoning Hu DeLiang Wang Biophysics Program Deparmen of Compuer and Informaion The Ohio Sae Universiy Science & Cener of Cogniive Science Columbus, OH 4310 The Ohio Sae Universiy, Columbus, OH Absrac Monaural speech separaion has been sudied in previous sysems ha incorporae audiory scene analysis principles. A major problem for hese sysems is heir inabiliy o deal wih speech in he highfrequency range. Psychoacousic evidence suggess ha differen percepual mechanisms are involved in handling resolved and unresolved harmonics. Moivaed by his, we propose a model for monaural separaion ha deals wih low-frequency and highfrequency signals differenly. For resolved harmonics, our model generaes segmens based on emporal coninuiy and cross-channel correlaion, and groups hem according o periodiciy. For unresolved harmonics, he model generaes segmens based on ampliude modulaion (AM) in addiion o emporal coninuiy and groups hem according o AM repeiion raes derived from sinusoidal modeling. Underlying he separaion process is a pich conour obained according o psychoacousic consrains. Our model is sysemaically evaluaed, and i yields subsanially beer performance han previous sysems, especially in he high-frequency range. 1 Inroducion In a naural environmen, speech usually occurs simulaneously wih acousic inerference. An effecive sysem for aenuaing acousic inerference would grealy faciliae many applicaions, including auomaic speech recogniion (ASR) and speaker idenificaion. Blind source separaion using independen componen analysis [10] or sensor arrays for spaial filering require muliple sensors. In many siuaions, such as elecommunicaion and audio rerieval, a monaural (one microphone) soluion is required, in which inrinsic properies of speech or inerference mus be considered. Various algorihms have been proposed for monaural speech enhancemen [14]. These mehods assume cerain properies of inerference and have difficuly in dealing wih general acousic inerference. Monaural separaion has also been sudied using phasebased decomposiion [3] and saisical learning [17], bu wih only limied evaluaion. While speech enhancemen remains a challenge, he audiory sysem shows a remarkable capaciy for monaural speech separaion. According o Bregman [1], he audiory sysem separaes he acousic signal ino sreams, corresponding o differen sources, based on audiory scene analysis (ASA) principles. Research in ASA has inspired considerable work o build compuaional audiory scene analysis (CASA) 1

2 sysems for sound separaion [19] [4] [7] [18]. Such sysems generally approach speech separaion in wo main sages: segmenaion (analysis) and grouping (synhesis). In segmenaion, he acousic inpu is decomposed ino sensory segmens, each of which is likely o originae from a single source. In grouping, hose segmens ha likely come from he same source are grouped ogeher, based mosly on periodiciy. In a recen CASA model by Wang and Brown [18], segmens are formed on he basis of similariy beween adjacen filer responses (cross-channel correlaion) and emporal coninuiy, while grouping among segmens is performed according o he global pich exraced wihin each ime frame. In mos siuaions, he model is able o remove inrusions and recover low-frequency (below 1 khz) energy of arge speech. However, his model canno handle high-frequency (above 1 khz) signals well, and i loses much of arge speech in he high-frequency range. In fac, he inabiliy o deal wih speech in he high-frequency range is a common problem for CASA sysems. We sudy monaural speech separaion wih paricular emphasis on he high-frequency problem in CASA. For voiced speech, we noe ha he audiory sysem can resolve he firs few harmonics in he low-frequency range [16]. I has been suggesed ha differen percepual mechanisms are used o handle resolved and unresolved harmonics []. Consequenly, our model employs differen mehods o segregae resolved and unresolved harmonics of arge speech. More specifically, our model generaes segmens for resolved harmonics based on emporal coninuiy and cross-channel correlaion, and hese segmens are grouped according o common periodiciy. For unresolved harmonics, i is well known ha he corresponding filer responses are srongly ampliude-modulaed and he response envelopes flucuae a he fundamenal frequency (F0) of arge speech [8]. Therefore, our model generaes segmens for unresolved harmonics based on common AM in addiion o emporal coninuiy. The segmens are grouped according o AM repeiion raes. We calculae AM repeiion raes via sinusoidal modeling, which is guided by arge pich esimaed according o characerisics of naural speech. Secion describes he overall sysem. In secion 3, sysemaic resuls and a comparison wih he Wang-Brown sysem are given. Secion 4 concludes he paper. Model descripion Our model is a mulisage sysem, as shown in Fig. 1. Descripion for each sage is given below..1 Iniial processing Firs, an acousic inpu is analyzed by a sandard cochlear filering model wih a bank of 18 gammaone filers [15] and subsequen hair cell ransducion [1]. This peripheral processing is done in ime frames of 0 ms long wih 10 ms overlap beween consecuive frames. As a resul, he inpu signal is decomposed ino a group of imefrequency (T-F) unis. Each T-F uni conains he response from a cerain channel a a cerain frame. The envelope of he response is obained by a lowpass filer wih Mixure Segregaed Speech Peripheral and mid-level processing Iniial segregaion Pich racking Uni labeling Final segregaion Resynhesis Figure 1. Schemaic diagram of he proposed mulisage sysem.

3 passband [0, 1 khz] and a Kaiser window of 18.5 ms. Mid-level processing is performed by compuing a correlogram (auocorrelaion funcion) of he individual responses and heir envelopes. These auocorrelaion funcions reveal response periodiciies as well as AM repeiion raes. The global pich is obained from he summary correlogram. For clean speech, he auocorrelaions generally have peaks consisen wih he pich and heir summaion shows a dominan peak corresponding o he pich period. Wih acousic inerference, a global pich may no be an accurae descripion of he arge pich, bu i is reasonably close. Because a harmonic exends for a period of ime and is frequency changes smoohly, arge speech likely acivaes coniguous T-F unis. This is an insance of he emporal coninuiy principle. In addiion, since he passbands of adjacen channels overlap, a resolved harmonic usually acivaes adjacen channels, which leads o high crosschannel correlaions. Hence, in iniial segregaion, he model firs forms segmens by merging T-F unis based on emporal coninuiy and cross-channel correlaion. Then he segmens are grouped ino a foreground sream and a background sream by comparing he periodiciies of uni responses wih global pich. A similar process is described in [18]. Fig. (a) and Fig. (b) illusrae he segmens and he foreground sream. The inpu is a mixure of a voiced uerance and a cockail pary noise (see Sec. 3). Since he inrusion is no srongly srucured, mos segmens correspond o arge speech. In addiion, mos segmens are in he low-frequency range. The iniial foreground sream successfully groups mos of he major segmens.. Pich racking In he presence of acousic inerference, he global pich esimaed in mid-level processing is generally no an accurae descripion of arge pich. To obain accurae pich informaion, arge pich is firs esimaed from he foreground sream. A each frame, he auocorrelaion funcions of T-F unis in he foreground sream are summaed. The pich period is he lag corresponding o he maximum of he summaion in he plausible pich range: [ ms, 1.5 ms]. Then we employ he following wo consrains o check is reliabiliy. Firs, an accurae pich period a a frame should be consisen wih he periodiciy of he T-F unis a his frame in he foreground sream. A frame j, le τ( j) represen he esimaed pich period, and A(i,j,τ) he auocorrelaion funcion of u ij, he uni in channel i. u ij agrees wih τ( j) if A( i, j, τ ( j)) / A( i, j, τ ) > θ (1) m d 5000 (a) 5000 (b) Frequency (Hz) Figure. Resuls of iniial segregaion for a speech and cockail-pary mixure. (a) Segmens formed. Each segmen corresponds o a coniguous black region. (b) Foreground sream. 3

4 Here, θ d =0.95, he same hreshold used in [18], and τ m is he lag corresponding o he maximum of A(i,j,τ) wihin [ ms, 1.5 ms]. τ( j) is considered reliable if more han half of he unis in he foreground sream a frame j agree wih i. Second, pich periods in naural speech vary smoohly in ime [11]. We sipulae he difference beween reliable pich periods a consecuive frames be smaller han 0% of he pich period, jusified from pich saisics. Unreliable pich periods are replaced by new values exrapolaed from reliable pich poins using emporal coninuiy. As an example, suppose a wo consecuive frames j and j+1 ha τ( j) is reliable while τ( j+1) is no. All he channels corresponding o he T-F unis agreeing wih τ( j) are seleced. τ( j+1) is hen obained from he summaion of he auocorrelaions for he unis a frame j+1 in hose seleced channels. Then he re-esimaed pich is furher verified wih he second consrain. For more deails, see [9]. Fig. 3 illusraes he esimaed pich periods from he speech and cockail-pary mixure, which mach he pich periods obained from clean speech very well..3 Uni labeling Wih esimaed pich periods, (1) provides a crierion o label T-F unis according o wheher arge speech dominaes he uni responses or no. This crierion compares an esimaed pich period wih he periodiciy of he uni response. I is referred as he periodiciy crierion. I works well for resolved harmonics, and is used o label he unis of he segmens generaed in iniial segregaion. However, he periodiciy crierion is no suiable for unis responding o muliple harmonics because uni responses are ampliude-modulaed. As shown in Fig. 4, for a filer response ha is srongly ampliude-modulaed (Fig. 4(a)), he arge pich corresponds o a local maximum, indicaed by he verical line, in he auocorrelaion insead of he global maximum (Fig. 4(b)). Observe ha for a filer responding o muliple harmonics of a harmonic source, he response envelope flucuaes a he rae of F0 [8]. Hence, we propose a new crierion for labeling he T-F unis corresponding o unresolved harmonics by comparing AM repeiion raes wih esimaed pich. This crierion is referred as he AM crierion. To obain an AM repeiion rae, he enire response of a gammaone filer is half-wave recified and hen band-pass filered o remove he DC componen and oher possible 14 Pich Period (ms) (a) (b) Time (ms) Figure 3. Esimaed arge pich for he speech and cockail-pary mixure, marked by x. The solid line indicaes he pich conour obained from clean speech Lag (ms) Figure 4. AM effecs. (a) Response of a filer wih cener frequency.6 khz. (b) Corresponding auocorrelaion. The verical line marks he posiion corresponding o he pich period of arge speech. 4

5 harmonics excep for he F0 componen. The recified and filered signal is hen normalized by is envelope o remove he inensiy flucuaions of he original signal, where he envelope is obained via he Hilber Transform. Because he pich of naural speech does no change noiceably wihin a single frame, we model he corresponding normalized signal wihin a T-F uni by a single sinusoid o obain he AM repeiion rae. Specifically, M f ij, φij argmin [ˆ( r i, jt k) sin(π k f / f S + φ)], for f [80 Hz, 500 Hz], () f, φ k = 1 = where a square error measure is used. r ˆ( i, ) is he normalized filer response, f S is he sampling frequency, M spans a frame, and T=10 ms is he progressing period from one frame o he nex. In he above equaion, f ij gives he AM repeiion rae for uni u ij. Noe ha in he discree case, a single sinusoid wih a sufficienly high frequency can always mach hese samples perfecly. However, we are ineresed in finding a frequency wihin he plausible pich range. Hence, he soluion does no reduce o a degenerae case. Wih appropriaely chosen iniial values, his opimizaion problem can be solved effecively using ieraive gradien descen (see [9]). The AM crierion is used o label T-F unis ha do no belong o any segmens generaed in iniial segregaion; such segmens, as discussed earlier, end o miss unresolved harmonics. Specifically, uni u ij is labeled as arge speech if he final square error is less han half of he oal energy of he corresponding signal and he AM repeiion rae is close o he esimaed arge pich: f τ ( j) 1 < θ. (3) ij f Psychoacousic evidence suggess ha o separae sounds wih overlapping specra requires 6-1% difference in F0 [6]. Accordingly, we choose θ f o be Final segregaion and resynhesis For adjacen channels responding o unresolved harmonics, alhough heir responses may be quie differen, hey exhibi similar AM paerns and heir response envelopes are highly correlaed. Therefore, for T-F unis labeled as arge speech, segmens are generaed based on cross-channel envelope correlaion in addiion o emporal coninuiy. The specra of arge speech and inrusion ofen overlap and, as a resul, some segmens generaed in iniial segregaion conain boh unis where arge speech dominaes and hose where inrusion dominaes. Given uni labels generaed in he las sage, we furher divide he segmens in he foreground sream, S F, so ha all he unis in a segmen have he same label. Then he sreams are adjused as follows. Firs, since segmens for speech usually are a leas 50 ms long, segmens wih he arge label are reained in S F only if hey are no shorer han 50 ms. Second, segmens wih he inrusion label are added o he background sream, S B, if hey are no shorer han 50 ms. The remaining segmens are removed from S F, becoming undecided. Finally, oher unis are grouped ino he wo sreams by emporal and specral coninuiy. Firs, S B expands ieraively o include undecided segmens in is neighborhood. Then, all he remaining undecided segmens are added back o S F. For individual unis ha do no belong o eiher sream, hey are grouped ino S F ieraively if he unis are labeled as arge speech as well as in he neighborhood of S F. The resuling S F is he final segregaed sream of arge speech. Fig. 5(a) shows he new segmens generaed in his process for he speech and cockailpary mixure. Fig. 5(b) illusraes he segregaed sream from he same mixure. Fig. 5(c) shows all he unis where arge speech is sronger han inrusion. The foreground 5

6 sream generaed by our algorihm conains mos of he unis where arge speech is sronger. In addiion, only a small number of unis where inrusion is sronger are incorrecly grouped ino i. A speech waveform is resynhesized from he final foreground sream. Here, he foreground sream works as a binary mask. I is used o reain he acousic energy from he mixure ha corresponds o 1 s and rejec he mixure energy corresponding o 0 s. For more deails, see [19]. 3 Evaluaion and comparison Our model is evaluaed wih a corpus of 100 mixures composed of 10 voiced uerances mixed wih 10 inrusions colleced by Cooke [4]. The inrusions have a considerable variey. Specifically, hey are: N0-1 khz pure one, N1 - whie noise, N - noise burss, N3 - cockail pary noise, N4 - rock music, N5 - siren, N6 - rill elephone, N7 - female speech, N8 - male speech, and N9 - female speech. Given our decomposiion of an inpu signal ino T-F unis, we sugges he use of an ideal binary mask as he ground ruh for arge speech. The ideal binary mask is consruced as follows: a T-F uni is assigned one if he arge energy in he corresponding uni is greaer han he inrusion energy and zero oherwise. Theoreically speaking, an ideal binary mask gives a performance ceiling for all binary masks. Figure 5(c) illusraes he ideal mask for he speech and cockail-pary mixure. Ideal masks also sui well he siuaions where more han one arge need o be segregaed or he arge changes dynamically. The use of ideal masks is suppored by he audiory masking phenomenon: wihin a criical band, a weaker signal is masked by a sronger one [13]. In addiion, an ideal mask gives excellen resynhesis for a variey of sounds and is similar o a prior mask used in a recen ASR sudy ha yields excellen recogniion performance [5]. The speech waveform resynhesized from he final foreground sream is used for evaluaion, and i is denoed by S(). The speech waveform resynhesized from he ideal binary mask is denoed by I(). Furhermore, le e 1 () denoe he signal presen in I() bu missing from S(), and e () he signal presen in S() bu missing from I(). Then, he relaive energy loss, R EL, and he relaive noise residue, R NR, are calculaed as follows: = R EL e1 ( ) I ( ), (4a) = R NR e ( ) S ( ). (4b) 5000 (a) (b) (c) Frequency (Hz) Figure 5. Resuls of final segregaion for he speech and cockail-pary mixure. (a) New segmens formed in he final segregaion. (b) Final foreground sream. (c) Unis where arge speech is sronger han he inrusion. 6

7 Table 1: R EL and R NR Inrusion Proposed model Wang-Brown model R EL (%) R NR (%) R EL (%) R NR (%) N N N N N N N N N N Average SNR (db) N0 N1 N N3 N4 N5 N6 N7 N8 N9 Inrusion Type Figure 6. SNR resuls for segregaed speech. Whie bars show he resuls from he proposed model, gray bars hose from he Wang-Brown sysem, and black bars hose of he mixures. The resuls from our model are shown in Table 1. Each value represens he average of one inrusion wih 10 voiced uerances. A furher average across all inrusions is also shown in he able. On average, our sysem reains 96.60% of arge speech energy, and he relaive residual noise is kep a 3.3%. As a comparison, Table 1 also shows he resuls from he Wang-Brown model [18], whose performance is represenaive of curren CASA sysems. As shown in he able, our model reduces R EL significanly. In addiion, R EL and R NR are balanced in our sysem. Finally, o compare waveforms direcly we measure a form of signal-o-noise raio (SNR) in decibels using he resynhesized signal from he ideal binary mask as ground ruh: 10 SNR = 10log [ I ( ) ( I( ) S( )) ]. (5) The SNR for each inrusion averaged across 10 arge uerances is shown in Fig. 6, ogeher wih he resuls from he Wang-Brown sysem and he SNR of he original mixures. Our model achieves an average SNR gain of around 1 db and 5 db improvemen over he Wang-Brown model. 4 Discussion The main feaure of our model lies in using differen mechanisms o deal wih resolved and unresolved harmonics. As a resul, our model is able o recover arge speech and reduce noise inerference in he high-frequency range where harmonics of arge speech are unresolved. The proposed sysem considers he pich conour of he arge source only. However, i is possible o rack he pich conour of he inrusion if i has a harmonic srucure. Wih wo pich conours, one could label a T-F uni more accuraely by comparing wheher is periodiciy is more consisen wih one or he oher. Such a mehod is expeced o lead o beer performance for he wo-speaker siuaion, e.g. N7 hrough N9. As indicaed in Fig. 6, he performance gain of our sysem for such inrusions is relaively limied. Our model is limied o separaion of voiced speech. In our view, unvoiced speech poses he bigges challenge for monaural speech separaion. Oher grouping cues, such as onse, offse, and imbre, have been demonsraed o be effecive for human ASA [1], and may play a role in grouping unvoiced speech. In addiion, one should consider he acousic and phoneic characerisics of individual unvoiced consonans. We plan o invesigae hese issues in fuure sudy. 7

8 Acknowledgmens We hank G. J. Brown and M. Wu for helpful commens. Preliminary versions of his work were presened in 001 IEEE WASPAA and 00 IEEE ICASSP. This research was suppored in par by an NSF gran (IIS ) and an AFOSR gran (F ). References [1] A. S. Bregman, Audiory scene analysis, Cambridge MA: MIT Press, [] R. P. Carlyon and T. M. Shackleon, Comparing he fundamenal frequencies of resolved and unresolved harmonics: evidence for wo pich mechanisms? J. Acous. Soc. Am., Vol. 95, pp , [3] G. Cauwenberghs, Monaural separaion of independen acousical componens, In Proc. of IEEE Symp. Circui & Sysems, [4] M. Cooke, Modeling audiory processing and organizaion, Cambridge U.K.: Cambridge Universiy Press, [5] M. Cooke, P. Green, L. Josifovski, and A. Vizinho, Robus auomaic speech recogniion wih missing and unreliable acousic daa, Speech Comm., Vol. 34, pp , 001. [6] C. J. Darwin and R. P. Carlyon, Audiory grouping, in Hearing, B. C. J. Moore, Ed., San Diego CA: Academic Press, [7] D. P. W. Ellis, Predicion-driven compuaional audiory scene analysis, Ph.D. Disseraion, MIT Deparmen of Elecrical Engineering and Compuer Science, [8] H. Helmholz, On he sensaions of one, Braunschweig: Vieweg & Son, (A. J. Ellis, English Trans., Dover, 1954.) [9] G. Hu and D. L. Wang, Monaural speech segregaion based on pich racking and ampliude modulaion, Technical Repor TR6, Ohio Sae Universiy Deparmen of Compuer and Informaion Science, 00. (available a [10] A. Hyvärinen, J. Karhunen, and E. Oja, Independen componen analysis, New York: Wiley, 001. [11] W. J. M. Level, Speaking: From inenion o ariculaion, Cambridge MA: MIT Press, [1] R. Meddis, Simulaion of audiory-neural ransducion: furher sudies, J. Acous. Soc. Am., Vol. 83, pp , [13] B. C. J. Moore, An Inroducion o he psychology of hearing, 4h Ed., San Diego CA: Academic Press, [14] D. O Shaughnessy, Speech communicaions: human and machine, nd Ed., New York: IEEE Press, 000. [15] R. D. Paerson, I. Nimmo-Smih, J. Holdsworh, and P. Rice, An efficien audiory filerbank based on he gammaone funcion, APU Repor 341, MRC, Applied Psychology Uni, Cambridge U.K., [16] R. Plomp and A. M. Mimpen, The ear as a frequency analyzer II, J. Acous. Soc. Am., Vol. 43, pp , [17] S. Roweis, One microphone source separaion, In Advances in Neural Informaion Processing Sysems 13 (NIPS 00), 001. [18] D. L. Wang and G. J. Brown, Separaion of speech from inerfering sounds based on oscillaory correlaion, IEEE Trans. Neural Neworks, Vol. 10, pp , [19] M. Weinraub, A heory and compuaional model of audiory monaural sound separaion, Ph.D. Disseraion, Sanford Universiy Deparmen of Elecrical Engineering,