Noise Mask for TDOA Sound Source Localization of Speech on Mobile Robots in Noisy Environments

Transcription

1 Noise Mask for TO Sound Source Locaization of Speech on Mobie Robots in Noisy Environments François Grondin and François Michaud bstract Sound source ocaization is an important chaenge for mobie robots operating in rea ife settings. Sound sources of interest, such as speech, are often corrupted by broadband coherent noise sound source(s) that are nonstationary during transitions between steady-state segments. The interfering noise introduces ocaization ambiguities eading to the ocaization of invaid sound sources. Masks to reduce such interferences perform we under stationary noise, but the performance degrades as ocaization of invaid sound sources generated by noise appear and disappear suddeny during transitions between steady-state. This paper presents a new mask based on speech non-stationarity to discriminate between the time difference of arriva (TO) of speech source and noise transition. Simuations and experiments on a mobie robot suggest that the proposed technique improve TO discrimination and reduces significanty ocaization of invaid sound sources caused by noise. I. INTROUTION Sound source ocaization on mobie robots can be performed using Mutipe Signa assification based on Standard Eigenvaue ecomposition (SEV-MUSI). This method was initiay used for narrowband signas [], and was adapted for broadband sound source ocaization []. This method improves noise robustness by decomposing the covariance matrix obtained from the spectra observations at each microphone in both noise and noisy signa subspaces. The direction of a sound source corresponds to the directiona vector orthogona to the noise subspace. This method performs we as ong as the noise power is weaker than the sound source of interest. Mutipe Signa assification based on Generaized Eigenvaue ecomposition (GEV- MUSI) is proposed to dea with this issue []. SEV- MUSI and GEV-MUSI significanty improve robustness to noise, but remains sensitive to reverberation, and eigenvaue decomposition invoves a high computationa oad. Mutipe Signa assification based on Generaized Singuar Vaue ecomposition (GSV-MUSI) is proposed to reduce computationa oad []. Whie this method remains sensitive to highy reverberant environments, it improves ocaization accuracy because eigenvectors are mutuay orthogona. nother approach for sound source ocaization on mobie robots is to use a deay-and-sum beamformer []. This method reies on the Generaized ross-orreation with Phase Transform (G-PHT) to perform time difference This work was not supported by the Fonds de recherche du Québec - Nature et technoogies (FRQNT). F. Grondin and F. Michaud are with the epartment of Eectrica Engineering and omputer Engineering, Interdiscipinary Institute for Technoogica Innovation, bou. de Université, Québec (anada) JK, {Francois.Grondin,Francois.Michaud}@USherbrooke.ca of arriva (TO) estimation. This method is robust to reverberation but remains sensitive to broadband additive noise. eamforming based on weighted G-PHT, as impemented in the ManyEars framework [], [], is appeaing to perform sound source ocaization: it is robust to reverberation, has ow-compexity, and is robust to additive noise if noise can be masked. Noise masking highy reies on an accurate background noise estimation. To improve robustness to noise of G-PHT, a non-recursive unbounded mask is derived from the instantaneous signa-to-noise ratio (SNR), and the resuts is correated with the intensity of the sound source of interest []. To make the resut independent of the ampitude of the sound source, a recursive soft mask with bounded vaues is used in ManyEars [], []. We aso recenty introduced in ManyEars a hard binary mask to improve its robustness to broadband coherent noise [9]. The masks impemented so far in ManyEars [], [9] rey on the Minima ontroed Recursive veraging (MR) method to estimate background noise [], []. However, background noise usuay changes over time if the robot operates in dynamic environments. omputer fans on the robot can aso start and stop according to the centra processing unit (PU) usage. When noise changes, the MR method requires a few seconds to adapt and to provide an accurate noise estimation. uring this transition, the masks no onger hide noise components, which can ead to ocaization of noise sources instead of speech. To our knowedge, this probematic has not yet been addressed in other works. To overcome this probem, this paper presents a new mask that can dea with noise transition. This mask, which we refer to as transition mask, is an extension of the hard mask previousy introduced [9] and measures the stationarity of previous and future frames to mask noisy time-frequency regions. The paper is organized as foows. Section II briefy presents the weighted G-PHT method and noise masks used for background noise estimation, to then expain our new transition noise mask. Section III describes the experiments conducted in simuation and on a mobie robot patform. II. WEIGHTE G-PHT WITH NOISE MSKS To impement G-PHT, the Short-Time Fourier Transform (STFT) X m[k] is computed using () for each microphone m with frames of N sampes, and spaced by a hop of N sampes. The expression x m [n] represents the signa from each microphone and n is the discrete-time index. Hann window w[n] of N sampes is used to reduce spectra

2 eakage. The variabes and k stand for the frame and bin indexes, respectivey. X m[k] = N n= w[n]x m [ N + n]e jπkn/n () The weighted G-PHT between microphones p and q is expressed by (). The expression (...) stands for the compex conjugate operator, whie the variabe ζpq[k] is a frequency mask introduced to reduce coherent noise contribution. The expression ɛ is aso added to avoid overfow when the spectrum magnitude goes to zero. The weighted G-PHT resut is Rpq[n]. R pq[n] = N N k= ζ pq[k]x p[k]x q[k] X p[k] X q[k] + ɛ ejπkn/n () To estimate TO, the cross-correation resut R pq ies in the interva [, N ] and is mapped on the new interva [ N/ +, N/] using the expression ˆR pq as in (). The moduo N operation ensures that a negative TO vaue, initiay mapped to a vaue greater than N/ with the Inverse Fourier Transform (IFFT) in (), is brought back to the desired range. ˆR pq[n] = R pq [n mod N] () The maximum cross-correation index is obtained by () to provide the pausibe range of a TO vaue, which ies in the interva [ n max pq, n max pq ]. ( τpq = arg max ˆR pq [n]) n n max pq n n max pq () The maximum cross-correation index corresponds to the case when the sound source and the pair of microphones ie on the same pane, and is given by (). The variabes f s, c, x p and x q stand for the samping rate (in sampes/sec), the speed of sound (in m/sec), and the positions of microphones p and q, respectivey. n max pq = ( ) fs x p x q () c The maximum cross-correation energy E pq is given by (). E pq = ˆR pq[τ pq] () To reduce noise contribution, a soft mask introduced in [] reies on the a priori SNR, obtained from the background noise estimated with the MR method. mask is estimated for each microphone channe and both masks are combined to obtain ζ pq[k] defined by (). ζ pq[k] = ζ p[k]ζ q[k] () When this mask is used, broadband coherent noise may eak from each bin and generate undesired dominant peak in the TO estimation resut. To sove this issue, a binary mask for each microphone channe, caed hard mask because it ony takes discrete vaues of and, can be used with the MR method [9].. Transition Noise Mask The MR method is efficient to estimate stationary noise as ong as it can take a few seconds to adapt to the background noise. When a robot operates in a dynamic environment, noise is often non-stationary during sound transitions. For instance, fans instaed on the robots or in the room may start and stop as time goes by, and this introduces abrupt transitions between steady-state segments. To consider such conditions, we define two working hypotheses []: ) Speech phonemes are stationary for ess than msec. ) fter a fast transition, noise stays stationary for more than msec. The transition noise mask expoits these characteristics to differentiate noise from speech. The mask is computed using the product of Xp[k] and Xq[k], as opposed to the soft and hard masks that are computed directy from the individua spectrum of each microphone. The expression Xpq[k] = Xp[k] Xq[k] improves speechnoise discrimination as the inter-channe spectrum magnitude correation is higher for speech than for noise. rectanguar window smooths the og magnitude of the spectrum frequency-wise according to (). The smoothed resut is expressed by Y pq[k], and the rectanguar window is made of (W + ) sampes. The spectra enveope is extracted with the ogarithm and this operation aso reduces the spectra distortion caused by the high energy outiers when the moving average is computed. The expression ɛ is added to avoid underfow when the spectrum power goes to zero. This operation reduces the variance of the og magnitude between adjacent frames. Ypq[k] = W + W k= W og ( X pq[k + k] + ɛ ) () Y pq[k] is used to evauate two parameters: pq[k], the difference between the current power eve and the minimum vaue of a buffer made of the previous frames, using (9): pq[k] = Ypq[k] min { Ypq + [k],..., Ypq[k] } (9) pq[k], the difference between the current power vaue and the minimum vaue within a window of future frames, defined by (): pq[k] = Ypq[k] min { Ypq[k],..., Ypq + [k] } () This introduces a atency of N /f s sec because the window requires sampes ahead in time.

3 To expain the roe of these two parameters, Fig. iustrates their infuences with four types of signa: ) steady state noise (NSS); ) noise eve rises quicky (NRI); ) noise eve drops quicky (NP); ) speech (SPH). Fig. a) to d) iustrate how Ypq[k] reduces the power variance. For pq[k], Fig. f) and h) iustrates that arge differences are observed for NRI and SPH, whie sma differences are observed for the NSS and NP shown in Fig. e) and g). For pq[k], Fig. i) and j) shows that sma differences are observed for NSS and NRI, and Fig. k) and ) iustrates that differences are greater for the NP and SPH. These observations indicate that ony speech generates arge vaues for both the signas pq[k] and pq[k]. This condition, defined as pq[k], can be identified using (), with θ and θ proposed as fixed threshod vaues. pq[k] = { ( ) ( ) pq [k] > θ pq [k] > θ otherwise () The transition noise mask ζpq[k] can therefore be defined in reation to pq[k] as in () and used in () during G-PHT computation. majority vote is performed in (), with being the window size in frames and θ the majority threshod, to fiter out fase detections. For instance, pq[k] may sporadicay trigger a fase vaue of when nonstationary percussive sounds are observed. ( + ) ζpq[k] = pq[k] > θ () = otherwise III. RESULTS In this section, performance using the soft, hard and transition noise masks are presented for experiments conducted in simuation and with a mobie robot. Tabe I presents the parameters used. Ony W,,,, θ, θ, and θ have to be fine tuned empiricay, the others are being set according to environmenta settings. f s is set at khz for the acquisition of high frequency components in speech. N is the same as in [9] and the hop size N ensures an overap of %. The speed of sound c is defined at and. kpa []. The expression ɛ is chosen to avoid overfow or underfow whie sti be sma enough to preserve the precision of () and (). The expression W is set to a vaue that provides a smoothing window that preserves frequency resoution. and are chosen to expoit the non-stationarity of speech, and aso dea with reverberation. and θ, θ and θ are chosen empiricay to capture most of the speech features and reject noisy regions. With the proposed parameters, a atency of N /f s = msec is introduced. atency of this range can be considered negigibe in human-robot voca interaction. The theoretica deays τ t and τ c are associated to the target (i.e., the sound source of interest) and the noise sources, respectivey. Positions of the target source and the noise source in reation to the microphones are known for the TLE I: Parameters used in the experiments Parameters Vaues Parameters Vaues N N f s (Hz) c. θ. ɛ E- θ. W θ experiments conducted, aowing to evauate the performance of TO sound ocaization using the different noise masks. τ t and τ c are defined by () and (), where s t and s c are vectors which hod the cartesian positions of the sound source of interest and noise sources, respectivey. ( ) fs τ t = ( s t x p s t x q ) () τ c = c ( fs c ) ( s c x p s c x q ) () To characterize the performance of the transition noise mask, the foowing metrics are introduced. The expression t pq takes a vaue of when the deay τpq is assigned to the sound source of interest, and a vaue of otherwise. deay τpq is considered to be assigned to the sound source of interest when the absoute vaue of the difference with the theoretica deay τ t is ess or equa to a constant τ, as shown in (). t pq = { τpq τ t > τ τpq τ t τ () Simiary, the expression c pq is set to when the noise source generates the coherence noise deay, and a vaue of otherwise. When coherent noise dominates, the TO is assigned to the noise source and the absoute vaue of the difference with the deay τ c is ess or equa to τ, provided by (). c pq = { τ pq τ c > τ () τ pq τ c τ higher energy eve normay indicates a higher confidence eve, which is usuay used with a subsequent confidence-based decision stage []. To evauate energy eves, t pq and c pq are mutipied by the associated maximum cross-correation energy Epq, as given by () and (). These weighted averages give more importance to deays associated to a high energy eve, and thus provide meaningfu metrics to measure overa performance. ( L ) ( / L ) T pq = () pq = t pqepq = ( L c pqepq = = ) ( / L = E pq E pq ) ()

4 (a) NSS: Y pq[k] (b) NRI: Y pq[k] (c) NP: Y pq[k] (d) SPH: Y pq[k] (e) NSS: pq [k] (f) NRI: pq [k] (g) NP: pq [k] (h) SPH: pq [k] (i) NSS: pq[k] (j) NRI: pq[k] (k) NP: pq[k] () SPH: pq[k] Fig. : Signas Y m[k], m[k] and m[k] for NSS, NRI, NP and SPH TLE II: Simuation parameters Parameters. Simuation Resuts Vaues x.î.ĵ +.ˆk x.î +.ĵ +.ˆk s t +.î +.ĵ +.ˆk s c.î +.ĵ +.ˆk τ Simuations are performed under a wide range of reverberation eves and SNRs. Speech is used as the sound source of interest, and the noise source is made of white noise bursts. These sources are convoved with their respective room impuse responses (RIRs) generated for a room of dimensions m m m using en and erkey image method []. Tabe II presents the parameters used for the simuations. The unit vectors î, ĵ and ˆk represent the x, y and z axes of a three-dimensiona cartesian coordinate system. The vector ˆk points toward the ceiing of the simuated room. The expression τ is chosen to be arge enough to capture deays cose to the theoretica vaue. The simuations are performed with more than minutes of speech, i.e., frames. Figure iustrates the noisy speech spectrum and the soft, hard and transition noise masks, when the reverberation time (RT) is msec and the SNR is d. Noise bursts are active for one second, and spaced by periods of one second of sience. The soft and hard masks erroneousy capture noise bursts in segments, and. The transition mask rejects noise bursts in segments and, and eave the noisy speech unatered in segment for the regions where speech is more powerfu than noise, as desired. For a masks, the cean speech segment is eft unatered, as desired. Figure iustrates the distribution of τ t and τ c for each type of masks, when the energy eve Epq is greater than zero. The soft and hard masks ead to a distribution peak at the noise deay τ c, whie the distribution obtained with the transition mask reaches its maximum vaue at the target deay τ t. Tabe III presents T pq and pq for a wide range of reverberation eves and SNRs. The sum of T pq and pq may not equa to % if some deay vaues are not assigned to the sound source of interest nor to the coherent noise source. In the idea case, T pq shoud reach %, and pq %. Resuts indicate that the transition mask provide better performance for a reverberation eves and SNR vaues. With ow reverberation (RT = msec) and high signa-tonoise ratio (SNR = d), T pq and pq reach 99% and %, respectivey, using the transition mask, compared T pq = 9% and pq = % with the hard mask and T pq = % and pq = % using the soft mask. In high reverberation conditions (RT = msec) and ow signa-to-noise ratio (SNR = d), the transition mask outperforms the hard and soft masks, with T pq = % and pq = % compared to T pq = % and pq = 9% using the hard mask and

5 TLE III: Tpq and pq (in parenthesis) using the soft, hard and transition masks RT (msec) (a) Noisy speech spectrum (b) Soft mask (c) Hard mask 9 9 (d) Transition mask Fig. : Sound spectrum with RT = msec and SNR = d. In a), high and ow power eves are shown in red and bue respectivey. In b) to d), the back and white coors stand for the vaues and, respectivey. Percentage (%) SNR (d) Soft Mask Hard Mask Transition Mask % (%) 9% (%) % (%) % (%) % (9%) % (%) % (%) % (%) % (9%) % (9%) % (9%) % (%) % (%) % (9%) % (9%) % (%) % (%) % (%) % (9%) % (%) 9% (%) % (%) % (9%) 9% (%) % (%) % (%) % (9%) % (9%) % (%) % (%) % (%) % (9%) % (9%) % (%) % (%) % (9%) % (9%) % (%) % (%) % (9%) % (9%) % (%) % (%) % (9%) % (9%) 99% (%) 99% (%) 99% (%) 9% (%) 99% (%) 9% (%) 9% (%) 9% (%) 9% (%) 9% (%) 9% (%) 9% (%) % (%) % (%) % (%) % (%) % (%) % (%) % (%) % (%) % (9%) % (9%) % (9%) % (9%) % (%) Soft Hard Proposed eay (sampe) Fig. : eay distribution generated with the soft, hard and transition masks for simuation with RT = msec and SNR = d Tpq = % and pq = 9% using the soft mask. Fig. : Pair of microphones used on the IRL- robot. Experiments using a Mobie Robot Figure shows the IRL- robot equipped with an microphone array. IRL- was used to vaidate the performance of the approach in a rea-ife environment. To scae down the compexity of the experiment and to simpify anaysis, ony two microphones ocated on the front of the IRL- are used. Since ManyEars reies on the sum of the weighted G-PHT between each pair of microphones, the proposed method with two microphones can easiy be adapted to an -microphone approach such as ManyEars. Mae speech is payed by a oudspeaker instaed on the eft of the robot, and a noisy hairdryer is turned on and off by a participant standing on the right of the robot. The reverberation eve in the room is RT = msec. When the hairdryer is active, the SNR is -. d for the microphone on the oudspeaker side, and -. d for the second microphone on the hairdryer side. Figure a presents the signa spectrum for this experiment. It shows that the hairdryer dominates speech in segments,,,, E and F, and that the soft and hard masks erroneousy capture the hairdryer noise. The noise is dominant over speech, and therefore the transition mask reject a time-frequency regions when the noise is active, and captures speech when noise is inactive. Figure iustrates the distribution of τt and τc obtained

6 [k] ) (a) Noisy speech spectrum (og Xm E F (b) Soft mask E F (c) Hard mask E F (d) Transition mask Fig. : Signa spectrum for the experiment conducted using IRL-. High and ow power eves are shown in red and bue respectivey, in a. The back and white coors stand for the vaues and, respectivey, in b, c and d. Percentage (%) Soft Hard Proposed eay (sampe) Fig. : eay distribution generated using the soft, hard and transition masks on the IRL- robot with each type of masks, when the energy eve Epq is not zero. Using the soft or the hard masks ead to a distribution peak at the noise deay τc, whie the distribution obtained with the transition mask reaches its maximum vaue at the target deay τt. The soft mask generates weighted averages of Tpq = % and pq = %, and the hard mask produces weighted averages of Tpq = % and pq = %. Using the transition mask resuts in Tpq = 9% and pq = %, significanty improving robustness to noise compared to the soft and hard masks. IV. ONLUSION This paper presents a transition noise mask for TO estimation based on the weighted G-PHT to improve robustness to coherent broadband noise with abrupt transitions. The transition noise mask reies on the non-stationarity of speech, and resuts indicate that it outperforms soft and hard masks used by the weighted G-PHT approach. It requires the introduction of a sma atency, which reveas to be an acceptabe trade-off to avoid invaid ocaization of sound sources generated by noise: for instance, the presence of invaid sound sources coud ead to inappropriate responses of the robot (such as reorienting its head in direction of the oudest sound source) if additiona processing is required to identify invaid sound sources as noise. The next step with this work is to integrate the use of this transition noise mask with the impementation of the weighted G-PHT in ManyEars. We aso pan to use the information provided by the transition mask to improve computation in ManyEars, for instance by pausing when most frequency bins in the transition mask equa to. R EFERENES [] R. Schmidt, Mutipe emitter ocation and signa parameter estimation, IEEE Transactions on ntennas and Propagation, vo., no., pp., 9. []. Ishi, O. hatot, H. Ishiguro, and N. Hagita, Evauation of a musicbased rea-time sound ocaization of mutipe sound sources in rea noisy environments, in Proc. of the IEEE/RSJ Int. onf. on Inteigent Robots and Systems, 9, pp.. [] K. Nakamura, K. Nakadai, F. sano, and G. Ince, Inteigent sound source ocaization and its appication to mutimoda human tracking, in Proc. of the IEEE/RSJ Int. onf. on Inteigent Robots and Systems,, pp.. [] K. Nakamura, K. Nakadai, and G. Ince, Rea-time super-resoution sound source ocaization for robots, in Proc. of the IEEE/RSJ Int. onf. on Inteigent Robots and Systems,, pp []. adai, J.-M. Vain, F. Michaud, and P. arabi, Evauating reatime audio ocaization agorithms for artificia audition in robotics, in Proc. of the IEEE/RSJ Int. onf. on Inteigent Robots and Systems, 9, pp.. [] J.-M. Vain, F. Michaud, and J. Rouat, Robust ocaization and tracking of simutaneous moving sound sources using beamforming and partice fitering, Robotics and utonomous Systems, vo., no., pp.,. [] F. Grondin,. Le tourneau, F. Ferand, V. Rousseau, and F. Michaud, The ManyEars open framework, utonomous Robots, pp.,. [] J.-M. Vain, F. Michaud, J. Rouat, and. Le tourneau, Robust sound source ocaization using a microphone array on a mobie robot, in Proc. of the IEEE/RSJ Int. onf. on Inteigent Robots and Systems,, pp.. [9] F. Grondin and F. Michaud, Time difference of arriva estimation based on binary frequency mask for sound source ocaization on mobie robots, in Proc. of the IEEE/RSJ Int. onf. on Inteigent Robots and Systems,. [] I. ohen and. erdugo, Noise estimation by minima controed recursive averaging for robust speech enhancement, IEEE Signa Processing Letters, vo. 9, no., pp.,. [] Y. Ephraim and. Maah, Speech enhancement using a minimum mean-square error og-spectra ampitude estimator, IEEE Transactions on coustics, Speech and Signa Processing, vo., no., pp., 9. [] J. enesty, M. M. Sondhi, and Y. Huang, Springer Handbook of Speech Processing. Springer Science & usiness Media,. []. Zuckerwar, Handbook of the Speed of Sound in Rea Gases. Esevier Science,. [] J. en and. erkey, Image method for efficienty simuating sma-room acoustics, Journa of the coustica Society of merica, vo., no., pp. 9 9, 99.