An Application System of Probabilistic Sound Source Localization

Size: px

Start display at page:

Download "An Application System of Probabilistic Sound Source Localization"

Abel Shields
6 years ago
Views:

Inernaional Conference on Conrol, Auomaion and Sysems 28 Oc.

2 1 Cener for Cogniive Roboics Research, Korea Insiue of Science and Technology, Seoul, Korea (Tel : +82-2-958-563; E-mail: ssyeom, himelys, hskim, moon_777, cjs@kis.re.kr) 2 Deparmen of Elecrical and Elecronic Engineering, Yonsei Universiy, Seoul, Korea (Tel : +82-2-2123-2868; E-mail: mignpark@yonsei.

Time Delay of Arrival (TDOA) beween wo microphones hrough cross-correlaion mehod has been used for sound localizaion in our robo plaforms.

Here, a probabilisic mehod for sound source localizaion is presened. And is DSP implemenaion is applied o he acive audiion sysem ha we developed. Keywords: sound source localizaion, DSP, probabilisic.

Bu i has some problem in porabiliy and power consumpion from a roboic poin of view.

I includes wo funcions: VAD (voice aciviy deecion) funcion, and enhanced SSL (sound source localizaion) algorihm. The SSL is one of ways ha keep he relaion wih robo.

Kalman filers are commonly used o localize a single objec under a Gaussian models and linear dynamics. These mehods have been applied o audio and visual objec racking problems.

1 Inernaional Conference on Conrol, Auomaion and Sysems 28 Oc , 28 in COEX, Seoul, Korea An Applicaion Sysem of Probabilisic Sound Source Localizaion Seung Seob Yeom 1,2, Yoon Seob Lim 1, Hong Sick Kim 1,2, Jae Moon Lee 1,2, Jong Suk Choi 1 and Mignon Park 2 1 Cener for Cogniive Roboics Research, Korea Insiue of Science and Technology, Seoul, Korea (Tel : ; ssyeom, himelys, hskim, moon_777, cjs@kis.re.kr) 2 Deparmen of Elecrical and Elecronic Engineering, Yonsei Universiy, Seoul, Korea (Tel : ; mignpark@yonsei.ac.kr) Absrac: This paper describes a DSP implemenaion of probabilisic sound source localizaion algorihm. Time Delay of Arrival (TDOA) beween wo microphones hrough cross-correlaion mehod has been used for sound localizaion in our robo plaforms. However, since cross-correlaion values are highly dependen on he upcoming sound signal and acousic environmen, he ime delay values and he corresponding localizaion resuls are easily perurbed. Here, a probabilisic mehod for sound source localizaion is presened. And is DSP implemenaion is applied o he acive audiion sysem ha we developed. Keywords: sound source localizaion, DSP, probabilisic. 1. INTRODUCTION Sound processing sysem of a robo is commonly composed of compuer, daa acquisiion board and sound amplificaion board for complicaed signal processing. Bu i has some problem in porabiliy and power consumpion from a roboic poin of view. Insead of PC sysem, we made a high performance DSP sysem composed of SLP (sound localizaion processing board) and NAB (nonlinear amplified board). I includes wo funcions: VAD (voice aciviy deecion) funcion, and enhanced SSL (sound source localizaion) algorihm. The SSL is one of ways ha keep he relaion wih robo. There exis numerous approaches which deal wih a reliable sound source localizaion, employing a variey of feaures, racking schemes and sensor seups. Kalman filers are commonly used o localize a single objec under a Gaussian models and linear dynamics. These mehods have been applied o audio and visual objec racking problems. However, using he Kalman filers only, adequae racking performance can no be obained when measuremen is done in a noisy and cluered environmen. 2.1 Overview Our sysem archiecure is composed of wo pars. One is he Nonlinear Amplified Board (NAB) ha amplifies sound signals and digiizes he amplified sound signals. The oher is Sound Localizaion Processing Board (SLP) ha processes he digiized sound signals. Fig.2 shows he hardware archiecure in his sysem. Fig. 2 Hardware Archiecure 2.2 NAB (Nonlinear Amplified Board) Our nonlinear amplifier board is composed of hree pars, nonlinear amp, gain conrol, and codec as shown in Fig.3. Fig. 3 NAB (Nonliear Nonlinear Amplified Board) Fig. 1 SLP & NAB Sysem 2. SYSTEM ARCHITECTURE Nonlinear amplifier Nonlinear amplificaion is able o make he amplificaion rae variable according o he signal magniude. This is he reason why i can have wide deecable range. If he raio of amplificaion is fixed a 243 Auhorized licensed use limied o: Sichuan Universiy. Downloaded on February 26, 29 a 22:29 from IEEE Xplore. Resricions apply.

a small one, speech signal occurring a he long disance can be hardly exraced from he acquired signal whose magniude is small enough for he conens of speech o be cancelled by noise.

2 a small one, speech signal occurring a he long disance can be hardly exraced from he acquired signal whose magniude is small enough for he conens of speech o be cancelled by noise. To he conrary, wih large raio, he speech signal occurring nearby someimes may be sauraed in he AD conversion. Here, we propose nonlinear amplificaion where smaller signals can be amplified wih larger amplificaion raio o implemen he nonlinear propery Gain conrol The SSM2166 has o se up resisor values for compression raio and amplificaion raio. Here, we use programmable mehod wih digial poeniomeer chip (AD524). The AD524 is conrolled by values in regiser buffer. The values in he regiser buffer are generaed by parameer exracion algorihm in SLP module Codec Nonlinearly amplified analog signals pass hough a codec, AD1836. The codec does A/D conversion ino 16 khz digial signals. And hen i sends he convered signals o he SLP module. 2.3 SLP (Sound Localizaion Processing Board) We have developed SLP module wih a DSP (BF533) for sound localizaion. The SLP module has feaures of low power, high performance, adapively embedded sysem of small-size and convenien developmenal environmen serviced by Analog Device Company. Therefore i has higher marke compeiion, and can be mouned on various producs. Here, we will show he developmen sage of he SLP module and will inroduce is feaures. Firs of all, in his subjec, le us describe BF533 series made by Analog Device Company. The BF533 has feaures as follows. 1) Up o 6 MHz high performance Blackfin processor 2) Two 16-bi MACs, wo 4-bi ALUs, four 8-bi video ALUs, and 4-bi shifer 3) RISC-like regiser and insrucion model for ease of programming and compiler-friendly suppor 4) Advanced debug, race, and performance monioring.8v o 1.2V core VDD wih on-chip volage regulaion 3.3V and 2.5V oleran I/O 5) 16-ball mini-bga, 169-ball lead free PBFA, and 176-lead LQFP packages 6) Memory - Up o 148K byes of on-chip memory 7) Two dual-channel memory DMA conrollers 8) Exernal memory conroller wih glueless suppor for SDRAM, SRAM, FLASH, and ROM 9) Flexible memory booing opions from SPI and exernal memory Because BF533 has high performance of 6MHz, wo dual-channel memory DMA conrollers and ec, i is he proper sysem suiable o complex algorihms and high compuing power. Tha is he reason why we seleced BF533 series. We conduced sound localizaion algorihms based on DSP (BF533 series) and experimens as follows. Firs, we have designed SLP block diagram comprising ADSP-BF533 for main processor, flash memory for local memory, SPI por for auo gain conrol of NAB, UART por for PC inerface, and TDM por for codec (AD1836). The deailed procedures of sound localizaion are he original speech signals are acquired by hree - microphones and inpu sound signal is amplified by nonlinear amplifier board. Here, SSM2166 values of CR and VCA gain are conrolled by DSP (BF533) module. The SLP evaluaion Board is shown in Fig.4 Fig. 4 SLP (Sound Localizaion Processing Board) 3. LOCALIZATION ALGORITHM 3.1. Microphone Arrangemen There are los of possible causes ha resul in false localizaion of sound source, such as noises around he source, reverberan signals ino he microphone and arrangemen of microphones. The errors in sound localizaion due o noises and reverberan signals can be deal wih various filering mehods, one of which is proposed in his aricle. Errors by hardware configuraion should be prevened in advance. If he region of ineres is only fron side of robo, we can deermine he locaion of sound source wih ime delay value wih only wo microphones. However, i is difficul o find ou wheher he source lies in he fron side or back side of robo because he source locaed in he opposie of real source can generae he same ime delay value. To clear he ambiguiy in source locaion wih only laerally posiioned wo microphones, addiional microphones are needed and some researchers deploy several microphones in line. However, we jus added one microphone and displaced hem riangularly as shown in see Fig.5. Source C III θ II IV Fig. 5 Microphone arrangemen A I V VI B 244 Auhorized licensed use limied o: Sichuan Universiy. Downloaded on February 26, 29 a 22:29 from IEEE Xplore. Resricions apply.

3 3.2. Probabilisic Process Even if microphones are arranged riangularly, environmenal noise and acousical condiion around robo may cause error in calculaing ime delay beween microphones. To deal wih an inappropriae change of ime delay beween wo microphones which is he main reason of error in localizaion, markovian process is applied o compue a ime delay. Time delay value beween wo microphones is seleced as a sae variable. X { T } = Δ (1) where Δ T is ime delay beween microphone i and j. Wha we wan o calculae wih his mehod is how much currenly calculaed ime delay is reliable according o he observed sound signals. The probabiliy of curren ime delay up o curren observaion of microphone pair ( Z = { z1, z2,, z }) can be wrien as equaion (2). ( ) Bel() = P X Z (2) Summarizing, o localize a sound source we need o calculae he probabiliy recursively a each ime sep. This process is composed of wo seps predicion and updae sep Predicion Sep In predicion sep, moion model is defined o predic he curren sae probabiliy under he assumpions ha a sound source can no move over a cerain angle in one ime frame and curren sae depends on he previous sae. Hence, he moion model ha we applied is described wih Gaussian model as equaion (3) 2 ( X X 1) ~ ( X 1, ) P N σ (3) Wih his moion model, we can esimae he probabiliy of each ime delay sae as equaion (4). ( ) X P X Z = P( X X ) P( X Z ) (4) 1 Predicion of curren sae is presened as summaion because ime delay value beween wo microphones is discree and is dependen on he sampling frequency Updae Sep To obain he poserior disribuion ( X Z) P, measuremen likelihood model is proposed o incorporae sound informaion from microphones. We propose a measuremen likelihood model as cross-correlaion value spanning around possible ime delay value of cerain microphone pair. Pz ( X = τ ) = + x i() xj ( τ) 2 2 i() xj() x Where xi( ) is sound signal of microphone i a ime. The poserior disribuion over X is obained by Bayses heorem: 1 Pz X PX Z Z 1 Pz ( Z ) d (5) ( ) ( ) P( X ) = (6) The poserior represens he probabiliy disribuion of ime delay and we may expec ime delay of cerain microphone pair as he value when his probabiliy is he highes. And his whole process is independenly applied o each microphone pair Pos Processing In our microphone configuraion, ime delay values are specifically deermined according o he sound source locaion. Fig. 6 shows his characerisic of our microphone configuraion a each sound posiion. Since poserior disribuion depends on he measuremen likelihood model, undesirable ime delay ses can be seleced wih only probabilisic process shown above. To filer ou hose unwaned ime delay se, only if calculaed ime delay value corresponds o one of 6 regions in Fig. 6, we adop he ime delay as curren ime delay value and vice versa afer poserior compuaion of each microphone pair. Time Delay(msec) VI I II III IV V VI Angle(deg) Fig. 6 Characerisics of ime delay value Sound Localizaion Since we deermine a highly probable ime delay pairs, we can localize a sound source. A ime delay grid for a sound source is pre-deermined along 36 degrees and i is assumed o be siuaed a a specific disance. Localizaion likelihood is defined as (7). L( θ) = 1/ ( ( θ) ˆ ) (7) i j where ( θ ) is ime delay grid and ˆ is ime delay 2 AC AB BC 245 Auhorized licensed use limied o: Sichuan Universiy. Downloaded on February 26, 29 a 22:29 from IEEE Xplore. Resricions apply.

value deermined by probabilisic mehod. Sound locaion is deermined such ha he likelihood has maximum value. Microphones are arranged riangularly.

To deal wih he inappropriae change of ime delay beween wo microphones which is he main reason of error in localizaion, we apply Markovian process o he

7 shows he srucure of acive audiion sysem applying he probabilisic SSL. The sysem is composed of SLP & NAB par, moor par, and vision par.

And i has a monior funcion hrough vision daa from a camera deached such ha i can oupu he deeced human on he LCD.

EXPERIMENTAL RESULT We apply proposed mehod o localize a sound source in several siuaions; when source sands sill or moves a cerain angular speed

A each cross-correlaion, 256 samples are used wih 5% overlap according o number of cross-correlaion.

SNR of inpu signal is around 13~15dB during experimen. Fig. 1 shows a comparison resul when we applied his algorihm or no.

This means ha he proposed algorihm can filer ou unwaned signals such as environmenal noise a he end of inpu sound signals, which cause flucuaions of

9 shows monior program ha checks he presen condiion of sysem. I can know ha direcion, error message, noise level, VAD level.

4 value deermined by probabilisic mehod. Sound locaion is deermined such ha he likelihood has maximum value. Microphones are arranged riangularly. Here, environmenal noises and acousical condiion around he robo may cause error in calculaing ime delay beween microphones. To deal wih he inappropriae change of ime delay beween wo microphones which is he main reason of error in localizaion, we apply Markovian process o he calculaion of he ime delay. 4. APPLICATION SYSTEM Fig.9 Monior program Table.1 Proocol beween sysem and PC Fig. 7 Srucure of acive audiion sysem Fig. 7 shows he srucure of acive audiion sysem applying he probabilisic SSL. The sysem is composed of SLP & NAB par, moor par, and vision par. Direcional value of SLP & NAB sysem conrols moor and he vision par is resulanly moved oward he speaker. And i has a monior funcion hrough vision daa from a camera deached such ha i can oupu he deeced human on he LCD. The sysem can be applied o auo-order sysems, conference sysems and smar webcams in near fuure. Fig. 8 shows real acive audiion sysem. 5. EXPERIMENTAL RESULT We apply proposed mehod o localize a sound source in several siuaions; when source sands sill or moves a cerain angular speed around a robo. Sound signals are sampled a 16 khz. Each frame is composed of 16 samples and moved while some samples (ex. 5%) are overlapped. A each cross-correlaion, 256 samples are used wih 5% overlap according o number of cross-correlaion. Afer he cross-correlaion sep, we performed a probabilisic process and pos filering sep for each cross-correlaion value. SNR of inpu signal is around 13~15dB during experimen. Fig. 1 shows a comparison resul when we applied his algorihm or no. The sound source is posiioned a 3 degree. The proposed mehod shows more sable ime delay han hose ha are deermined only by cross-correlaion. This means ha he proposed algorihm can filer ou unwaned signals such as environmenal noise a he end of inpu sound signals, which cause flucuaions of ime delay values. Localizaion for moving sound is also imporan for a reliable ineracion wih a robo for speaker. 1 Fig.8 Real acive audiion sysem Fig. 9 shows monior program ha checks he presen condiion of sysem. I can know ha direcion, error message, noise level, VAD level. An Arrow in he monior program indicaes he direcion of sound source. Oher informaion is represened by he characer. And Table. 1 Shows he proocol beween acive audiion sysem and PC. Angle Time Delay Time Delay Afer Before Before 1 AC AB BC -1 Afer 1 AC AB BC -1 Time(sec) Fig. 1 Resul of Sound Source Localizaion 246 Auhorized licensed use limied o: Sichuan Universiy. Downloaded on February 26, 29 a 22:29 from IEEE Xplore. Resricions apply.

Fig. 11 shows a localizaion resul when a person is moving around a robo from -7 o o 6 o. Solid line is esimaed posiion of source by proposed mehod and do line is a real posiion of he source.

(duraion is abou 16msec). 8 6 4 Esimaed Angle Original Angle Table. 2 Resul of experimen Angle(deg) 2-2 -4-6 -8 5 1 15 2 25 3 Time(sec) Fig.

Array microphones are locaed beween TV and air condiioner. And he speaker is locaed in fron of he sofa. Disance beween mics and speaker is 2 meers.

5 Fig. 11 shows a localizaion resul when a person is moving around a robo from -7 o o 6 o. Solid line is esimaed posiion of source by proposed mehod and do line is a real posiion of he source. Overall, racking is accepable even hough here is some delay for localizaion, which is mainly because of moion model in he predicion sep under he assumpion ha he source would no move in one ime frame (duraion is abou 16msec) Esimaed Angle Original Angle Table. 2 Resul of experimen Angle(deg) Time(sec) Fig. 11 Localizaion of moving source Nex, we performed more realisic experimens in our demo room. Fig. 12 shows he experimenal se-up. There are oal 36 spos for sound s sources. Array microphones are locaed beween TV and air condiioner. And he speaker is locaed in fron of he sofa. Disance beween mics and speaker is 2 meers. This experimenal se-up is buil for a scenario ha a person siing on he sofa calls for he robo. 6. CONCLUSIONS Sound is one of he essenial modaliies for a reliable ineracion wih robo, which delivers a subsanial amoun of spaial, emporal informaion beween human and robo (locaion, ime delay). According o he localizaion performances shown in his aricle, spaial informaion hrough he improved compuaion of ime delay beween wo microphones is feasible in a probable siuaion. In sum, we have enhanced a sound localizaion using a probabilisic mehod for a reliable ineracion wih a robo via acousic informaion. Using his mehod, a robo can localize a cerain speaker and can use he resul while ineracing wih human or robo. Acually we implemened he mehod o DSP sysem and applied he auo order sysem. We see he performance of he mehod is beer han common TDOA s. Also, he sysem has a plan o be mouned on human-like robos. REFERENCES Fig. 12 Experimenal se-up Experimens are performed 1 imes repeaedly a each angle. Table. 2 shows he experimenal resuls. A se of phrases, Hi, H-robo, come on. (SNR 15dB~2dB) was used for he speech. And we pu he es on hree experimenal siuaions. The siuaions are quie circumsance, circumsance wih he air condiioner urned on (considering only indirec influence of he air), and circumsance wih he TV urned on. The performance of he common TDOA mehod resul in ha he successful rae is 64% in quie circumsance, 77% wih only he air condiioner urned on, and 84% wih only he TV urned on. However our proposed probabilisic SSL showed he improved resuls of 84%, 87%, and 84% respecively. [1] H.D. Kim, J.S. Choi, and M.S. Kim, Human-Robo Ineracion in Real Environmens by Audio-Visual Inegraion, Inernaional Journal of Conrol, Auomaion, and Sysems, vol. 5, no. 1, 27, pp. 61nal Deecion and Esimaion. New York: Springer-Verlag, 1985, ch. 4. [2] D.E. Surim, M.S. Brandsein, and H.F. Silverman, Tracking muliple alkers using microphone-array measuremens, IEEE Inernaional Conference on Acousics, Speech, and Signal Processing, [3] V. Pavlovic, A. Garg, and J. Rehg, Mulimodal speaker deecion using error feedback dynamic Bayesian neworks, Proceedings of he IEEE CVPR, Hilon Head Island, SC, 2. [4] J. Vermaak, M. Gangne, A. Blake, and P. Perez, Sequenial Mone Carlo fusion of sound and vision for speaker racking, Proceedings of IEEE ICCV, Vancouver, July 21. [5] A. Douce, N. de Freias, and N. Gordon, Sequenial Mone Carl o Mehods in Pracice, Springer-Verlag, 21 [6] D.B. Ward, E.A. Lehmann, and R.C. Williamson, Paricle Filering Algorihms for Tracking an Acousic Source in a Reverberan Environmen, IEEE Transacions on Speech and Audio Processing, vol. 11, 23, pp [7] SSM2166 Technical Manual, Analog Device, [8] Okuno, H., e al, Assessmen of General Applicabiliy of Robo Audiion Sysem by Recognizing Three simulaneous Speeches, Proc. of 24 IEEE/RJS In. Conf. on Inelligen Robos and Sysems, Sep. 28 ~ Oc. 2, 24, Sendai, Japan [9] K. Nakadai, T. Masui, H. G. Okuno, and H. Kiano, "Acive Audiion Sysem and Humanoid Exerior Design," in Proc. IEEE/RSJ In. Conf. Inelligen Robos and Sysems, Takamasu Japan, Nov. 2, pp Auhorized licensed use limied o: Sichuan Universiy. Downloaded on February 26, 29 a 22:29 from IEEE Xplore. Resricions apply.

6 [1] K. Nakadai, H. G. Okuno, and H. Kiano, "Epipolar Geomery Based Sound Localizaion and Exracion for Humanoid Audiion," in Proc. IEEE/RSJ In. Conf. Inelligen Robos and Sysems, Maui, Hawaii USA, Nov. 21, pp [11] Seung Seob Yeom, Jong Suk Choi, Yoon Seob Lim, Min-young Park and Munsang Kim, DSP Implemenaion of Sound Source Localizaion wih Gain Conrol, Inernaional coference on Conrol, Auomaion and Sysems 27, Oc, 17-2, 27 in COEX, Seoul, Korea. [12] Yoon Seob Lim, Jong Suk Choi and Munsang Kim, Probabilisic Sound Source Localizaion, Inernaional coference on Conrol, Auomaion and Sysems 27, Oc, 17-2, 27 in COEX, Seoul, Korea. 248 Auhorized licensed use limied o: Sichuan Universiy. Downloaded on February 26, 29 a 22:29 from IEEE Xplore. Resricions apply.

Role of Kalman Filters in Probabilistic Algorithm

Role of Kalman Filters in Probabilistic Algorithm Volume 118 No. 11 2018, 5-10 ISSN: 1311-8080 (prined version); ISSN: 1314-3395 (on-line version) url: hp://www.ijpam.eu doi: 10.12732/ijpam.v118i11.2 ijpam.eu Role of Kalman Filers in Probabilisic Algorihm