Speaker Tracking and Identifying based on Indoor Localization System and Microphone Array

Transcription

1 Speaker Trackng and Identfyng based on Indoor Localzaton System and Mcrophone Array Xaoe Chen, Yuanchun Sh, Wenfeng Jang Department of Computer Scence and Technology, Tsnghua Unversty, Beng, Chna Abstract Ths paper presents a novel multmodal system to track the partcpants and dentfy the actve speaker n the smart meetng room. Indoor localzaton system, Ccada, s used to obtan the locaton and dentty nformaton of the partcpants. Ccada, developed by our research group, can provde much hgher localzaton precson than most vsual trackng systems avalable. Four eght-element mcrophone arrays are used to estmate TDOA (Tme Dfference of Arrval) of the audo sgnals, whch are ndcators of the locaton of the speaker. Integratng these two cues from Ccada and mcrophone arrays nto a statstcal framework, results n a more robust soluton to speaker localzaton and dentfcaton. The experments show that the average error n the partcpant locaton estmaton s about 15 cm and the system can acheve approxmately 90% accuracy n speaker dentfcaton on average. 1. Introducton Precse person localzaton, whch has become a basc technology n a smart meetng room, s useful n a varety of domans, ncludng experence recordng, meetng analyss, and camera steerng. Meanwhle, the relable dentfcaton of the speaker, pontng who s speakng, can provde addtonal nformaton for vdeoaudo ndexng and retreval tasks n experence recordng. The task of speaker localzaton and dentfcaton poses three basc problems: locatng all the partcpants n the smart meetng room, dstngushng the speaker from other slent partcpants, and determnng the speaker s dentty. Conventonally, vsual-audo trackng systems, such as [1][2][3], am at addressng the frst two problems. Combnng the vsual and the acoustc modalty n a multmodal framework, these systems adopt approaches such as Kalman flter and Partcle flter to track a sngle speaker, takng advantage of the complementary nformaton provded by the two modaltes. Vsual cues are used to track all the partcpants n the room, whle mcrophone arrays are used to select the actve speaker. However, vsual trackng systems usually suffer from obstructng and lghtng varatons, resultng n low precson. The speaker dentfcaton problem s recently addressed by applyng bo-metrc measurements, ncludng acoustc speech features classfcaton [4] and face recognton [7]. However, these systems need to acqure pror bo-metrc nformaton of all the partcpants n the smart meetng room, so they can hardly be relable, f several unfamlar users enter the room to on the meetng. A novel multmodal system s proposed n ths paper to cope wth these problems, whch ams to determne the 2D poston of all the partcpants n the smart meetng room and dentfy the actve speaker. In our system, a small Ccada badge contanng dstnctve user ID s attached to each partcpant to determne hs/her dentty and track hm/her, whle mcrophone arrays are used to dstngush the speaker from other slent partcpants. The experment results demonstrate that our system s more effectve than tradtonal vsual-audo trackng systems and bometrc dentfyng systems. 2. Related Work Numerous approaches have been proposed to solve the speaker localzaton problem. In Pngal s work [1], a sngle camera s used to locate all the users n the camera vew, whle a two-element mcrophone array s used to select the actve speaker. They use cross correlaton technque to estmate the TDOA, whch determnes a 3D hyperbola ndcatng the locus of possble acoustc source. Then the speaker poston can be obtaned by calculatng the ntersecton ponts of the hyperbola and the lne from the camera to the user. However, they have gnored the speakers who are not 21st Internatonal Conference on Advanced Informaton Networkng and Applcatons Workshops (AINAW'07)

2 vsble n the camera but probably n the same drecton as a vsble person relatng to the mcrophone array. In ths case, the system would make wrong decsons n speaker detecton. In Sracusa s work [2], the nvsble speaker problem s tackled by adoptng a statstcal framework to assocate the audo sgnal wth a subect n the scene. Though they have acheved mproved performance when a person who s not tracked s speakng, they could hardly handle the stuaton that the face of a specfc user s not captured clearly because of head orentaton. Both [1] and [2] rely on face detecton to locate the users. These two systems, both usng only one camera, wll probably be problematc when the speaker turns away from the camera whle talkng to other users. Furthermore, they can not determne the dentty of the speaker. In Busso s system [4], a full-crcle 360 degree tabletop camera and four synchronzed cameras are used to cover every corner of the meetng room. Meanwhle, the locaton of each user s computed based on 3D polygon surface model and a face detecton technque. Therefore, head orentaton wll not affect the performance sgnfcantly. However, polygonal 3D approxmaton and large dstorton of the full-crcle camera wll result n lower localzaton precson. Speaker dentfcaton s mplemented usng the Gaussan Mxture Model. Snce pror model tranng s needed, the system s not flexble n the case that some unfamlar users enter the room durng the meetng. Moreover, the acoustc sgnals captured by mcrophone array n far-feld are easly corrupted by nose, so the performance of speaker dentfcaton wll be much worse n comparson wth the case when clean sgnal captured by a close-talkng mcrophone s used. Other bo-metrc speaker dentfcaton systems, face recognton n [7] for nstance, wll face smlar problems. As dscussed above, some researches have been conducted concernng speaker localzaton and dentfcaton. However, the precson of localzaton n conventonal vsual trackng system s relatvely low, due to the obstructng, lghtng varatons and camera dstorton. Bo-metrc speaker dentfcaton system wll suffer from cumbersome pror model tranng. Hence, we propose a novel multmodal system to tackle the problem. 3. The proposed system Our multmodal system s based on ndoor localzaton system called Ccada [5] and mcrophone array. The goal of Ccada s to detect the locaton of the partcpants n the meetng room and determne ther denttes. A small badge contanng dstnctve user ID s attached to each partcpant, so every partcpant has a unque ID and can be dstngushed from the others. Wth the badges, all the partcpants can be tracked smultaneously and precsely. Four mcrophone arrays fxed to the walls are used to capture the audo feature of the speaker. Each mcrophone array has eght omndrectonal mcrophones n a horzontal lne wth an nter-sensor separaton of 8 cm. Audo sgnals obtaned from the mcrophone arrays are processed to estmate the TDOA between dfferent mcrophone channels. These two modaltes are subsequently ntegrated by a speaker assocaton algorthm for the purpose of dentfyng where and who the current actve speaker s. The followng subsectons wll detal our system Trackng partcpants by Ccada Ccada s based on the TDOA between ultrasound and rado frequency (RF) to estmate the dstance, and adopts Unscented Kalman Flter (UKF) [9] to calculate the locaton. The framework of Ccada s shown n Fgure 1. Fgure 1. The wreless framework of Ccada Ccada conssts of four man parts: CBadge, CReader, CBase and locaton server. CBadges are small embedded hardware devces carred by partcpants, whch emt RF and ultrasound at the same tme perodcally. The RF sgnal s modulated to contan the ID nformaton of correspondng CBadge, whch s used to determne the dentty of the person who carres the CBadge. Several CReaders are fxed on the celng of the room. The coordnaton of each CReader s measured and stored beforehand. Because the propagaton speed of RF (the speed of lght n ar) and that of ultrasound (the speed of sound, approxmately 340 m/s n ar) are dfferent, then the TDOA between RF and ultrasound can be used to calculate the dstance between a CBadge and a CReader. Once a CReader receves the RF and ultrasound sgnal from a certan CBadge, t multples TDOA by the sound propagaton speed to estmate the dstance, neglectng the propagaton tme of RF. Then 21st Internatonal Conference on Advanced Informaton Networkng and Applcatons Workshops (AINAW'07)

3 the CReader uses RF to send the dstance nformaton to the CBase n a specal format, whch s shown n Fgure 2. The dstance nformaton s used to calculate the locaton, whle the TDOA nformaton s redundant for verfcaton purpose. The tme stamp reveals the tme when the CReader receves the RF sgnal. Fgure 3. An eght-element mcrophone array Fgure 2. Data pack sent by a CReader to the CBase. The locaton server collects the data packs from the CBase through a seral port, and then UKF s used to work out the locaton. We assume that the partcpants n the room move n a constant rate, so the system can be expressed as T X = [ x y z dx dy dz ] (1) X = 1 AX + + w (2) Z = (x x ) + (y y ) + (z z ) + v (3) r r r where (x,y,z) s the locaton of the CBadge, whle (x r,y r,z r ) s the poston of the CReader and Z s the dstance fgured out by the CReader. The random varables w and v represent the process and measurement nose respectvely. When the locaton server receves data from three or more CReaders, t can work out the locaton of the CBadge. The average localzaton error of Ccada s about 5 cm. Wth the small badges, each of whch s about 8.8*3.0*4.3 cm n sze, the partcpants n the meetng room can be tracked precsely. And more mportantly, every partcpant wll have a dstnctve ID to determne hs/her dentty Capturng audo feature by mcrophone array Our system uses four mcrophone arrays to capture the audo feature of the speaker. As shown n Fgure 4, four mcrophone arrays are placed on dfferent walls of the room at the heght of 150 cm. The X, Y plane of mcrophone array coordnate system s parallel to the ground. Snce the arrays are placed at approxmately mouth level of the users, we can make the smplfyng assumpton that sound sources are on the mcrophone array plane. Fgure 4. Four arrays fxed on dfferent walls The audo feature s based on estmates of the TDOA between dfferent mcrophone channels n the array. Several practcal TDOA estmaton algorthms have been explored n [6] and [10], and we choose to use the PHAT-GCC (Phase Transform Generalzed Cross-Correlaton) method, whch s relatvely robust n the room envronment. Consder two of the mcrophones m and m n an array, and let x and x respectvely be the sgnal recorded at correspondng mcrophone. Then x and x can be modeled respectvely as x (t) = h(t) s(t) + n(t) (4) x (t) = h (t) s(t ) + n (t) (5) where s(t) denotes the source sgnal, h(t) s the acoustc mpulse response between the sound source and the mcrophone, n(t) s the respectve addtve nose (assumed to be zero mean and uncorrelated wth the source sgnal), and presents the relatve delay of the sgnal. Then the PHAT-GCC functon can be expressed as 1 + X( ω)x ( ω) ω (6) R( ) = e dω 2π X( ω)x ( ω) where X ( ω ) s the Fourer transforms of x (t). The relatve tme delay s estmated as the tme lag wth the global maxmum peak n the PHAT-GCC functon ˆ = arg max R ( ) (7) For any par of mcrophones, the TDOA can determne a hyperbolod where the sound source s 21st Internatonal Conference on Advanced Informaton Networkng and Applcatons Workshops (AINAW'07)

4 located. Theoretcally, by groupng mcrophones nto pars and estmatng the TDOA of each par, the source locaton can be found at the pont where all assocated hyperbolods ntersect. Normally, the sound s assumed to be n the far feld of the array, and then a hyperbolod can be smplfed to a cone, whch ndcates the DOA (Drecton of Arrval) of the source. However, the smplfcaton can ntroduce sgnfcant estmaton error when the aperture of the array s large or when the source s relatvely near the array. Snce we can acqure precse locaton nformaton of all the partcpants n the room from Ccada system, mcrophone arrays do not need to work out the speaker locaton drectly. Instead we can smply use TDOA to assocate the sound source wth one of the partcpants. In ths way, we can avod the estmaton error ntroduced by smplfcaton, and mprove the performance of the system Detectng speaker by ntegraton Ccada and mcrophone arrays are ntegrated by a fuson algorthm to assocate the sound source wth one of the partcpants n order to determne who the current speaker s. We assume that each of the partcpants n the meetng room carres a CBadge provded by Ccada system, so every partcpant wll have a dstnctve ID and can be precsely tracked. For each partcpant tracked, we obtan a 3D poston, X, whch represents the locaton (x,y,z) of the CBadge n Ccada coordnate system. Snce Ccada and mcrophone array coordnate system are algned, (x,y) s approxmately the partcpant s mouth poston n mcrophone array plane. Let s denote the two dmensonal poston of a partcpant n the mcrophone array plane, and let m 1 and m 2 respectvely be the poston of the mcrophones n par. If the partcpant s speaks, then the TDOA between the par of mcrophones can be calculated as s m1 s m2 (8) (s ) = (m 1,m 2,s ) = c where c s the sound propagaton speed. We set up the assocated statstcal model as follow p( ˆ s ˆ ) = Ν( ; ( s ), f ( σ )) (9) where Ν (; µ, σ ) s a Gaussan densty wth mean µ and standard devaton σ, and ˆ ndcates the actual TDOA estmated by mcrophone par. σ s the standard devaton obtaned from a tranng TDOA dataset of a speaker at a known locaton, whle f( σ ) s a lnear functon of σ. We can add up the probablty densty values from all m pars of mcrophones for the partcpant s, and normalze by m to get a better estmate m 1 p( ˆ Τ s ˆ ) = p( (10) s ) m = 1 In our system, the pror probablty of each partcpant to speak s set to be equal. That s 1/n f the number of partcpants s n. Then accordng to Bayes formula, the posteror probablty can be expressed as ˆ ˆ p( Τ s )P( s ) (11) P( s Τ ) = n p( ˆ s )P( s ) = 1 Τ The partcpant wth the hghest posteror probablty s selected to be the speaker. The pror probablty can also be set to be unequal accordng to the applcaton scenaro. For example, n a classroom, the teacher s the person that s normally speakng. Therefore, hgher pror probablty value can be assgned to the teacher. We suppose that there s no smultaneous speech n the room, so each speech segment belongs to one specfc partcpant. Then, the system needs to assocate the sound source wth a certan partcpant only at the begnnng of each speech segment. In order to detect the speech actvty, we adopt a smple and effcent speech detector, constructed by calculatng the sgnal zero-crossng rate. A speech detector s mplemented on each mcrophone channel respectvely. If more than half of all the channels detect speech actvty, the system wll start the speaker assocaton process. 4. Experments and evaluaton In order to evaluate the performance of the descrbed system, two experments were carred out n a typcal meetng room measurng cm. As shown n Fgure 5, sx Ccada CReaders are fxed on the celng of the room, and four mcrophone arrays are placed on each wall respectvely. All the mcrophones were recorded synchronously at a sample rate of 32 khz and a resoluton of 16 bts. In the experments, all the mcrophone channels are used to detect the speech actvty, whle only one par of mcrophones n each array s used to estmate the TDOA. Both the speech actvty detecton and TDOA estmate are computed over 64ms wndows to enable the system to detect rapd speaker changes. There are 25 predefned ponts (from 1 to 25, depcted at Fgure 5) labeled on the ground for speaker assocaton test. The dstance between two neghborng ponts s 60 cm. In the followng experments, the partcpants roamed n 21st Internatonal Conference on Advanced Informaton Networkng and Applcatons Workshops (AINAW'07)

5 the room and stopped at the test pont to utter a few sentences n Chnese or have conversatons wth the other partcpants Experment n two-partcpant scenaro In ths experment, two partcpants were walkng n the meetng room and conversng wth each other. Ther conversaton was restrcted to a turn-takng dalog and there was no smultaneous speech. Each partcpant carred a Ccada CBadge for localzaton and dentfcaton. The tracks of the partcpants were predefned. One of them (P1) walked along the red sold lne drawn on the ground, shown n Fgure 5, and the other (P2) walked along the cyan dashed lned. The experment was carred out n two steps. In the frst step, P2 stood stll at pont 22; meanwhle P1 walked from pont 2 to pont 24 along the red lne. At each labeled pont, P1 asked a queston and P2 responded. In the second step, P1 stood stll at pont 24 and P2 walked from pont 22 to pont 4. The partcpant count wll affect the performance of speaker assocaton algorthm, snce the possblty of selectng a wrong speaker wll ncrease f more partcpants stay n the room. We carred out an experment n a 25-partcpant scenaro to demonstrate the effectveness of the speaker assocaton algorthm. However, Ccada can not support so many partcpants smultaneously n the current system; hence n the experment, 24 partcpants were smulated by software. In each of our expermental sequences, one real partcpant was standng on one of the 25 labeled ponts, where he uttered forty sentences n Chnese, whle the smulated partcpants were set to be standng on the remanng ponts. Accuracy Speaker Poston Fgure 6. Speaker assocaton accuracy Fgure 5. Sensors set-up n the experment envronment The results show that n the two-partcpant scenaro the speaker assocaton algorthm gves approxmately 100% accuracy rate. The average error of the speaker locaton estmaton s about 15 cm, whch s relatvely hgher than the average localzaton error of Ccada (5 cm, descrbed n the secton 3.1). The explanaton to ths phenomenon s that the Ccada CBadge moves along wth the gesture or movement of the partcpant, so the actual poston of CBadge can not be acqured. We obtan the localzaton error smply by comparng the locaton nformaton provded by Ccada wth the coordnate poston of the labeled ponts, where the partcpant stands. Therefore, the observed error augment s due to the movement of CBadge Experment n 25-partcpant scenaro wth smulaton The accuracy rate of speaker assocaton s shown n Fgure 6. These results show that n the 25-partcpant scenaro, the speaker assocaton algorthm can acheve approxmately 90% average accuracy. Comparatvely, the speaker assocaton accuracy s about 79.5% n [4] and 89.01% n [2]. Meanwhle, the partcpant counts, four n [4] and three n [2], are smaller than that n our experment. Furthermore, [4] can not determne the dentty of the speaker. In a word, ths system provdes mproved performance n the accuracy of speaker localzaton and dentfcaton. 5. Concluson In ths paper, we present a novel multmodal system for speaker localzaton and dentfcaton. Every partcpant wll have a dstnctve ID and can be tracked precsely n our system. Moreover, the speaker can be dstngushed from the slent partcpants correctly by mcrophone arrays. Future work wll contnue n two aspects. Frstly, the MAC mechansm of Ccada can be mproved to support more partcpants smultaneously. Then we can test our speaker localzaton and dentfcaton algorthm n a scenaro that contans more partcpants 21st Internatonal Conference on Advanced Informaton Networkng and Applcatons Workshops (AINAW'07)

6 wthout usng smulaton. Secondly, more sophstcated ntegraton strategy can be examned. In ths study, we smply set equal pror speakng probablty of each partcpant. In future, we can utlze a speaker model to adust the pror probablty of each partcpant durng the meetng. Therefore, the system can adapt to dfferent scenaros. 6. Acknowledgement Ths research s supported by NCET (Program for New Century Excellent Talents n Unversty, Chna). 7. References [1] Gopal Pngal, Gamze Tunal, Ingrd Carlbom, Audo- Vsual Trackng for Natural Interactvty, Proceedngs of the 7th ACM nternatonal conference on multmeda, Orlando USA, 1999, pp [2] Mchael Sracusa, Lous-Phlppe Morency, Kevn Wlson, John Fsher, Trevor Darrell, A Mult-Modal Approach for Determnng Speaker Locaton and Focus, Proceedngs of the 5th nternatonal conference on Multmodal nterfaces, Vancouver Canada, 2003, pp [3] Ka Nckel, Tobas Gehrg, Raner Stefelhagen, John McDonough, A Jont Partcle Flter for Audo-vsual Speaker Trackng, Proceedngs of the 7th nternatonal conference on Multmodal nterfaces, Trento Italy, 2005, pp [4] Carlos Busso, Serg Hernaz, Ch-We Chu, Soon-l Kwon, Sung Lee, Panayots G. Georgou, Isaac Cohen, Shrkanth Narayanan, Smart Room: Partcpant and Speaker Localzaton and Identfcaton, Proceedngs of the 30th nternatonal Conference on Acoustcs, Speech, and Sgnal Processng, Vol. 2, Phladelpha USA, 2005, pp [5] Honglang Gu, Yuanchun Sh, Yu Chen, Bbo Wang, and Wenfeng Jang, Ccada: a hghly-precse, easy-embedded and omn-drectonal ndoor locaton sensng system, Proceedngs of the 1st nternatonal Conference on Grd and Pervasve Computng, Tawan Chna, 2006, pp [6] Charles H. Knapp, G. Clfford Carter, The Generalzed Correlaton Method for Estmaton of Tme Delay, IEEE Transacton on Acoustcs, Speech, and Sgnal Processng, Vol. 4, No. 4, 1976, pp [7] Tanzeem Choudhury, Bran Clarkson, TonyJebara, Alex Pentland, Multmodal person recognton usng nconstraned audo and vdeo, Proceedngs of the 2nd Internatonal Conference on Audo-Vdeo Based Person Authentcaton, Washngton D.C. USA, 1999, pp [8] Matthas Wolfel, Ka Nckel, John McDonough, Mcrophone Array Drven Speech Recognton: Influence of Localzaton on the Word Error Rate, Proceedngs of the 2nd Jont Workshop on Multmodal Interacton and Related Machne Learnng Algorthms, Ednburgh UK, 2005, pp [9] Greg Welch, Gary Bshop, An Introducton to the Kalman Flter, Unversty of North Carolna at Chapel Hll Department of Computer Scence, [10] Mchael S. Brandsten, Harvey F. Slverman, A practcal methodology for speech source localzaton wth mcrophone arrays, Computer, Speech, and Language, Vol.11, No.2, 1997, pp st Internatonal Conference on Advanced Informaton Networkng and Applcatons Workshops (AINAW'07)