ALIGNING AUDIO AND VISUAL CUES WHEN PRESENTING FAST MOVING SOUND SOURCES WITHIN A MULTISENSO- RY VIRTUAL ENVIRONMENT

Transcription

1 ALIGNING AUDIO AND VISUAL CUES WHEN PRESENING FAS MOVING SOUND SOURCES WIHIN A MULISENSO- RY VIRUAL ENVIRONMEN Ian Drumm and John O Hare Universi of Salford, Compuer Science and Sofware Engineering, M54W, UK i.drumm@salford.ac.uk his paper will address challenges in aligning audio and visual cues when rendering fas moving objecs wihin a high end muli-sensor virual environmen facili which emplos 3D sereo visual projecion and wave field snhesis. he visual and audio ssems are linked via a nework connecion and updaes from he visual ssem occur a discree ime inervals. his paper will demonsrae and assess he use of moion predicion sraegies for he opimum updaing of dnamic audio scenes independenl of he consrains presened b he visual rendering ssem and nework communicaion. his work has proven paricularl useful for ecologicall valid simulaions of road raffic, rail and urban soundscapes.. Inroducion his paper perains o work which has faciliaed he mulimodal/mulisensor simulaion of fas moving road and rail raffic wihin Salford Universi s OCAVE, a sae of he ar virual environmen facili. Figure shows he OCAVE, an ocagonal space wih a maimum diameer of ~6m surrounded b sereographic wall and floor projecion conrolled b a dedicaed visual rendering ssem, and an ocagonal ring of 8 loudspeaker unis conrolled b a dedicaed acousic rendering ssem emploing wave field snhesis. he visual and audio ssems are linked via IP for aligning visual and audio cues. Acousic rendering over muliple loudspeakers for virual reali using wave field snhesis [] offers he poenial for rich dnamic soundscapes wih high immersion and presence. As here is no lisening swee spo several users can collaborae and naurall communicae wih each oher wihin he space whils eperiencing real and rendered sounds from heir own perspecives. Modeling moving virual sound sources wih wave filed snheses is relaivel sraigh forward and provides a warping of he ime ais which is percepuall similar o Doppler shifs. Imporan arifacs associaed he aural rendering of moving sources such as specral broadening and spaial aliasing [][3] will no be addressed in his paper, which will insead focus specificall on errors associaed wih conrolling he sound ssem from a visual rendering ssem.

2 he 3 rd Inernaional Congress on Sound and Vibraion Figure : he OCAVE Virual Environmen Facili consiss of 3D sereo projecion ono 8 screens and a floor. he ssem has inegraed a8 channel wave field snhesis ssem conrolled from he visual ssem via UDP or CP. Wave field ssem has found applicaion in mulimodal cones, especiall wih muli-viewer sereo displas and Cave Auomaic Virual Environmens [4][5] wih good alignmen and localizaion accurac for scenarios wih slow moving sources. However, because high end ssems require dedicaed hardware for visual and audio rendering and a mechanism for communicaion beween hese separae compuer ssems, each ssem will run independen applicaion hreads, rendering hreads and communicaion middleware. Source posiion updaes from he visual ssem are limied b he rae a which he visual ssem is able o send updaes and he rae a which he audio ssem is able o parse hem. Furhermore, wih so man independen hreads running and variaions in frame raes and nework raffic, ssemaic and sochasic errors in posiion wih respec o ime are inroduced. his will become increasingl imporan wih faser moving sources. o help characerize hese errors consider a moorwa simulaion in he OCAVE. he visual ssem uses 6 cores (3 hreads), 64G of memor, SSD, and 4 Nvidia K5 wih a ksnc card running foureen channels of DVI. his runs Uni and Open Scene Graph based applicaions wih Middle VR and in-house developed sofware componens for head racking, moion conrol and he communicaion of source posiion updaes via IPv6 o an 8 core dedicaed windows based audio ssem running in-house sofware wrien in C++ for rendering wih wave field snhesis. he audio is conrolled via wo 64 channel RME soundcards and associaed MADI conrolled digial o analog converers. he audio hread runs independenl of he visual ssem s applicaion hread. Ke processes are shown in figure. If he ssem runs a visuall rich virual environmen a 3fps and emulaes raffic moving a 7mph, he spaial resoluion of movemen will be As a virual source ransiions from ouside o inside he ring i ma appear as a focused source in onl a handful of posiions resuling in a ver noiceable and unconvincing audio jump as he speaker selecion funcion swiches beween opposie sides of he arra. his rendering paholog can be furher eacerbaed b an communicaion gliches and bole necks over a nework connecion. Conras his wih resoluion of an audio onl simulaion. For eample, in-house developed sofware using he Windows Audio Session Applicaion Programming Inerface (WASAPI) in eclusive mode audio can render muliple moving sources across a 8 channel sound ssem wih a refresh rae of ~.9ms or ~34 updaes per second, hus giving a more accepable spaial resoluion of movemen of Figure 3 shows how a 7mph virual source changes posiion wih respec o ime for 9 updaes re- ICSV3, Ahens (Greece), -4 Jul 6

3 he 3 rd Inernaional Congress on Sound and Vibraion ceived a a rae of around ~3 per second b and audio hread running a ~34 updaes per second. he posiion updaes and heir imes were sen from he visual ssem via UPD, hough CP/IP shows similar resuls. he saircase in he source posiion graph demonsraes how he audio source jumps o new posiions. he accompaning hisogram shows posiional errors wih respec o 5 updaes. he sandard deviaion is.9m. Figure : he visual ssem sends source posiion updaes o he audio ssem. Audio is conrolled via an audio hread. Figure 3. Source posiion and posiion error of a 7mph source conrolled a a rae of 3 updaes per second. Clearl a soluion is o le he audio server run updaes on he audio hread based on predicions of source posiions. B sending posiions and imings from he visual server and offseing hese imings wih he elapsed ime of he audio hread, predicions of posiions in beween updaes are possible. his paper will presen such predicion scheme wih objecive and subjecive assessmens of is implemenaion. he auhors will also eplore he addiion of a more sophisicaed probabilisic approach using a Kalman filer [6] o deal wih sochasic errors in posiion updaes.. heor and Implemenaion o erapolae source posiions on he audio hread consider a consan acceleraion moion predicion for he D posiion vecor and veloci v a ime for a moving objec given values a a previous ime denoed as. v a () ICSV3, Ahens (Greece), -4 Jul 6 3

4 he 3 rd Inernaional Congress on Sound and Vibraion v v a he prediced posiion can be epressed in mari form wih () Where A Ba (3) v v A / / B (4) Where a is a conrol variable (in his case acceleraion) and is he ime elapsed from he previous change in. In pracice can be given as he difference beween he elapsed ime when a new posiion updae is received and he elapsed ime for he curren ieraion on he audio hread. Figure 4: Source posiion and posiion error wih moion predicion for a 7mph source conrolled a a rae of 3 updaes per second. Figure 4 shows he effeciveness of his simple approach. he saircase of figure 3 is replaced b a ver smooh ransiion of source posiions beween updaes. his resuls in ver believable moving sources in a mulimodal cone, especiall for higher speeds and moion pahs near or hrough he wave field snhesis loudspeaker ring. he subjecive imporance will eplored laer. Noe a sochasic error sill eiss. his is effecivel noise in associaed wih he iming and posiion of updaes when received b he audio hread and is proporional o source speed, in his case giving a sandard deviaion of.64m for 7mph. o miigae for his error a Kalman filer is proposed and run b audio hread whenever a new posiion updae is received. his predics he likelihood of based on previous posiion updaes. his Kalman filer works in conjuncion wih he previous moion predicion scheme. 4 ICSV3, Ahens (Greece), -4 Jul 6

5 he 3 rd Inernaional Congress on Sound and Vibraion Kalman filers are founded on Baes s famous equaion o predic he poserior probabili of a hpohesis wih respec o daa P ( H D), given he prior probabili of he hpohesis P(H ) and he likelihood of he daa wih respec o he hpohesis P ( D H ) such ha P( H D) P( H ) P( D H ) P( D) (5) he poserior probabili is he made he prior probabili in a recursive calculaion for beer sae esimaion. Kalman filers include an acion sae wih he prior esimae and daa esimae. In implemening he Kalman filer for moving sources, consan acceleraion moion predicion was again used, his ime o predic he source posiion, which can be epressed wih Where E is a mari of covariances given b A Ba E E GG where G v v,,, and v v are he sandard deviaions of respecive posiions and velociies abou heir means. Also (6) v v a a A B he measuremen predicion can be given b where z C C and Ez m m is he square of he sandard deviaion of measuremen noise m, i.e. he epeced error in posiion. Hence an esimae based on his predicion is given b finding he covariance mari E z (7) And hence he Kalman gain P AP A E (8) K P C CP C E z (9) ICSV3, Ahens (Greece), -4 Jul 6 5

6 he 3 rd Inernaional Congress on Sound and Vibraion he mos likel posiion for he measuremen is hence given b he difference beween he prediced measuremen z and he acual measuremen z scaled b he Kalman gain. es K z z () Subsequenl he covariance mari is adjused for he ne updae P I K C P () As shown in figure 5, implemening a Kalman filer in he audio hread for source posiion updaes from he visual ssem resuls in a significan improvemen in posiional error, picall giving sandard deviaions ~.83m for 7mph. Figure 5: Source posiion and posiion error wih moion predicion and a Kalman filer for a 7mph source conrolled a a rae of 3 updaes per second. 3. Subjecive Evaluaion Boh simple moion predicion and Kaman filer based predicion implemened in he audio rendering sofware showed disinc improvemens in he perceived realism of dnamic acors wih colocaed audio wih visuals wihin virual environmens. For eample, for moving vehicles his was mos eviden for disances beween ~m o ~5m perpendicular o he direcion of movemen. o suppor his, a simple subjecive es was conduced where eigheen paricipans ook par in mulimodal eperimens wihin Salford Universi s OCAVE Virual Environmen suie wih inegraed 8 channel Wave Field Snhesis. Paricipans were asked o eperience virual cars passing b a ~7mph along a virual roadside. hree posiions were esed: Posiion he paricipan was m from he roadside, Posiion he paricipan was m from he roadside, Posiion 3 he paricipan was 6m from he roadside. Each respecive paricipan wore sereo 3D shuer goggles wih head racking, so was able o look and lisen in an direcion. Using a able PC paricipans were able o freel oggle beween 6 ICSV3, Ahens (Greece), -4 Jul 6

7 he 3 rd Inernaional Congress on Sound and Vibraion hree differen coloured cars rendered and so rank each car wih respec o heir perceived realism. For his paper we onl required a qualiaive validaion of he audio rendering approaches in heir mulimodal cone, so a simple ranking based approach was chosen (e.g. Red Car s, Green Car nd and Silver car 3 rd ). Beween each eperimen car colours were changed randoml. Unknown o he paricipans each car colour had associaed wih i a differen audio rendering echnique, which we ll denoe as follows N: No predicion mehod. he visual rendering ssem simpl sends posiion updaes o he audio ssem. : Simple predicion mehod. he visual rendering ssem sends posiion updaes o he audio ssem, he audio ssem predics he subsequen posiions of sound sources beween updaes from he visual ssem. K: A Kalman based predicion mehod. he audio rendering ssem includes a Kalman filer o predic he subsequen posiions of sound sources beween updaes from he visual ssem. Figure 6 shows he mean scores of paricipan choices wih sandard error wih respec o he rendering echnique and posiions, where s ranking was recorded as, nd as and 3 rd as 3. Figure 6. he mean scores of paricipan rankings wih sandard error for posiion a m from he roadside, posiion a meer from he roadside and posiion 3 a 6m from he roadside. K represens Kalman, N no predicion and simple predicion of source posiions b he audio rendering ssem. An un-weighed rank aggregaion [7] was emploed o deermine overall order of preference for respecive posiions. his saisical approach reas rank aggregaion as an opimizaion problem, using brue force o minimise he disance (Spearman foorule disance) of a super lis beween he individual ordered liss given b respecive paricipans. Hence, an overall ranking and score are given (smaller scores correspond o less variaion beween individual resuls). Aggregae rankings in order of s, nd and 3rd were posiion : K,, N score.5, posiion : K,, N score.78 and posiion 3:, K, N score.. ICSV3, Ahens (Greece), -4 Jul 6 7

8 he 3 rd Inernaional Congress on Sound and Vibraion his shows he Kalman () and Simple Predicion () approaches o rendering audio in a mulimodal cone were ranked higher for realism, hough wih greaer uncerain for posiions ver near and far awa from he moving sources. Alhough in he eperimen fied lisener posiions were used, i should be noed ha when he lisener was able o move freel in he scene he improvemens gained from he moion predicion schemes were sill ver noiceable. his has led o furher work wih moorwa raffic / road crossing simulaions wih high ecological validi and presence. 4. Conclusions he resuls show he usefulness of including sources posiion predicion when rendering sound sources wih associaed visuals. he compuaional load of consan acceleraion moion predicion on he sound rendering hread is low, for eample calculaing moion predicion for an audio source is picall.8% of he duraion of an updae on he audio hread. Kalman filering of updaes from he visual ssem is also inepensive, for eample aking.39% of he ime beween updaes. For rich dnamic soundscapes wih muliple moving sources, some compuaional headroom can be gained b auomaicall choosing when o use one of he predicion algorihms for a given source. For eample, when sources are slow moving, far awa or moving wih a moion in line wih he view of he lisener, posiional errors are less percepible. In conclusion source posiion predicion a he audio hread for moving sources such as road raffic will resul in a ver noiceable gain in perceived realism and subsequenl an inferred higher immersion and presence o be assessed in laer work. REFERENCES Figure 7: Moorwa raffic simulaion in Ocave mulimodal research laboraor.. Berkhou A. J., A holographic approach o acousic conrol, Journal of he Audio Engineering Socie, 36() , December (988).. Franck A., e al., Reproducion of moving sound sources b wave field snhesis: An analsis of arifacs, Audio Engineering Socie Conference: 3nd Inernaional Conference: DSP For Loudspeakers. Audio Engineering Socie, (7). 3. Ahrens J. and Spors S., Reproducion of moving virual sound sources wih special aenion o he doppler effec, Audio Engineering Socie Convenion 4. Audio Engineering Socie, (8). 4. Springer J., e al., Combining wave field snhesis and muli-viewer sereo displas, Virual Reali Conference, IEEE, (6). 5. DeFani., e al., he SarCAVE, a hird-generaion CAVE and virual reali OpIPoral. Fuure Generaion Compuer Ssems , (9). 6. Kalman R A new approach o linear filering and predicion problems, Journal of Basic Engineering , (96). 7. Pihur V., e al., RankAggreg, an R package for weighed rank aggregaion, Deparmen of Bioinformaics and Biosaisics, Universi of Louisville, hp://vpihur.com/biosa, Augus 5, (4). 8 ICSV3, Ahens (Greece), -4 Jul 6