Gestures Everywhere: A Multimodal Sensor Fusion and Analysis Framework for Pervasive Displays

Transcription

1 Gesures Everywhere: A Mulimodal Sensor Fusion and Analysis Framework for Pervasive Displays Nicholas Gillian 1, Sara Pfenninger 2, Spencer Russell 1, and Joseph A. Paradiso 1 {ngillian, saras, sfr, joep}@media.mi.edu 1 Responsive Environmens Group Massachuses Insiue of Technology Media Lab, Cambridge, MA, USA 2 Wearable Compuing Laboraory, ETH Zurich, Swizerland ABSTRACT Gesures Everywhere is a dynamic framework for mulimodal sensor fusion, pervasive analyics and gesure recogniion. Our framework aggregaes he real-ime daa from approximaely 100 sensors ha include RFID readers, deph cameras and RGB cameras disribued across 30 ineracive displays ha are locaed in key public areas of he MIT Media Lab. Gesures Everywhere fuses he mulimodal sensor daa using radial basis funcion paricle filers and performs real-ime analysis on he aggregaed daa. This includes key spaio-emporal properies such as presence, locaion and ideniy; in addiion o higher-level analysis including social clusering and gesure recogniion. We describe he algorihms and archiecure of our sysem and discuss he lessons learned from he sysems deploymen. 1. INTRODUCTION Modern digial displays are now equipped wih a myriad of sensors o deec a user s acions. These sensors include cameras [16], capaciive ouchscreens [3], infrared proximiy sensors and beyond [13]. As hese displays become pervasive hroughou our workplaces, reail venues and oher public spaces, hey will become an invaluable par of he rapidly growing sensor neworks ha observe every momen of our lives. While hese sensor neworks are becoming ubiquious, hey are ofen deployed as independen closed-loop sysems, designed using a single echnology for a single applicaion [12]. Consequenially, muliple sysems frequenly observe he same evens, bu fail o accuraely deec hese evens because of occlusions, sensor noise or gaps in sensor coverage. To ruly ake advanage of hese pervasive neworks, we need dynamic frameworks ha suppor he heerogeneous sensor daa from nearby devices o be shared and fused. This aggregaed daa can hen be used o make beer inferences of higher-level informaion, such as a user s precise locaion or he classificaion of a gesure, which can hen be disribued back o nearby pervasive applicaions hrough reusable sofware absracions. By developing frameworks Permission o make digial or hard copies of all or par of his work for personal or classroom use is graned wihou fee provided ha copies are no made or disribued for profi or commercial advanage and ha copies bear his noice and he full ciaion on he firs page. Copyrighs for componens of his work owned by ohers han he auhor(s) mus be honored. Absracing wih credi is permied. To copy oherwise, or republish, o pos on servers or o redisribue o liss, requires prior specific permission and/or a fee. Reques permissions from Permissions@acm.org. Perviasive Displays 2014 Copenhagen, Denmark Copyrigh is held by he owner/auhor(s). Publicaion righs licensed o ACM. ACM /14/06...$ hp://dx.doi.org/ / for sensor fusion and he absrac disseminaion of analyical daa, we can maximize he heerogeneous sensor neworks disribued hroughou our environmens, and provide beer inference on he evens wihin range of hese sensors; ulimaely faciliaing improved user experiences on boh ineracive displays and devices wihin heir viciniy. In his paper, we describe Gesures Everywhere, a real-ime sysem for mulimodal sensor fusion, pervasive analyics and gesure recogniion. Gesures Everywhere (GE) is disribued across 30 digial displays locaed in key public areas of he MIT Media Lab and aggregaes he real-ime daa from approximaely 100 sensors ha include RFID readers, deph cameras and RGB cameras. Our sysem fuses his mulimodal sensor daa from disparae sensors across differen locaions using radial basis funcion (RBF) paricle filers and performs real-ime analysis on he aggregaed daa. This analysis includes key spaio-emporal properies, such as presence, proximiy, locaion and ideniy; in addiion o higher-level analyics, such as social clusering and gesure recogniion. This daa is hen made accessible o provide pervasive analyics and gesure recogniion suppor o he ineracive displays and oher ubiquious devices ha run hroughou he building. The main conribuions of our work include: (i) a flexible framework for mulimodal sensor fusion using RBF paricle filers; (ii) a modular infrasrucure for he absrac disseminaion of fused analyical daa; (iii) he inegraion of hese frameworks ino a robus, scaleable, cusomizable sysem ha suppors ineracive displays and pervasive devices wihin heir viciniy. Firs, we provide a general overview of our sysem followed by a review of prior sensor fusion work and a descripion of he RBF paricle filer. In Secion 4, we ouline how he RBF paricle filer is applied o esimae spaio-emporal merics, such as presence and locaion. Secion 5 presens how his fused daa is hen applied o make beer inferences of higher-level informaion, such as gesure recogniion. Finally, Secion 6 oulines wha we have learned from he developmen of his sysem and Secion 7 concludes. 2. SYSTEM ARCHITECTURE Gesures Everywhere is buil on op of an exising ineraciveinformaion sysem [3], which has been running hroughou he Media Lab since early This sysem, referred o as he Glass Infrasrucure, consiss of over 30 digialinformaion displays disribued a key locaions hroughou he building. Each 46-inch Samsung display feaures an inegraed capaciive ouch screen and has been exended wih

2 Figure 1: From lef o righ: (i) wo pervasive displays locaed in a public corridor a he MIT Media Lab; (ii) wo users deeced in fron of a pervasive display by he RGB-D clien applicaion, showing each user s cener of mass and bounding box; (iii) an overview of he Gesures Everywhere archiecure; (iv) he ineracive map applicaion visualizing he real-ime locaion of hree anonymized users. a ThingMagic ulrahigh-frequency (UHF) RFID reader and cusom sensor node [13] conaining microphones, moion, emperaure and humidiy sensors. An Apple Mac mini, embedded behind each display, is used o receive evens from he capaciive ouch screen and o run an RFID-reader polling applicaion wrien in Pyhon. The Apple Mac mini also runs each display s user inerface, buil wih HTML, CSS and JavaScrip, running in fullscreen-porrai mode inside a WebKi-based browser. The displays provide coninuous place-based informaion abou he lab s research in addiion o ineracive building floor plans, even posers and even-cenric applicaions. Building on he Glass Infrasrucure, we added 25 Microsof Kinecs o key displays locaed hroughou he Lab. A cusom-buil C++ sofware applicaion, running on he Apple Mac mini driving he associaed display, manages each Kinec. This sofware applicaion (RGB-D clien) capures he deph and RGB images from he Kinec a 30 frames per second, processing each image locally using a suie of compuer vision and machine-learning algorihms. These algorihms exrac feaures for presence esimaion and people deecion from he RGB images, in addiion o segmening a user and esimaing heir overlapping skeleon model from he deph daa1. The RGB-D clien sreams hese feaures o a cenral server via he Open Sound Conrol (OSC) nework proocol [21]. This includes basic saisics such as he number of users currenly visible a a display, o more complex feaures such as he hree-dimensional posiion of each deeced user s cenre of mass (see Figure 1), he color hisogram values of a specific user, or he esimaed locaion of he 24 joins of a user s body. A he core of he GE archiecure, a cenral server receives he real-ime deph and image feaure daa from he RGB-D cliens. This is in addiion o he real-ime daa from he nework of RFID readers locaed hroughou he building. To maximize his heerogeneous daa, he GE server fuses 1 We use he OpenNI and NITE libraries for skeleon fiing, however all oher merics are esimaed direcly from he deph images so we can deec and rack a user from he insan hey ener he deph field. he daa using a a series of hierarchical analysis modules. Firs, low-level analysis modules use RBF paricle filers o fuse he mulimodal daa from nearby sensors. Each lowlevel analysis module compues a specific spaio-emporal propery, such as presence, proximiy, locaion or ideniy. The oupu of hese lower-level modules are hen fed ino higher-level modules, which perform more complex analysis such as social clusering or gesure recogniion. Togeher, hese modules deermine wheher, which, and how users are ineracing wih a paricular display and are used by a hos of applicaions running on he GE infrasrucure (see Secion 6). To make boh he fused daa and analysis resuls accessible o oher displays and devices, he GE sysem provides applicaion programming inerfaces (API), implemened in boh OSC and HTTP/JSON2. These APIs ac as an absracion layer ha suppors pervasive applicaions o query, for insance, he curren locaion of a specific user or he number of individuals wihin he proximiy of a display. Once queried, he sysem will reurn he bes esimaion possible (using he fused sensor daa from disparae sensors a ha locaion); paired wih a probabilisic represenaion of he sysem s confidence for ha esimaion. This enables clien applicaions o respond accordingly, based on he sysem s esimaion confidence. 3. SENSOR FUSION In his secion, we describe he RBF paricle filer algorihm used for sensor fusion hroughou he GE sysem. Firs, we describe oher relaed work in sensor fusion. 3.1 Relaed Work Sensor fusion has received exensive research across a diverse range of fields in compuer science and engineering, including aerospace [1]; roboics [18]; people racking [17, 5]; and pervasive compuing [6]. Bayesian-based echniques, such as Kalman filers [4], and paricle filers [6, 18], are paricularly useful as hey provide a powerful saisical ool o help manage measuremen uncerainy and perform mulisensor fusion. Paricle filers main advanages over oher 2 You can view he real-ime oupu of he GE sysem via he Gesures Everywhere websie: hp://ge.media.mi.edu

3 Figure 2: An illusraion of fusing he oupu of hree sensors using a Gaussian RBF. The lef-hand graph uses equal weighs for each sensor, however he righ-hand graph places more weigh on he firs and second sensors. Noe how changing hese weighs impacs he resuling likelihood funcion. The disribuion of paricles are shown beneah each graph for = 0, 1, 2. Paricles a ime = 0 are randomly disribued. The size of each paricle reflecs ha paricles weigh, wih larger paricles having a larger weigh. Bayesian echniques are heir abiliy o represen arbirary probabiliy densiies and ha hey can incorporae virually any sensor ype [6]. A common approach o sensor fusion using a paricle filer is o model he sensor fusion process as he produc of independen condiional probabiliies across muliple sensors [11]. This approach has been applied widely o rack and idenify users by fusing inerial measuremen unis and WiFi [20]; vision and RFID daa [7]; or a nework of fixed cameras [2]. While his approach provides a convenien mehod o combine daa from muliple sensors, i does have one disadvanage in ha any sensor wih a condiional probabiliy approaching zero, will drive he produc of he remaining sensors owards zero. Consequenly, one damaged or noisy sensor could poenially suppress oher funcioning sensors which can make i difficul o implemen a robus model ha can be absraced across a large sensor nework. To miigae his problem, RBFs can be combined wih paricle filers, as his provides an efficien algorihm o combine he sensor daa from muliple heerogeneous sources, while remaining robus o unresponsive sensors. Radial basis funcions have been applied successfully in several conexs wih paricle filers, such as RBF Neural Neworks for sensor fusion [5] and arge racking [19], or using RBFs as an addiional sage for inerpolaing sparse paricles [14]. To provide a flexible framework for sensor fusion, we inegrae he RBF direcly ino he compuaion of he sensor s likelihood model as we now describe. 3.2 Radial Basis Funcion Paricle Filer Paricle filers are a se of on-line poserior densiy esimaion algorihms ha probabilisically esimae a dynamic sysem s sae from noisy observaions. The sysem s sae vecor could be he one-dimensional presence esimae of a digial display, or a more complex vecor including he posiion, pich, roll, yaw, and linear and roaional velociies of a moving objec. Paricle filers probabilisically esimae a sysem s sae, x, a ime using he sensor measuremen y. This is achieved using a se of N paricles, each wih heir own sae vecor and weigh {x (i), w (i) }: P (x y 1: ) 1 N N i=1 x (i) w (i) (1) For Gesures Everywhere, we apply he Sequenial Imporance Sample wih Resampling (SISR) algorihm o recursively updae he paricle filer a each ime sep [18]. SISR involves updaing each paricle s sae vecor using a predicion model: x (i) = f(x (i) 1 ); weighing each paricle by he sensor s likelihood model, given he curren sensor measuremen and prediced sae vecor: w (i) = g(y x (i) ); normalizing he paricle weighs, and resampling N new paricles for +1 using imporance sampling [18]. To faciliae he inegraion of sensor daa from several heerogeneous sources, we represen he sensor s likelihood model as a radial basis funcion: y g(y x) = α j φ(x, y j) (2) j=1 where α j is he weigh for he j h sensor and φ is he radial basis funcion. Figure 2 illusraes he resuling funcion from fusing hree sensors using a Gaussian RBF: φ(x, y) = 1 σ (x y) 2 2π e 2σ 2 (3) Approximaing he sensor fusion ask as an RBF paricle filer has several advanages. Firs, using an RBF provides an easy mehod o conrol how much we should rus each sensor, as his can be direcly encoded in α j, i.e. sensors ha provide a beer esimaion of he rue underlying sae will have a larger weigh. Weighs can be se using a-priori knowledge or direcly learned from he daa using weighed leas squares. Second, he RBF miigaes an erroneous sensor (wih a low weigh) from dominaing oher sensors ha are performing well. Furher, he RBF can approximae any mulimodal funcion, such as he funcions shown in Figure 2. As paricle filers can converge o he rue poserior even in non-gaussian, nonlinear dynamic sysems, RBFs provide an excellen complimen o he paricle filer. Finally, RBF paricle filers are paricularly suied for large-scale sensor neworks as he fusion algorihm can coninue o funcion even if several of he sensors in a nework fail, as he sensor fusion process simply ses φ( ) for ha sensor o zero.

4 4. ANALYSIS MODULES In his secion, we describe how he RBF paricle filers are applied by Gesures Everywhere o aggregae he mulimodal sensor daa and esimae he sysem s spaio-emporal properies. To address each of he spaio-emporal esimaions, we define individual paricle filers for each low-level propery. Depending on he esimaion ask, each paricle filer can represen a local esimae (e.g. he presence esimae a display k), or a global esimae (e.g. he locaion of a user wihin he enire building). Given space limiaions, we only describe he presence and locaion modules, however he proximiy and ideniy modules use he same RBF framework. 4.1 Presence Esimaion Presence informaion can play a key role in improving he design of digial displays. For insance, a sysem ha can sense he presence and approach of people can use ha informaion o reveal possible ineracions [15], even a a disance. The GE presence esimaion for display k is defined as: x = {p}, where p is a real number in he range [0 1] ha defines he likelihood of a user being presen a display k. The predicion model is se as: p (i) = p (i) 1 γ + N (0,σ) (4) where γ is empirically se as 0.99 and σ conrols how much each paricle randomly explores he sae space. To compue each paricle s weigh, feaures from he RFID reader (y 1), deph camera (y 2), and RGB camera (y 3) a display k are plugged ino he RBF o give: w (i) = 3 j=1 α j φ(p (i), y j) (5) For he RFID reader and deph camera, we direcly map he number of users deeced by he RFID reader and user segmenaion algorihm, scaling hese values o a uniform range. For he RGB camera, we exrac a movemen feaure, m, compued from he RGB image I a frame : [ 1 ] m = I (i, j) I 1(i, j) + m (6) Z i j where Z is a normalizaion consan designed o scale he movemen feaure o a uniform range, and i and j are pixel indices. Each of hese feaures independenly provide good indicaions of presence, however hey all have weaknesses. For example, he RFID and deph feaures have excellen precision and are robus o false posiives, however he deph camera only has a range of approximaely five meers and no every user carries an RFID card. Alernaively, he movemen feaure deecs presence if a user is moving, bu can fail o deec saionary users and is sensiive o false posiives errors. By applying sensor fusion, he srenghs and redundancies of each feaure can be combined o miigae each individual sensor s weakness. Table 4.2 shows how he individual accuracy, precision and recall of each of hese sensors can be improved using sensor fusion. 4.2 Locaion Esimaion Locaion informaion is ofen essenial for ubiquious compuing sysems [12] as i provides an imporan source of conex [20]. The locaion module esimaes he locaion and orienaion of each user wihin he viciniy of each display in he building. In addiion, he locaion module coninually racks each user for he duraion ha individual is wihin he viciniy of a display. A new paricle filers is used o rack each new user in range of display k, wih each paricle filer using 500 paricles. The paricle filer used o esimae each user s locaion is defined as follows. The sae vecor is x = {x, y, θ, c}, where {x, y} is he user s esimaed locaion, θ is he user s curren heading, and c is a 10 dimensional vecor ha represens he user s hue color hisogram. The locaion predicion model is defined as: θ (i) x (i) y (i) c (ij) = θ (i) 1 + ϕ (7) = x (i) 1 + β cos(θ(i) ) (8) = y (i) 1 + β sin(θ(i) ) (9) = c (ij) 1 + δ, 1 j c (10) where ϕ, β, and δ are Gaussian random variables drawn from N (0,σϕ), N (0,σβ ), and N (0,σδ ) respecively. σ ϕ, σ β, and σ δ represen he uncerainy ha has buil up in he user s x and y locaion, orienaion and color hisogram since he previous sample. The sensor s likelihood model is defined as: w (i) = α 1 φ(x (i), ˆx) + α 2 φ(y (i), ŷ) + α 3 φ(θ (i), ˆθ) + α 4 φ(c (i), ĉ) (11) where ˆx, ŷ, ˆθ and ĉ are he cener of mass (COM), orienaion and color models of one of he M user s currenly visible a display k. ˆθ is compued from a normal vecor projeced from he user s skeleon orso. Our user segmenaion and COM esimaion algorihms running on each RGB-D clien can robusly deec a new user from he momen hey ener he deph field, however, i akes approximaely wo seconds for he OpenNI/NITE skeleon model o auomaically be fied o each user. The ˆθ feaure is herefore no available for he iniial deecion of a new user. In his case, we ake advanage of he RBF model and se φ(θ (i), ˆθ) o zero unil new daa is received from he skeleon model. The raw esimae of he user s cener of mass already provides a robus esimae of he user s locaion, however fusing his wih he feaures from he skeleon model and hue color hisogram provide an esimae of he user s curren orienaion in addiion o robus racking of a user while hey are in he viciniy of a display. Furher, if several deph and RGB cameras overlap a he same locaion hen he raw sensor daa from each overlapping sensor can be fused o improve he overall locaion esimae. Tesing he oupu of he locaion module agains a one hour recording of hand-labeled daa recorded from six digial displays hroughou he Media Lab, he racking algorihm achieved an accuracy of 97% while racking 51 users in complex scenes wih several occlusions. I is imporan o noe ha he locaion module esimaes he locaion of each user wihin each display s local frame of reference. Consequenly, he locaion modules can efficienly rack a user wih as lile as 500 paricles, which is several orders of magniude less paricles required o accuraely rack a user hroughou an enire building [20]. Neverheless, as we know he locaion and orienaion of each display wihin he building, we can projec he user s locaion ino he building s coordinae sysem so oher racking sysems can ake advanage of he fused daa, such as hose described in [16]. While increasing

5 Presence Analysis Resuls Accuracy Precision Recall RGB Camera Feaure 80% Deph Feaure 80% RFID Feaure 79% RBF Paricle Filer 86% Table 1: The resuls from esing he oupu of he presence module agains one hour of hand-labeled daa ( sample poins) recorded across six digial displays locaed hroughou he MIT Media Lab. Noe ha he paricle filer achieves beer accuracy, precision and recall han any of he individual sensors wih jus 100 paricles. he number of paricles can improve he racking accuracy, we balance overall accuracy wih realime responsiveness as he sysem poenially needs o simulaneously rack several users a each display. 5. GESTURE RECOGNITION In addiion o he spaio-emporal properies described in secion 4, Gesures Everywhere also runs higher-level analysis modules, wih he mos prominen of hese being gesure recogniion. Gesure recogniion offers many advanages for large-scale public displays, as i enables users o sar ineracing wih he display a a disance or on he move. The gesure recogniion module deecs if any user a a display performs any of he gesures wihin he GE gesural dicionary. This consiss of a se of fundamenal gesural primiives ha oher ubiquious applicaions migh wan o recognize. Based on prior research [10], we oped o consrain he size of he gesural vocabulary o a small se of upper body gesures, each wih high posural dispariy from he oher gesures o ensure high reliabiliy and recogniion raes. This includes a se of core ineracions illusraed in Figure 3 ha include poining; nex/previous swipes; and a home gesure. These gesures are recognized using he Gesure Recogniion Toolki [8] wih a Naïve Bayes classifier [9]. The inpu o he classifier consiss of a welve-dimensional feaure vecor, wih he x, y, and z values of he user s lef and righ hands, ranslaed and roaed in relaion o he user s orso (making each gesure invarian o he locaion ha he gesure is performed), in addiion o he x, y, and z velociies of he user s lef and righ hands. To miigae false-posiive classificaion errors, a user s gesures are ignored if he overall likelihood of he skeleon join model is low, such as when he user is occluded by anoher individual, or is oo close o he deph camera. Clien applicaions can also acively reduce false-posiive recogniion errors by placing addiional conexual consrains on wha gesures are acioned. This can include limiing gesures o only be acioned if he individual is saionary, or using he oupu of he locaion module o filer any gesure evens from users ha are no orienaed owards a specific arge (such as one of he Glass Infrasrucure displays). 6. DISCUSSION Gesures Everywhere has been deployed hroughou he Media Lab since early In his secion, we describe some of he real-ime pervasive applicaions ha leverage he GE daa and discuss a number of key lessons ha have been learned hroughou he sysems deploymen. 6.1 Pervasive Applicaions Public digial displays can significanly benefi from pervasive analyics as applicaions can reac o wheher, which, and how users are ineracing wih a paricular display. The real-ime GE analyical daa are currenly being used o power a number of pervasive applicaions ha run on he digial displays locaed hroughou he Media Lab. For example, using he proximiy and ideniy daa we have buil an on-he-go updae applicaion ha displays ime-sensiive and locaion-sensiive informaion o regisered users as hey approach he screens. This includes a personal welcome screen o users as hey ener he building each morning, or imporan reminders ha can be riggered by boh a user s curren locaion or ime-of-day. This applicaion can also be used o propagae messages o specific users as hey move abou he building. As his applicaion may conain sensiive personal informaion, he on-he-go applicaion will only be shown if he user-idenificaion module reaches a specific confidence hreshold; which he user can conrol via heir personal daa-poral on he GE websie. This enables users o conrol he privacy seings of he sysem o bes mach heir individual needs. In addiion o on-he-go, we have inegraed he real-ime locaion daa ino he Glass Infrasrucure ineracive map. The ineracive map visualizes he locaion of anonymized users as hey walk hroughou he building and faciliaes users o direcly search for anoher individuals curren locaion wihin he map (see Figure 1). 6.2 Scaling & Sysem Modulariy Building flexible pervasive sysems ha can scale o an enire building can be challenging. Our sysem has currenly scaled o include approximaely 100 sensors disribued across a 100K sq. f. six-floor building, wih a daabase of 1800 idenifiable users. This is managed by one server which runs all he daa collecion, sensor fusion, analysis modules and API inerfaces. A core design of he GE archiecure ha has faciliaed he sysem o easily scale o an enire building is ha much of he processor-inensive compuer vision and machine learning algorihms are run locally on each of he devices managing he Glass Infrasrucure displays. The resul of his archiecure is ha he remoe RGB-D cliens send pre-compued feaures o he GE server. This significanly reduces he overall-processing overhead and amoun of real-ime daa being sreamed o he server. A second imporan design of he sysem is is absrac modulariy. Each of he sysem s componens, such as remoe sensor inerfaces, spaio-emporal analysis modules, gesure recogniion, ec., are separaed by robus sofware absracions. These absracions faciliae new sensors o easily be added o he sysem as hey are added o he environmen; popup displays o rapidly be inegraed ino he sysem for a one-day-even; or addiional analysis modules o be added ha build on he oupu of he exising modules. Furher, our sysem exposes he daa from all of hese modules via wo accessible APIs in HTTP/JSON and OSC. This enables muliple pervasive applicaions o access he raw, fused, and analyical daa; faciliaing he GE daa o be used as inpu o oher pervasive sysems ha may be added in he fuure o he building s digial ecosysem. By separaing he remoe sensor inerfaces from he pervasive analyics, and absracing he oupu of he sysem from he pervasive applicaions ha run on op of i, we provide a robus, flexible infrasrucure for dynamically connecing muliple sensing

6 Figure 3: The core se of gesures recognized by he gesure recogniion modules. The poining, swipe and ap gesures can all be performed by eiher he lef or righ hand. echnologies o muliple applicaions. 6.3 Privacy Privacy is clearly an imporan issue in he developmen of any pervasive sysem. To proec an individuals privacy, personally idenifiable daa is only released hrough he GE sysem if he user in quesion allows his. A cenral designphilosophy of he sysem is ha users have full conrol over heir own daa and can choose o share heir personal racking daa wih a specific group of individuals; users currenly inside he building; everyone; or no one a all. Users can easily conrol heir sharing seings a anyime via he Glass Infrasrucure displays or on heir personal devices via a web browser. In addiion o conrolling how oher users view an individuals daa, each regisered user has full access and ownership of heir own daa. Regisered users can login o he GE sysem via heir personal daa-poral and browse heir daa reposiory; faciliaing individuals o undersand jus how visible hey are o he sysem, who hey are sharing heir daa wih and monior who has been searching for hem. 7. CONCLUSION In his paper, we have presened Gesures Everywhere, a dynamic framework for mulimodal sensor fusion, pervasive analyics and gesure recogniion. We have described how our sysem applies radial basis funcion paricle filers o maximize he heerogeneous sensor daa deployed on a nework of digial displays, and demonsraed he inegraion of hese algorihms ino a scaleable sysem ha suppors a nework of ineracive displays and he pervasive devices wihin heir viciniy. 8. REFERENCES [1] J. K. Aggarwal and N. Nandhakumar. Sensor fusion and aerospace applicaions. In Sensor Fusion and Aerospace Applicaions, volume 1956, [2] K. Bernardin and R. Siefelhagen. Audio-visual muli-person racking and idenificaion for smar environmens. In Proceedings of he 15h inernaional conference on Mulimedia, pages ACM, [3] M. Blesas. The mi media lab s glass infrasrucure: An ineracive informaion sysem. Pervasive Compuing, IEEE, 11(2):46 49, [4] A. Brooks and S. Williams. Tracking people wih neworks of heerogeneous sensors. In Proceedings of he Ausralasian Conference on Roboics and Auomaion, [5] A. Douce, N. de Freias, K. Murphy, and S. Russell. Rao-blackwellised paricle filering for dynamic bayesian neworks. In Proceedings of he Sixeenh Conference on Uncerainy in Arificial Inelligence, UAI 00, pages , [6] D. Fox, D. Schulz, G. Borriello, J. Highower, and L. Liao. Bayesian filering for locaion esimaion. IEEE pervasive compuing, 2(3):24 33, [7] T. Germa, F. Lerasle, N. Ouadah, and V. Cadena. Vision and rfid daa fusion for racking people in crowds by a mobile robo. Compuer Vision and Image Undersanding, 114(6): , [8] N. Gillian. Gesure Recogniion Toolki. hp:// [9] N. Gillian, R. B. Knapp, and S. O Modhrain. An adapive classificaion algorihm for semioic musical gesures. In he 8h Sound and Music Compuing Conference, [10] K. Grace, R. Wasinger, C. Ackad, A. Collins, O. Dawson, R. Gluga, J. Kay, and M. Tomisch. Conveying ineraciviy a an ineracive public informaion display. In Proceedings of he 2nd ACM Inernaional Symposium on Pervasive Displays, pages ACM, [11] J. Highower and G. Borriello. Paricle filers for locaion esimaion in ubiquious compuing: A case sudy. In UbiComp 2004: Ubiquious Compuing, pages [12] J. Highower, B. Brumi, and G. Borriello. The locaion sack: A layered model for locaion in ubiquious compuing. In Mobile Compuing Sysems and Applicaions, Proceedings Fourh IEEE Workshop on, pages IEEE, [13] M. Laibowiz, N.-W. Gong, and J. A. Paradiso. Wearable sensing for dynamic managemen of dense ubiquious media. IEEE, [14] J. Madapura and B. Li. Muli-arge racking based on kld mixure paricle filer wih radial basis funcion suppor. In Acousics, Speech and Signal Processing, ICASSP IEEE Inernaional Conference on, pages [15] N. Marquard and S. Greenberg. Informing he design of proxemic ineracions. IEEE Pervasive Compuing, 11(2):14 23, [16] S. Pfenninger. A people racking sysem uilizing paricle filers in a non-overlapping sensor nework. Maser s hesis, Swiss Federal Insiue of Technology (ETH), [17] T. Teixeira, D. Jung, and A. Savvides. Tasking neworked ccv cameras and mobile phones o idenify and localize muliple people. In Proceedings of he 12h ACM inernaional conference on Ubiquious compuing, pages ACM, [18] S. Thrun, W. Burgard, and D. Fox. Probabilisic roboics. Inelligen roboics and auonomous agens, The MIT Press (Augus 2005), [19] X. Wang, S. Wang, and J.-J. Ma. An improved paricle filer for arge racking in sensor sysems. Sensors, 7(1): , [20] O. Woodman and R. Harle. Pedesrian localisaion for indoor environmens. In Proceedings of he 10h inernaional conference on Ubiquious compuing, pages ACM, [21] M. Wrigh. Open sound conrol: an enabling echnology for musical neworking. Organised Sound, 10(03): , 2005.