Appearance-Based Multimodal Human Tracking and Identification for Healthcare in the Digital Home

Transcription

1 Sensors 24, 4, ; doi:.339/s Aricle OPEN ACCESS sensors ISSN Appearance-Based Mulimodal Human Tracking and Idenificaion for Healhcare in he Digial Home Mau-Tsuen Yang * and Shen-Yen Huang Deparmen of Compuer Science & Informaion Engineering, Naional Dong-Hwa Universiy, No., Sec. 2, Da-Hsueh Rd., Shoufeng, Hualien 974, Taiwan; @ems.ndhu.edu.w * Auhor o whom correspondence should be addressed; myang@mail.ndhu.edu.w; Tel.: ; Fax: Received: 2 April 24; in revised form: 3 July 24 / Acceped: 8 July 24 / Published: 5 Augus 24 Absrac: There is an urgen need for inelligen home surveillance sysems o provide home securiy, monior healh condiions, and deec emergencies of family members. One of he fundamenal problems o realize he power of hese inelligen services is how o deec, rack, and idenify people a home. Compared o RFID ags ha need o be worn all he ime, vision-based sensors provide a naural and noninrusive soluion. Observing ha body appearance and body build, as well as face, provide valuable cues for human idenificaion, we model and record muli-view faces, full-body colors and shapes of family members in an appearance daabase by using wo Kinecs locaed a a home s enrance. Then he Kinecs and anoher se of color cameras insalled in oher pars of he house are used o deec, rack, and idenify people by maching he capured color images wih he regisered emplaes in he appearance daabase. People are deeced and racked by mulisensor fusion Kinecs and color cameras) using a Kalman filer ha can handle duplicae or parial measuremens. People are idenified by mulimodal fusion face, body appearance, and silhouee) using a rack-based majoriy voing. Moreover, he appearance-based human deecion, racking, and idenificaion modules can cooperae seamlessly and benefi from each oher. Experimenal resuls show he effeciveness of he human racking across muliple sensors and human idenificaion considering he informaion of muli-view faces, full-body clohes, and silhouees. The proposed home surveillance sysem can be applied o domesic applicaions in digial home securiy and inelligen healhcare. Keywords: home healhcare; human deecion; human racking; human idenificaion

2 Sensors 24, Inroducion Wih he advances in medical echnologies, he global populaion is aging, and he elderly are becoming he fases growing populaion secor in mos developed counries. In addiion o elders, oddlers and paiens are also a a higher risk of falling and require coninuous and long-erm monioring. There is an urgen need for an inelligen and inexpensive home surveillance sysem o provide home securiy, monior healh condiions, and deec emergencies of family members. To realize he power of hese inelligen services in digial homes, one of he fundamenal problems is how o deec, rack, and idenify people in a home environmen. Wearable RFIDs can idenify humans effecively, bu users are forced o wear ags all he ime. Alernaively, vision-based surveillance provides a naural and non-inrusive soluion o human deecion, racking, and idenificaion. Typical smar homes deploy a variey of visual sensors or cameras) locaed in and around house o monior human aciviies and deec criical evens. Each ype of visual sensors has is own unique srenghs and limiaions. For example, iris scanners are very accurae in human idenificaion bu only work on saionary people wihin a close range. Similarly, fingerprin scanners require people o show fingers from a very shor disance. Alernaively, a deph camera can resolve he problems of casing shadows and dynamic illuminaions bu is sensing range is limied, for example,.8 ~ 4 m for a Kinec []. In addiion, face recogniion algorihms provide passive sensing bu rely on high resoluion facial images. As a resul, ypical video surveillance sysems uilize omni-direcional OD) cameras o locae people and subsequenly guide pan-il-zoom PTZ) cameras o capure close-up facial images. However, clear facial images are unavailable when a person wears a mask or urns back o he camera inenionally. Table compares he pros and cons of various sensors in a home environmen. Table. Comparison of various sensors for human deecion, racking, and idenificaion a home. Hardware Requiremen User Inrusion Deecion under Various Poses Idenificaion Accuracy RFID-Based Technology RFID reader, ag Vision-Based Technology Fingerprin Iris Face Body Appearance Fingerprin scanner Iris scanner Close-up camera Camera Wear RFID ag Provide finger Show eye Turn o face camera Non-inrusive Yes No No Limied by angle Yes Absolue High Highes Medium-high Medium-high Dynamic Illuminaion Unaffeced Unaffeced Infrared beer Shadow, lighing Shadow, lighing Visual Occlusion Unaffeced No No Limied Parial Muli-View Tracking No accurae No No Limied by angle Yes Recenly, Rice e al. [2] revealed ha when facial feaures are difficul o make ou, we readily use body informaion o idenify a person. Our brains use informaion from a person s body size, shape, build, and sance for recogniion even before we can disincly see a face. Compared wih recogniion

3 Sensors 24, solely based on faces, hey discovered ha human recogniion is far more accurae when boh he face and body of he person are shown. In addiion o human recogniion, human deecion and racking can also uilize he body appearance o handle he challenging problems of muli-view variaions, posure changes, shape deformaions, far away views, and parial occlusions. Therefore, we propose appearance-based human deecion, racking, and idenificaion modules ha cooperae wih each oher seamlessly based on mulimodal fusion of muli-view faces, body colors, and silhouees capured by muliple sensors. Several echniques have been proposed o perform mulimodal fusion a various levels using differen mehods. Mulimodal inpus can be inegraed a hree levels: he signal low) level, he feaure inermediae) level, and he decision high) level. Fusion a a higher level offers scalabiliy and flexibiliy, bu loses signal or feaure) correlaion among modaliies. Fusion a a lower level offers less ineracion, bu provides more simpliciy because only one learning phase on he combined vecor is required. Mulimodal observaions can be combined for an esimaion using various mehods such as hisogram echnique, mulivariae Gaussian, linear weighed sum, Kalman filer, or paricle filer. Also, a decision can be made by mulimodal fusion using differen approaches such as majoriy voing, arificial neural nework ANN), suppor vecor machine SVM), or hidden Markov model HMM). These mulimodal fusion levels and mehods are usually applicaion specific and ailor-designed according o he naures and requiremens of he arge problem. The framework of he proposed human deecion, racking, and idenificaion sysem is shown in Figure. In he modelling sage, human skeleon and face racking are performed based on deph images capured by wo Kinecs insalled a he home enrance. Then he capured color images are uilized o capure faces, exrac body silhouees, and consruc a muli-view Flaened Cylindrical Templae FCT) in an appearance daabase. The FCT conains he appearance informaion of an individual person in an uprigh sanding posiion wih a view of 36. In he guidance sage, a person is deeced and racked based on color images capured by he Kinecs as well as oher color cameras insalled in oher pars of he house. The image of he deeced person is compared wih each regisered emplae in he appearance daabase. If a mach is found, he person is idenified and he corresponding emplae can be used o guide he subsequen human deecion and racking; oherwise he sysem promps for password o furher classify he person as a miss-idenified family member, a gues, or an inruder. The proposed human deecion, racking, and idenificaion sysem plays a key role in he applicaion of securiy and healhcare in inelligen digial homes. In his paper, we discuss he sysem design, developmen, and evaluaion of he proposed appearance-based mulimodal human deecion, racking, and idenificaion sysem. The remaining pars of he paper are organized as follows: Secion 2 reviews he relevan sae-of-he-ar echniques. Secion 3 presens he muli-view modelling of he full-body appearance using color and deph images capured by Kinecs and he consrucion of an appearance daabase. Secion 4 explains he human deecion and racking using a mulisensor fusion Kinecs and color cameras) based on a Kalman filer. Secion 5 presens he human idenificaion using a mulimodal fusion face, body appearance, and silhouee) based on a majoriy voing. Secion 6 evaluaes he proposed sysem. Secion 7 handles special cases and discusses he limiaions and poenial applicaions in smar homes. Secion 8 offers conclusions.

4 Sensors 24, Figure. Framework of he proposed appearance-based mulimodal human deecion, racking, and idenificaion sysem for inelligen healhcare and securiy in digial homes. Color & Deph Camera Deph Color Kinec Skeleon Tracking Kinec Face Tracking Foreground Silhouee Exracion Model Color Camera Muli-view Appearance Daabase wih Faces, Body clohes FCT), & Silhouees Home Nework Guide Acive Camera Color Vision-based Human Deecion Mulisensor Human Tracking Mulimodal Human Idenificaion Trigger Alarm & Person Following using Mobile Robo Parial occlusion & Seaed posure Handling 2. Background In recen years, echniques for human deecion, racking and idenificaion have progressed significanly. A RFID-based mehod employs eiher acive or passive ags. Acive RFID ags have a larger range bu require baeries o provide power. Passive RFID ags are less expensive bu only work over a shor disance. Similarly, infrared IR) or ulrasound ransmiers can be insalled in known posiions and each person can carry an IR or ulrasound receiver ha moniors signals in a range for localizaion. A person is locaed using a riangulaion mehod based on he disance and angular measuremens from a leas hree known locaions. Generally, he IR-based mehods are accurae bu can suffer inerference from background illuminaion. Relaively, he ulrasound-based approaches are cheaper bu less accurae. However, people do no feel comforable abou wearing a ag, ransmier, or receiver for a long ime. To reduce disurbance, passive visual cameras provide a non-inrusive way for human deecion, racking, and idenificaion. Sixsmih and Johnson [3] developed a smar inaciviy monior using an array-based deecor, called SIMBAD, for elderly fall deecions. Tao e al. [4] presened an infrared ceiling sensor nework wih binary responses o recognize eigh aciviies including walking, idying, waching TV, reading, aking drinks, using PC, lying, and sweeping in a home environmen. Ni e al. [5] designed a ge-up even deecor o preven poenial falls in hospials based on color and deph images capured by a Kinec. Moion and shape feaures from muliple modaliies and channels were exraced and combined hrough a muliple kernel learning process. Based on a mobile robo equipped wih a Kinec, Mozos e al. [6] used local binary paerns LBP) and SVM o caegorize

5 Sensors 24, indoor places including corridor, kichen, laboraory, sudy room, and office. Yang and Chuang [7] adoped a Kinec o classify behaviors and assess fall risks of oddlers a home. For human deecion using a ypical surveillance camera, Viola and Jones [8] proposed inegral images for fas feaure compuaions, an AdaBoos algorihm for auomaic selecions, and a cascade srucure for efficien human deecions. Dalal and Triggs [9] used hisogram of oriened gradiens HOG) as feaures and a linear suppor vecor machine SVM) as a classifier for pedesrian deecion and demonsraed promising accuracy. Zhu e al. [] combined inegral image, cascade srucure, and HOG for fas human deecion. Dollar e al. [] proposed a fas human deecor, called ChnFrs, by exracing and inegraing Harr-like feaures over muliple channels. They also developed a fas muli-scale varian of ChnFrs, called FPDW [2], by using a sparsely sampled image pyramid o approximae feaures a inermediae scales. Benenson e al. [3] modified FPDW o avoid resizing he inpu monocular images a muliple scales and provided human deecion a 5 frames per second FPS). By exploiing geomeric informaion using sixel esimaions from sereo images, hey achieved 35 FPS in a CPU + GPU enabled compuer. For vision-based human racking, Mean-Shif [4] is a non-parameric mehod o find he mode of a probabiliy disribuion funcion PDF). I can be applied o visual racking by creaing a PDF in he new frame based on a arge model, and performing an ieraive algorihm o find he peak of he PDF near he objec s las posiion. However, he Mean-Shif algorihm only worked well on arges wih saic PDFs. CamShif [5] exended he Mean-Shif o handle dynamic PDFs by updaing he arge model based on he color hisogram of he objec in he previous frame, and solve scaling problem by adjusing he search window size based on he updaed PDF. To model he appearance of boh objec and is background dynamically, Collins e al. [6] developed an online feaure selecion mechanism using a wo-class variance raio o discriminae beween a racked objec and is surrounding background. Babenko e al. [7] presened an online muliple insance learning algorihm, called MILTrack, by exracing posiive and negaive examples as an adapive appearance model for objec racking. Kalal e al. [8] proposed a racking-learning-deecion TLD) framework wih a pair of compuerized expers ha can learn from missed deecions and false alarms. Humans can be idenified visually by face recogniion if he deeced person faces he camera wihin a close range. A geomeric approach creaes a facial signaure by measuring disances beween key feaures o capure a unique facial profile for each face. Alernaively, a phoomeric approach analyzes he variance of faces over a high-dimensional vecor space o form a basis se of facial images. To perform dimension reducion, Eigenfaces [9] employed principal componen analysis PCA) while Fisherfaces [2] adoped linear discriminan analysis LDA). In addiion o human idenificaion, a relevan problem, called human re-idenificaion, is o idenify a specific person across disjoin camera views and o recognize if a person has been observed over a nework of cameras. I is a challenging problem due o changes in poins of view, background, illuminaion, pose deformaion, and visual occlusion. Gandhi and Trivedi [2] proposed a panoramic appearance map PAM) as a compac signaure o mach people observed in differen camera views. Prosser e al. [22] formulaed he re-idenificaion as a ranking problem and developed an Ensemble RankSVM. Generally, racking-based algorihms can generae a smooh rajecory of an objec by esimaing is moion, bu hey require iniializaion and can accumulae drif error during run-ime. On he oher hand, deecion-based algorihms can esimae he objec locaion in every frame independenly.

6 Sensors 24, However, a deecor requires an offline raining sage and canno deec unknown objecs. Unlike hese mehods, we proposed appearance-based human deecion, racking, and idenificaion modules which cooperaed seamlessly and benefied from each oher by using a mulimodal fusion of facial images, body colors, and silhouees across muliple sensors in home environmens. 3. Human Modelling of Face, Appearance and Silhouee Based on Kinecs The proposed sysem modelled muli-view human appearances faces, body colors, and silhouees) semi-auomaically using a se of he laes consumer marke deph cameras, called Kinecs. A Kinec capured boh color and deph images a hiry frames per second wih resoluion. The disance beween a hree-dimensional poin and he camera is called deph denoed as z). A pixel in a deph image indicaed he calibraed deph of he pixel s corresponding hree-dimensional poin in he scene. For he human skeleon racking, each deph image was segmened ino a dense probabilisic body par labeling so ha a human body was divided ino hiry-one pars [23]. The body pars were defined o be spaially localized near weny skeleal joins, hence he hree-dimensional locaions of he skeleal joins can be deermined by back-projecing hese inferred pars ino a world space. As shown in Figure 2a, a complee skeleon was represened by a sixy-dimensional vecor conaining hree-dimensional coordinaes of weny skeleal joins. In he modelling sage, he face, body appearance, and silhouee of each family member should be modelled and regisered in he appearance daabase. As shown in Figure 7, wo Kinecs were insalled a opposie sides in a living room o faciliae he modelling process. Figure 2. Human racking using a Kinec. a) Skeleon racking; b) Face racking. Yaw Roll Til a) b) 3.. Modelling of Face In addiion o he skeleon racking, he Kinec s face racking algorihm can deermine he locaion and hree-dimensional pose of a face in real-ime. Wih he availabiliy of wo head joins he head and he neck) in he racked skeleon, he rough locaion of he face in he capured images was deermined. The Kinec s face racker exended color-based Acive Appearance Model AAM) o incorporae deph informaion [24]. Based on boh color and deph images, he Kinec s face racker deeced 87 conour poins along facial pars as shown in Figure 2b) as well as addiional 3 non-conour poins including eye ceners, mouh corners, a nose cener, and a bounding box around he

7 Sensors 24, head. By regisering he racked facial feaures wih a hree-dimensional facial model, he head pose was esimaed and represened by hree roaional angles: il, yaw, and roll as shown in Figure 2b). In he modelling sage, family members were asked o sare a he Kinec a a shor disance. ~ 2. m) so clear facial images can be capured and recorded in he appearance daabase. As shown in Figure, a leas wo face emplaes fron and 45 view) were sored in he appearance daabase for each regisered person. Assuming ha a human face is symmeric horizonally, he face emplae in 45 view can be simply obained by mirroring ha in +45 view Modelling of Body Appearance Whenever a person passed he living room, muli-view full-body color images were capured and he skeleal informaion was analyzed simulaneously by wo Kinecs. Then he muli-view full-body appearance informaion was compaced and sored in a emplae image, called he Flaened Cylindrical Templae FCT), in he appearance daabase. As shown in Figure 3, he FCT combined he capured images of a person from various viewpoins and covered he appearance of a full body in 36. The FCT was consruced by an image mosaicking process ha aligned and siched one verical slice in he capured color images a a ime along he racked skeleal spine. The aforemenioned head pose angles can also provide alignmen cues in FCT consrucion. As shown in Figure, a leas one consruced FCT was sored in he appearance daabase for each regisered person. Whenever he regisered person was idenified by face recogniion laer on bu wearing a differen se of clohes, a new FCT was consruced and added o he appearance daabase auomaically. Figure 3. FCT-based human idenificaion and racking. θ 3.3. Modelling of Body Silhouee In addiion o he face images and body colors FCT), body builds also provide imporan informaion for human idenificaion. We modeled a body build as a se of body silhouees observed from various poins of views. The player masks provided by he Kinec s SDK were coupled wih

8 Sensors 24, skeleon racking and failed o properly segmen human hair as foreground. In our modelling sage, he body silhouees were segmened by a background subracion echnique solely based on deph images. Because he deph informaion was invarian o he exisence of shadows, he problem of casing shadows was solved inherenly. Figure 4 shows he exraced body silhouees for an adul and a child in muliple views. Three upper-lef human silhouees in red recangles) indicae he segmened human masks in hree disinc facing direcions. Assuming ha a human body is symmeric horizonally, human silhouees in he oher five facing direcions can be simply obained by mirroring shown as green arrow in Figure 4) or cloning shown as blue arrow in Figure 4). As shown in Figure, a leas hree body silhouees fron, 45, and side view) are sored in he appearance daabase for each regisered person. Figure 4. Segmened body silhouees in various facing direcions. a) Adul; b) Child. a) b) 4. Mulisensor Human Deecion and Tracking In he guidance sage, humans can be deeced and racked by a Kinec or a color camera. For a person locaed a he world coordinae x, y, z), he ground plane coordinae x, z) and he facing direcion θ of he person were racked across muliple cameras. Because hese cameras can be insalled in a wide variey of posiions and orienaions inside he house, he relaionship beween he projecing image) coordinaes and world floor) coordinaes should be discovered for each camera. The projecive ransform beween he image plane and he ground plane of he house was unique for each saic camera and was compued by four pairs of corresponding poins by a camera calibraion process in he seup phase. Iniially, four poins on he floor of he house were specified manually. For each camera, he coordinaes of hese four poins on he ground plane and he four corresponding poins on

9 Sensors 24, he image plane were uilized o compue a 3 3 marix called homography. Wih he help of he homography, an image coordinae can be mapped o a ground plane coordinae so ha human deeced by differen cameras can be combined and racked in a unified ground plane coordinae sysem. Similarly, he facing direcion θ relaive o each camera s focal axis can be convered o a global facing direcion θ before he mulisensor fusion. In addiion, a muliple camera synchronizaion was performed using a sofware-based approach [25]. Finally, he complee or parial measuremens from muliple cameras were inegraed using a Kalman filer as shown in Figure 5a. I should be noed ha our applicaion is no 3D scene reconsrucion, hus sophisicaed camera calibraion using checkerboard paern) and synchronizaion using hardware genlock) approaches are no required. Figure 5. Two daa fusion approaches in he proposed sysem: a) mulisensor fusion for human racking; b) mulimodal fusion for human idenificaion. Kinec Color Camera Mulisensor Fusion for Human Tracking by Kalman Filer Person Locaion & Facing Direcion a) Face Feaure FCT Feaure Disance measured by Chi-square Disance measured by Bhaacharyya Mulimodal Fusion for Human Idenificaion by Track-based Majoriy Voing Person ID Silhouee Feaure Disance measured by Hasudorff b) 4.. Human Deecion and Tracking Using a Kinec Human deecion and racking using a Kinec is sraighforward because he Kinec skeleon racking provides he hree-dimensional coordinaes of weny skeleal joins as described in Secion 3). The ground plane coordinae x, z) of a person was exraced from he skeleon join HIP_CENTER. The facing direcion θ was deermined in wo ways. In a shorer disance where he Kinec face racking was effecive, he facing direcion θ was se o he yaw angle of he head pose as described in Secion 3.); Oherwise, he facing direcion θ was se o he angle beween he skeleal forward direcion and he focal axis of he camera Human Deecion and Tracking Using a Color Camera Based on color images acquired by a saic camera eiher a Kinec or a color camera), moving people were deeced by a background subracion algorihm ha inegraed he informaion of color, shading, exure, neighborhood, and emporal consisency [26]. Assuming ha a person sands on he

10 Sensors 24, floor, he deph z beween he person and he camera was deermined by mapping he person s foo coordinae in he image plane o he ground plane hrough he aforemenioned homography ransformaion. Wih he availabiliy of he deph z and he focal lengh f, each image plane coordinae x, y ) can be projeced o he world coordinae x, y) using he following equaions: = ; = The human body appearance provides addiional cues for human racking and idenificaion. The acquired color images were compared wih he regisered human emplaes in he appearance daabase. A deeced person was racked in he image space using a color-based Mean-Shif approach. The arge model was represened by a smaller bounding box BB) covering he arge person and including only he arge pixels in he capured color image. The BB was enlarged proporionally o form a larger BB such ha he surrounding pixels in he ring area beween he larger and smaller BBs were chosen o represen he background. For a feaure value i, we calculaed pi) as a normalized hisogram of he pixels on he arge, and qi) as a normalized hisogram of he pixels on he background. A log likelihood image [6] was consruced. Subsequenly, an ieraive Mean-Shif algorihm was performed in he log likelihood image unil he BB converged o he locaion of he arge person in he curren frame as shown in he upper par of Figure 3. A he same ime, he arge BB was used as a emplae o shif is corresponding BB in he FCT as shown in he lower par of Figure 3. In oher words, a person was racked simulaneously on wo image domains: horizonally/verically on he capured image o updae image coordinae x, y ), and horizonally on he FCT image o updae facing direcion θ. By repeaing his process ieraively, he arge, background, and FCT models evolved over ime all ogeher. In a ypical adapive racking procedure wih a dynamic updae of he arge model, he model drif problem appears over ime as misclassified background pixels gradually join he foreground model, evenually leading o a racking failure. To avoid his problem, he hisogram of he arge model was compued by considering he pixels in he arge BB in he curren capured image as well as he pixels in he corresponding BB in he FCT. As a resul, he arge model was adapive o keep up wih he newes condiions. Simulaneously, he arge model was consrained by he a priori informaion in he FCT o preven accumulaion of model drif errors Mulisensor Human Tracking Based on a Kalman Filer A person was racked using a hree dimensional vecor conaining he world coordinae x, z) and he facing direcion θ across muliple sensors as shown in Figure 5a. From ime o ime, a sensor can fail o rack in some dimensions and produce only parial measuremens. I is also possible ha muliple sensors observe he same person simulaneously and provide duplicae measuremens. To address hese problems, a mulisensor fusion was performed o accommodae parial and duplicae measured daa using a Kalman filer [27]. The four dimensional sae vecor X) and he hree dimensional measuremen vecor Z) a ime sep are shown as follows: )

11 Sensors 24, = = θ) z) x) ) v z x X) Z, ) ) ) ) θ 2) where x), z)) is he ground plan coordinae, θ) is he facing direcion, and v) is he velociy on he ground plan of he racked person a ime sep. The relaionship beween he sae vecor X) and he measuremen vecor Z) can be formulized as: + = + = + ) ) ) ) u X H Z w) X) A) ) X 3) where A) is he sae ransiion marix or called predicion marix) and H) is he observaion marix or called measuremen marix). The variables w) and u) are zero-mean whie Gaussian noise wih covariance marices Q) and R), respecively. In he proposed human racking, he Kalman filer was used o produce he opimal sae esimae given a sequence of measuremens. A each ime sep, a Kalman filer was applied by an ieraive process wih wo seps. The firs sep was he ime updae or called predicor) ha projeced forward he curren sae esimae o obain an a priori esimae for he nex ime sep. A linear model wih a consan velociy was used for he predicion of he human movemen, i.e., he sae ransiion marix was equal o: = )) sin )) cos ) A θ θ 4) The second sep was he measuremen updae or called correcor) ha incorporaed a new measuremen ino he a priori esimae o obain an improved a poseriori esimae. The predicion and measuremen values were combined according o he predicion and measuremen variance. For a complee measuremen wih hree dimensions, he observaion marix was equal o: = ) H 5) In case only parial measuremen was obained eiher facing direcion or ground plan coordinae was unavailable), he negaive impac of he missing daa can be cancelled by seing he observaion marix o: = = ) or ) eiher H H 6) The addiive naure of he updae sage makes he Kalman filer very aracive for mulisensor fusion wih duplicae measuremens. Supposing ha here were a se of N sensors, Z i ) was he measuremen produced by he i-h sensor, and K i ) was he Kalman gain for he daa fusion associaed

12 Sensors 24, o he i-h sensor a ime sep, he sae esimae X) was updaed according o he measuremens Z i ) using he following equaion [28]: X) = X) + N i= K i ) [Z i) H i) X )] 7) The covariance marix R) reflecs he uncerainy of he measuremens and depends on he characerisics of each sensor. In our implemenaion, he covariance marix R) of each sensor was deermined by finding he variance in he measuremen daa ha were colleced. The Kalman filer was furher exended o suppor muliple people racking. Supposing ha here were m mainained racks and n deeced persons, an m n cos marix was consruced by compuing he cos of assigning every deeced person o each rack based on he disance beween he posiion of a deeced person and he prediced locaion of an exising rack. The cos of assigning a deecion o a rack was defined as he negaive log-likelihood of he deecion corresponding o he rack. The associaion problem was solved by generaing a deecion-o-rack assignmen which minimized he oal coss. In a given frame, some deecions migh be assigned o exising racks, while oher deecions and racks may remain unassigned. If a deeced person was assigned o an exising rack, he informaion of he person was uilized o updae he parameers of he assigned rack by he Kalman filer; oherwise, a new rack was creaed for he unassigned person. Tracks ha have been coninuously updaed for a fixed number of frames were classified as seady. Opposiely, racks ha have no been updaed for a fixed number of frames were discarded. 5. Mulimodal Human Idenificaion Human can be idenified by comparing he color images capured by eiher a Kinecs or a color camera) wih he regisered emplaes in he appearance daabase using an individual modaliy of he face, body color, or silhouee. As shown in Figure 5b, a decision level fusion was applied for he human idenificaion o adop he mos suiable feaure se, disance measure, and idenificaion mehod for each single modaliy. Aferwards, he idenificaion oupus were inegraed by a rack-based majoriy voing o make a final idenificaion decision. 5.. Human Idenificaion Using Faces Face recogniion can be performed if he deeced person faces he camera wihin a close range. Eigenfaces [9] and Fisherfaces [2] are holisic approaches ha work in a high-dimensional image space and require several facial images for each person o achieve good recogniion raes. Alernaively, we adoped local binary paerns LBP) which summarized he local srucure in a facial image by comparing each pixel wih is neighborhood. By definiion he LBP operaor is robus agains illuminaion changes. Afer he locaion and size of he face of he capured person were deermined, he cropped face was aligned o uprigh pose and normalized o a sandard size. Subsequenly, he LBP image was divided ino local regions and a hisogram was exraced from each region. The spaially enhanced feaure vecor, called LBPH [29], was hen obained by concaenaing he hisograms of LBPs. The 32 mos relevan feaures seleced hrough PCA were used o recognize a face by comparing a capured face wih each regisered face emplae in he appearance daabase.

13 Sensors 24, Afer comparing various disance or dissimilariy) measures, we found ha he bes disance measure for he comparison of LBPH beween hisograms S and M was a Chi-square saisic:, ) = [ ) / + )] Human Idenificaion Using FCTs Because a clear facial image is no always available, he human body appearance provides addiional cues for he human idenificaion. A deeced person was racked using a color-based Mean-Shif approach described in Secion 4.2, and represened as a bounding box BB). To mach color informaion in he deeced BB and a BB of a FCT in he appearance daabase, a hisogram was divided o 32 bins and each bin covered a horizonal sripe of he human body. Afer comparing various color spaces and disance measures, we found ha he YC b C r color space and he Bhaacharyya disance performed he bes in he process of he Mean-Shif racking and color hisogram maching. The Bhaacharyya disance beween wo hisograms S and M was defined as, ) =. Similar o he human racking, he arge model was compared wih each FCT in he appearance daabase using he same disance measure o find he bes mach for human idenificaion as shown in he lower par of Figure Human Idenificaion Using Silhouees In addiion o face images and body colors FCT), body shape informaion also provides disincive clues for he human idenificaion. In he guidance sage, human silhouees were exraced by he background subracion. To compare a deeced silhouee mask wih each silhouee emplae in he appearance daabase, wo conours were scaled, aligned, and mached. Firs, he size of a 2D conour was normalized o reflec is real size in 3D. Each segmened human silhouee was scaled by a facor ha was inversely proporional o he person s deph z ha was deermined in he aforemenioned homography compuaion. Second, wo conours were ranslaed so heir baryceners overlapped, and roaed so heir major axes aligned. Third, a polar coordinae space was equally divided ino 32 secors and each human silhouee was sampled as a 32 dimensional vecor accordingly. Finally, he Hausdorff disance [3] beween wo silhouees was compued. Given one se of poins A conaining pixels along he boundary of a deeced silhouee and anoher se of poins B conaining pixels along he boundary of a emplae silhouee, he Hausdorff disance provided a mean o deermine he resemblance of hese wo se of poins and was defined as he greaes of all he disances from a poin in A o he closes poin in B. To compare a porion of shapes for parial shape maching, he K h ranked disance was seleced insead of he maximal disance and he consan K conrolled how many poins of he model needed o be near poins of he arge, i.e.,, ) =maxh, ),h, ), where h, ) = min. The Hausdorff disance can be compued efficienly and can handle ranslaion, scaling, and parial shape maching effecively Mulimodal Human Idenificaion Based on a Majoriy Voing A a ime sep, he human idenificaion was performed individually using face, FCT, and silhouee modaliies. Face recogniion was possible if facial images can be capured clearly wihin a close range.

14 Sensors 24, Adding appearance informaion of body colors FCT) and he body shapes silhouee) furher promoed he recogniion accuracy. In each aforemenioned rack described in Secion 4), he idenificaion resuls of hese hree modaliies face, FCT, and silhouee) were accumulaed over ime using a majoriy voing for human idenificaion as shown in Figure 5b. To measure he qualiy of he idenificaion resul of each modaliy in each frame, a confidence value was defined as C d = d 2 d )/d 2 where d was he disance of he bes mach and d 2 was he disance of he second bes mach. The confidence value C d ranged from zero o one. The greaer he confidence value C d was, he higher he probabiliy was of he fac ha he bes mach wih he shores disance was he acual mach. The idenified person ID of each modaliy in each frame can voe wih a weigh ha was equal o is confidence C d. The person ID receiving he maximal number of voes was oupued as he final idenificaion resul of he mulimodal fusion. Compared o a frame-based idenificaion, he rack-based idenificaion wih a long-erm majoriy voing was much more reliable Mulimodular Cooperaion As shown in Figure 6, he deecion, racking, and idenificaion modules can cooperae seamlessly and benefi from each oher o achieve a robus sysem. The deecor reaed every frame independenly and performed a full scan of he image o localize poenial foreground people. To reduce false alarms made by he deecor, he racker esimaed he deeced person s moion beween consecuive frames under he assumpion ha he frame-o-frame moion was limied. Neverheless, he racker was likely o fail if he racked person moved ou of he camera s view. In his case, he deecor discovered any newly appearing person, hen re-iniialized he racker, and hus minimized he racking failures. Moreover, he idenificaion module can improve he racking performance by consraining he arge model by he regisered FCT of he idenified person o avoid model drif errors in he adapive updae process. Also, he idenificaion module can help he deecor o reduce miss-deecions by providing color disribuions of he recen idenified person for beer discriminaion beween he foreground and background in he process of he background subracion. Figure 6. Cooperaion among deecion, racking, and idenificaion modules. Targe deeced Idenificaion ID-guided racking ID-guided deecion Targe under racking Deecion Targe deeced Targe los Tracking

15 Sensors 24, Evaluaions Figure 7 shows a ypical layou of an elderly aparmen. Two Kinec were insalled on opposie sides wih overlapping field of view FOV) in he living room for modelling he muli-view full-body appearance, and wo color cameras were insalled o cover he views of he oher pars of he aparmen for human deecion, racking, and idenificaion. Figure 7. Typical layou of an elderly aparmen. Two Kinecs and wo color cameras were insalled o cover mos open areas in he aparmen. The firs experimen evaluaed he effeciveness of he mulisensor human racking using a Kalman filer. Five people walked around in he open space in he aparmen in urn and each individual rack on he ground plan was marked manually as he ground ruh. A oal of racks 5 frames in each rack) were recorded and racked across wo Kinecs and wo color cameras. For evaluaion purpose, a measuremen error and a Kalman esimae error for each racking feaure dimension f f = x, z, or θ) were defined using mean square error MSE): E E M f K f = = m n m n m T = = m n n T = = [ G T, ) M f f [ G T, ) K f f T, )] T, )] 2 2 8) where m is he number of racks, n is he number of frames in each rack; G f T, ) represens he ground ruh, M f T, ) indicaes he measuremen, and K f T, ) sands for he Kalman esimae of he feaure dimension f in he -h frame in he T-h rack. For a ypical rack in he aparmen, Figure 8 plos he measuremens of disinc sensors red cross marks for Kinec#, green plus marks for Kinec#2, blue circle marks for color#, and yellow riangle marks for color#2). The final esimaed

16 Sensors 24, rajecory afer mulisensor fusion using a Kalman filer is shown as a hin whie curve, and he acual rajecory ground ruh) is shown as a hick purple curve in Figure 8e. As shown in Figure 7, he open space in he aparmen can be divided o hree zones. The firs zone ZONE, lower par in Figure 7) was moniored by hree cameras Kinec#, Kinec#2, and Color#2). The second zone ZONE2, middle par in Figure 7) was moniored by wo cameras Kinec# and Color#2). The hird zone ZONE3, upper par in Figure 7) was moniored by a single camera Color#). Table 2 compares he racking errors of measuremens before racking) and Kalman esimaes afer racking) using disinc sensor ypes. Generally, he measuremen errors of Kinecs were lower han which of color cameras, and he measuremen errors of he feaure dimension z were higher han which of feaure dimension x. The Kalman filer effecively reduced he racking error for boh Kinec and color camera. Table 3 compares he racking errors of measuremens before racking) and Kalman esimaes afer racking) combining differen number of sensors. I can be noed ha he negaive effecs caused by wrong or missed measuremens were suppressed by he mulisensor fusion using he Kalman filer. Compared o a Kalman racker using a single sensor ZONE3), a Kalman fusion of wo sensors ZONE2) made an improvemen of 2.4%, 37.4%, and 24.6% of he feaure dimension x, z, and θ, respecively; a Kalman fusion of hree sensors ZONE) achieved an improvemen of 38.8%, 5.%, and 46.% of he feaure dimension x, z, and θ, respecively. Figure 8. Measuremens of disinc sensors: a) red cross mark for Kinec#; b) green plus mark for Kinec#2; c) blue circle mark for color#; d) yellow riangle mark for color#2; e) mulisensor fusion; Thin whie curve indicaes he Kalman esimaed rajecory, and hick purple curve represens he acual rajecory ground ruh). a) b) c) d) e)

17 Sensors 24, Table 2. MSE errors of human racking using differen sensor ypes in each feaure dimension: x, z, and θ; he uni is cm, cm, and degree, respecively. Measuremens Esimaes Sensor Type Kinec Color Camera Table 3. MSE errors of mulisensor human racking using differen number of sensors in each feaure dimension: x, z, and θ. Number of Measuremens Esimaes Sensors ZONE3) ZONE2) ZONE) The second experimen evaluaed he reliabiliy of he mulimodal human idenificaion using he proposed majoriy voing. Five family members in an aparmen were involved: an elderly male ID#), a young adul male ID#2), a young adul female ID#3), a eenage female ID#4), and a oddler male ID#5). Tables 4 6 show he voe marices of human idenificaion of he proposed sysem solely based on faces, FCTs, and silhouees, respecively. Each number in a voe marix indicaed he number of voes of a specific person ID in an individual rack, each column represened an idenified person ID by he proposed sysem, and each row corresponded o an acual person ID in he rack. All correc voes were locaed in he diagonal of a voe marix. Similar o he aforemenioned confidence value of he idenificaion resul of each modaliy in each frame, he confidence value of he idenificaion resul of a rack was defined as C v = v v 2 )/v, where v was he highes number of voes and v 2 was he second highes number of voes. The confidence value C v ranges from zero o one. The greaer he confidence value C v is, he higher he probabiliy is of he fac ha he idenificaion resul wih he highes voes is he acual ID. The number wih he highes voes was emphasized in bold fon, and mismaches he winner ID no equal o he acual ID) were annoaed wih an exclamaion mark. Idenificaion solely based on faces ended o fail if clear face images were unavailable; Idenificaion solely based on FCTs could be confused by clohes wih similar colors and paerns; Idenificaion solely based on silhouees was ineffecive o differeniae people wih similar body builds. Table 7 shows he voe marix of he proposed human idenificaion considering faces, FCTs, and silhouees all ogeher. Wih he proposed mulimodal fusion, reliable idenificaion resuls were produced wih high confidence values. Figure 9a shows he voing resuls of a rack over ime. Three modaliies of faces, FCTs, and silhouees voed independenly o idenify a deeced person in he appearance daabase regisered wih five people. A he beginning of he rack, several ID compeed wih each oher and he compued confidence was low. As he rack coninues, more voes come in and ID# gradually go ahead over ime. Finally, ID# dominaed he voe a he end of he rack and he deeced person was idenified as ID# wih high confidence value. Figure 9b shows he changes of he winner ID wih he highes voes and is confidence value of a rack over ime. Figure 9c j shows he voing resuls of he oher four

18 Sensors 24, racks. Wih he accumulaion of voes in each rack, he proposed human idenificaion gradually obained more reliable resuls wih higher confidence values over ime. As shown in he hird and fourh racks, individual idenificaion modaliy ended o confuse beween ID#3 and ID#4 because hey have similar faces and body builds. Neverheless, he FCT modaliy can differeniaed hem well, and gradually accumulaed enough voes o make a correc idenificaion decision a he end. Table 4. Voe marix of he proposed human idenificaion solely based on Face. Real ID Recog. ID Confidence #. #2 #3 #4. #5 Winner ID Value # # 2% # #2 67% #3 4 #3 6% #4 7 6 #3!) 4% # #5 57% Table 5. Voe marix of he proposed human idenificaion solely based on FCT. Real ID Recog. ID Confidence # #2 #3 #4 #5 Winner ID Value # 3 5 # 55% #2 7 #2 94% # #3 64% # #4 6% # #3!) 44% Table 6. Voe marix of he proposed human idenificaion solely based on Silhouee. Real ID Recog. ID Confidence # #2 #3 #4 #5 Winner ID Value # #2!) 3% # #2 4% # #4!) 4% #4 2 3 #4 73% #5 2 6 #5 88% Table 7. Voe marix of he proposed human idenificaion based on a mulimodal fusion. Real ID Recog. ID Confidence # #2 #3 #4 #5 Winner ID Value # # 48% # #2 77% # #3 44% # #4 48% # #5 6%

19 Sensors 24, Figure 9. Track-based voing for he human idenificaion in five racks wih disinc IDs. a) he voing resul over ime in he firs rack; b) he winner ID wih he highes voes and is confidence value over ime in he firs rack; c j) for he second~fifh rack. # of voes ID# ID#2 ID#3 ID#4 ID#5 Winner ID Confidence value a) b) # of voes ID# ID#2 ID#3 ID#4 ID#5 Winner ID Confidence value c) d) # of voes ID# ID#2 ID#3 ID#4 ID#5 Winner ID Confidence value e) f) # of voes ID# ID#2 ID#3 ID#4 ID#5 Winner ID Confidence value g) h) # of voes ID# ID#2 ID#3 ID#4 ID#5 Winner ID Confidence value i) j)

20 Sensors 24, Figure. Consruced FCTs of five adul males wih similar body builds. a) b) c) d) e) Table 8. Comparison of he recogniion rae using differen mehods in various scenarios. Scenarios # #2 #3 #4 #5 Paricipans persons, one a a ime 5 adul males wih similar body builds 2 females wih similar faces 3 persons wearing similar colored dresses 2 or 3 persons in he scene simulaneously # of racks Recog. Rae of he Proposed 96% 92% 95% 87% 9% mulimodal mehod Recog. Rae of Gandhi s full-body 89% 88% 95% 63% 8% recogniion [2] Recog. Rae of Ahonen s face recogniion [29] 74% 76% 6% 7% 7% The hird experimen compared he performance of he proposed mulimodal idenificaion, Gandhi s full-body appearance idenificaion [2], and Ahonen s face recogniion [29] in an aparmen wih wo Kinecs and wo cameras. Gandhi s mehod was implemened o idenify full-body appearance samples which were obained from all visible cameras and inegraed using regisraion and emporal averaging described in heir paper. Ahonen s mehod was implemened o recognize fronal face samples whenever hey were deeced wihin a close range of any visible camera. To make a fair comparison, idenificaion resuls of each mehod were accumulaed over ime using he proposed majoriy voing described in Secion 5.4) o make a rack-based final decision. Table 8 shows he recogniion raes using differen mehods in various scenarios. The firs scenario was under normal condiions in ha en paricipans walked around in he aparmen one a a ime. Their FCTs were consruced and sored in he appearance daabase using he image mosaicking process described in Secion 3.2. The second, hird, fourh, and fifh scenarios were under more challenging condiions. The second scenario consised of five adul males wih similar body builds consruced FCTs as shown in Figure ). Even if he silhouee modaliy failed o make a disincion, he oher modaliies FCT and face) sill worked and helped o make a correc final decision in he process of he majoriy voing. The

21 Sensors 24, hird scenario consised of wo females wih similar faces. There was some ambiguiy using Ahonen s face recogniion due o he resemblance of face feaures. The proposed mulimodal mehod ouperformed heirs by uilizing more modaliies FCT and silhouee). The fourh scenario consised of hree persons wearing similar colored dresses. Gandhi s mehod was confused by he small disance measure beween full-body appearance samples wih similar color layous. The proposed mulimodal recogniion correcly disambiguaed hem by he oher modaliies face and silhouee). The fifh scenario evaluaed he robusness of he muliple people racking described in Secion 4.3, and compared he recogniion rae using differen mehods. Even if wo or hree persons appeared in he scene simulaneously, he resul indicaed ha he proposed mulimodal idenificaion was robus under muliple people condiions. The problem of visual occlusions beween wo persons was alleviaed in he proposed mulisensor environmen wih various perspecives and he mulimodal fusion. The fourh experimen involved en paricipans in four differen aparmens as shown in Figure ). The goal of he experimen was o collec feedback from real users. The paricipans were inerviewed regarding heir experiences using he proposed human deecion, racking, and idenificaion sysem a heir homes. Their commens indicaed he srenghs and weaknesses of he proposed sysem and provided direcions of furher improvemens. Even if cameras were no insalled inside he bedroom or bahroom, several paricipans expressed heir concerns regarding he privacy. Under normal condiions, he capured images can be immediaely desroyed once a rack is finished. In case an emergency is deeced, he capured images can be sored for record purpose only. The daabase and he capured images will no be shared wih anyone wihou auhorizaion. Also, family members can urn off any cameras a any ime. Anoher concern is he correcness of he auomaic modelling sage. Because a se of muliple FCTs can be sored for each person o remember heir favorie leisure wears a home, a FCT scan is performed and recorded in he appearance daabase semi-auomaically if a new sui of clohes is deeced. Human inervenion is only required in scheduled mainenance o correc any possible misplacemen made by he auomaic modelling process. Examples of commens regarding hese problems are as follows: Can I urn off he cameras by myself? I don wan o broadcas live videos of my personal space on Inerne Do I need o regiser again afer I change clohes a home? Despie hese problems, paricipans expec he sysem o provide useful informaion on heir daily rouines a home. Even if no camera was insalled inside he bahroom, each door of he bahroom or bedroom was in he FOV of a leas one camera. The proposed human deecion, racking, and idenificaion sysem can coun he number of imes a specific person ener he bahroom per day by monioring he image area around he door marked as red recangle in Figure 7). Similarly, he sysem can also record he ime a specific person ener or leave he bedroom for a rough measuremen of sleep amoun. The commens from paricipans indicaed ha hey anicipae he sysem o provide more inelligen services in he fuure: Can he sysem coun how many calories I burned by walks a home per day. Will he sysem warn me promply if my elders or kids fall a home? My docor wans o know how many imes I go o he bahrooms per day. I is nice o know he sysem will inform me if an inruder is deeced.