Activity Recognition using Hierarchical Hidden Markov Models on Streaming Sensor Data

Transcription

1 Aciviy Recogniion using Hierarchical Hidden Markov Models on Sreaming Sensor Daa Parviz Asghari Ambien Inelligence Research Lab. Deparmen of Compuer Engineering Amirkabir Universiy of Technology Tehran, Iran Ehsan Nazerfard Ambien Inelligence Research Lab. Deparmen of Compuer Engineering Amirkabir Universiy of Technology Tehran, Iran Absrac Aciviy recogniion from sensor daa deals wih various challenges, such as overlapping aciviies, aciviy labeling, and aciviy deecion. Alhough each challenge in he field of recogniion has grea imporance, he mos imporan one refers o online aciviy recogniion. The presen sudy ries o use online hierarchical hidden Markov model o deec an aciviy on he sream of sensor daa which can predic he aciviy in he environmen wih any sensor even. The aciviy recogniion samples were labeled by he saisical feaures such as he duraion of aciviy. The resuls of our proposed mehod es on wo differen daases of smar homes in he real world showed ha one daase has improved 4% and reached (59%) while he resuls reached 64.6% for he oher daa by using he bes mehods. Keywords Aciviy Recogniion, Hierarchical Hidden Markov models, Aciviy Segmenaion, Sliding Window, Smar homes, Inerne of hings I. INTRODUCTION The human aciviy affecs he person, sociey and he environmen. Therefore, human aciviy recogniion forms he basis of many research fields. Today, he advancemen of sensor echnology, as well as he evoluion of machine learning mehods, on he oher hand, provides a good conex for using sensors in various applicaions. One of hese applicaions is smar environmens ha are used o monior he behavior of a person living in he environmen. For example, monioring sysems ry o use aciviy recogniion echnology o deal wih erroris hreas. Assised living environmens used aciviy recogniion in order o help individuals o live independenly. Oher applicaions, for example, smar meeing rooms and smar hospials, are also dependen on aciviy recogniion. The aciviy recogniion is also used in video game console and applicaions for paiens' healh and finess on he smarphone. Smar environmens have recenly been used in many research laboraories hroughou world; herefore, many research projecs such as CASAS [1], MAVhome [2], PlaceLab [3], CARE [4], and AwareHome [5] have been deployed in his area. Aciviy recogniion refers o recognizing he daily aciviies of a person living in he environmen. Aciviies of daily living (ADLs) is a erminology defined in healhcare o refer o people's daily self-care aciviies. Aciviy recogniion can be examined from differen aspecs: A. Type of Sensor The firs models were using visible feaures such as camera images o monior individual behavior and environmenal changes. The sensors daa is a sequence of films or digial images. The sensor-based model uses a nework of sensors o monior he aciviy. In his approach, sensors can be aached o he person or placed in he environmen. B. Aciviy model In general, aciviy models are made by wo mehods. In he firs mehod, a large se of daa of each person's behavior are analyzed by using machine learning and daa mining mehods. In he second mehod, i was ried o obain sufficien prior knowledge in he field of ineress and personal endencies by using knowledge engineering and managemen echnologies. Aciviy recogniion deals wih various challenges, such as aciviy labeling, overlapping aciviies, sensors wihou useful informaion and aciviy exracion. Among hese challenges, online aciviy recogniion is idenified as he main challenge in his area because he firs sep in providing real-world sysems is o design sysems which can recognize online user behavior. In he presen sudy, sream of daa was made from a number of binary environmenal sensors. Mos mehods for online aciviy recogniion are based on he sliding window approach. These mehods deal wih he problem of opimizaion of he window size. The presen sudy ries o discover he sequence paern of he occurrence of he sensors a he beginning and ending of each aciviy o solve he window size problem, and also uses hese paerns o idenify he boundaries which means where he aciviy begins and ends on he sream of sensor daa. As a resul, i is no necessary o deermine he window size, and he model auomaically deermines he proper size for aciviy recogniion on he sream of daa. When he beginning of he aciviy wih he occurrence of each sensor is idenified, he online aciviy will be prediced and his predicion coninues unil he corresponding compleion paern of he aciviy occurs. When he aciviy was idenified, he aciviy recogniion was labeled by he saisical feaures such as he duraion of aciviy. The problem of sensors wihou informaion was also solved and he efficiency of he sysem performance increased. To implemen his idea, he XXX-X-XXXX-XXXX-X/XX/$XX.00 20XX IEEE

2 hierarchical Markov model was used. A se of hidden Markov models were responsible for recognizing of he beginning, ending, and ype of aciviy paerns. This se was implemened as a hierarchical hidden Markov model. The resuls showed ha he implemened algorihm was beer han oher available mehod or has he same performance. This paper is organized as follows: In secion 2, he previous sudies were examined. Then, he proposed mehod and implemenaion deails were described in Secion 3. In secion 4, various ess were performed o evaluae he efficiency of he model and compare he resuls of he proposed mehod wih oher mehods. In he final secion, conclusions and furher sudies were discussed. II. RELATED WORKS The basis of aciviy recogniion is o process of sequence evens of he sensors and recognizes he corresponding aciviy. I is required o have a robus model for aciviy recogniion due o he difference in he environmenal srucure and he sensors in our smar environmens. The Diane proposed a robus model of he sensors o exrac differen feaures, such as he duraion of he aciviy [6]. In he modeling, he problem was independen of he sensor environmen and could be implemened for differen users [7]. Since he accuracy of he model requires he daa wih proper labels, a mehod was proposed for he daa labeling [8]. Firs, he daa for aciviy recogniion have already been segmened (i.e. he beginning and ending of each aciviy are idenified) [9, 10]. In order o bring he aciviy recogniion based sysems closer o hose of acual world, some mehods were used for segmenaion of he sream of he daa [11, 12]. Aciviy labeling is one of he aspecs and i is ofen ignored. Mos researchers asked he residens o perform an aciviy and hen mark he aciviy based on he acivaed sensors. This mehod was no pracically possible for all people. Moreover, Szewcyzk e al. [8] presened an auomaed mehod o annoae he daases. Oher challenges for aciviy recogniion were overlapping, and simulaneiy of aciviies. Overlapping means ha differen classes have common sensors [13]. The nex problem is hose sensors which do no belong o any predefined classes; however, hey usually include a large segmen of he daa se. Rashidi e al. [14] suggesed ha he corresponding new aciviy of hese unprocessed sensors should be discovered, and added o predefined classes which improve he efficiency of he sysem. In addiion o he challenges for aciviy modeling, machine learning mehods were also discussed for aciviy recogniion. Therefore, differen mehods including he ensemble mehod [15], he non-parameric mehod [16], he suppor vecor machine mehod [9], as well as probabilisic models such as he hidden Markov model and he Markov random field have been used for aciviy recogniion [17, 18, 19]. Each mehod has is own advanages and disadvanages. Probabilisic mehods have high applicaion due o high noise olerance and he producion of probabiliy disribuions over differen classes [6]. Alhough each challenge has is own paricular imporance, online aciviy recogniion is one of he mos imporan challenges in he real world. Real world applicaions require sysems which can recognize he user behavior immediaely. Therefore, hese mehods should be able o process he sream of sensor daa. Mos mehods for daa sream processing are based on he sliding window approach [11, 20, 21, 22]. The sliding window approach, referred o as windowing is mainly based on he ime or number of sensors. Iniial mehods considered he window size consan [18, 20]. Since differen classes have differen numbers of acivaed sensor, he sliding window size was proposed as a soluion by many researchers [20, 22, 11, 23, 24]. Krishna e al. [20] menioned he ime dependence of he sensors as a crierion for dynamic window approach. They provided wo window approaches based on he ime and number of sensors. In he emporal sliding window approach, hose sensors, which ook place a a cerain inerval, were examined as one aciviy, while in he sensorbased mehod, firs, hose sensors which were coninuously acivaed ogeher, were recognized by using Muual informaion crierion and hen were classified in a window. The probabilisic approach was anoher proper idea which was provided by Fadi e al. [22]. In his approach, he window size was considered for each class. The iniial size of he class was esimaed according o he previous samples and was updaed based on he sensor evens. Yala e al. emphasized on he mehod [20] and he changes in compuing muual informaion and improved he sysem performance [21]. Kabir e al. [17] proposed a muli-sage mehod. In he firs sage, he aciviies were clusered by a hidden Markov model and hen he aciviies of each cluser were classified by using a separae hidden Markov model. Alhough he proposed mehods have grealy solved he window size problem, here are sill some problems such as he exac recogniion of boundaries which reduces he efficiency of he sysem. III. THE PROPOSED METHOD The problem of hese mehods was heir inabiliy in deermining he window size precisely. Regarding he uncerainy abou he window size and he approximae calculaion of he window size, he recogniion windows conain he evens of sensors which do no belong o any predefined aciviies. The segmenaion of he daa wihou clear window size, and classificaion of he pieces as predefined classes and oher class was proposed. Predefined classes included daily aciviies such as eaing, bahing, sleeping, and so on, which were deermined by medical specialiss. Oher classes are aciviies which are no included in his classificaion. A firs, he beginning and ending of he aciviies was ried o be recognized by idenifying repeiive sensors in he beginning and ending of aciviies in order o segmen he sream of daa. Afer he recogniion of he beginning of he aciviy, he corresponding aciviy will be prediced by each sensor even. When he aciviy ended and he corresponding piece was recognized, he pieces will be labeled by using he emporal feaures. Figure 1 illusraes he seps of he proposed algorihm. In he aciviy discovery phase, when he sream of daa enered, he sensors wih informaion were firs segmened and hen classified based on he hreshold of he recognized segmens in he labeling phase. Each phase will be explained in he following secions.

3 Figure 1 Seps of proposed mehod - phase one aciviy deecion, phase 2 labeling A. Aciviy Discovery Phase The firs problem for aciviy recogniion on sreams of daa is o recognize he occurrence of aciviies. The presen sudy ries o examine he sensor even a he beginning and ending of each aciviy in order o recognize he sar and end of i. As Figure 2 illusraes, sensors, which are acive before and end of each aciviy, are usually consan. This is resuled from he fac ha he person performs each aciviy in a specified locaion, and hus a specific se of sensors are acivaed a he beginning and he ending of he specified locaion. received sensors, he more accurae he recognized aciviy would be. As i is menioned earlier, he probabilisic models have an accepable performance in dealing wih noise daa. In hese models, he prior knowledge can be simply ransferred o he models. Since he hidden Markov model has less compuaional load recursive neural neworks han oher similar mehods, i is used as a common mehod for sequenial daa. Suppose X represen he hidden sae vecor and Y represens vecor observaions. K is he number of possible hidden saes for X. X {1,..., k}. Here, observaions mean he sequence of sensor evens. Equaion 1 illusraes how o calculae he mos probable class based on he observaions. P( X y1: ) = P( y X ) P( X x 1) P( x 1 y1: 1) x1 Hidden Markov model will face difficulies in dealing wih long sequences. Therefore, hierarchical hidden Markov model was seleced. Hierarchical hidden Markov model is a closed version of he hidden Markov model which is appropriae for domains which have a hierarchical srucure and mulilevel dependencies on lengh and ime. Figure 3 illusraes he srucure of he employed hierarchical hidden Markov model. In his nework, doed line vecors represen he verical ransiion and solid line vecors represen horizonal ransiion. When absrac sae is done, horizonal ransiion can occurs. In Figure 3, X 1, X 2, and X represen absrac saes. Each absrac sae 3 is a sub-hmm and produces sequence of observaions. The lower nodes are producion saes ha produce one observaion. The node X 1 is a sub-hmm which recognizes he beginning of he aciviy. Then, he conrol goes o node X 2. In his sage of he HHMM, he ype of aciviy is recognized, and he corresponding aciviy will be prediced by sensor even of each sensor. Afer his sage, node X 3 recognizes he erminaion of he aciviy upon is compleion. A he end, he conrol reurns o he roo node. In he oupu secion, a par of sensors wih informaion is idenified which is shown as S in he presen sudy. Figure 2 sensor frequency of personal hygiene acivy. (a)before aciviy begins- (b) a he end of acviy Some aciviies ake place in a common place. For example, aciviies such as personal hygiene and bahing ake place in one place. To disinguish hese aciviies from each oher, he sequence of sensor evens is used. Afer he recogniion of he beginning of he aciviy, he corresponding aciviy will be esimaed by receiving each sensor. The more B. Aciviy Labeling Phase Figure 3 Implemened HHMM srucure In he daase, undefined classes have no been labeled. Therefore, defined classes were used o idenify hese classes. The defined classes are denoed by C. In fac, empirical probabilisic disribuion of C k is exraced, and hen he likelihood ha S belongs o C k is calculaed. As

4 equaion 2 represens, if his probabiliy exceeds a hreshold, aciviy is considered as a class C k, bu if he probabiliy is lower han he hreshold, i is classified as he oher class. 1 f ( s TC ) k P( S Ck ) 0 f ( s TC ) k,, s Where f (.) and T c k represen likelihood funcion, he hreshold, duraion of he aciviy segmen S and he duraion empirical disribuion of he class C k, respecively. Also, k denoes he class index, where k {1,2,3,...,11}. IV. DATASETS AND EXPERIMENTAL RESULTS In order o evaluae he efficiency of he proposed model, he daa of wo inelligen homes were used, which will be described in he followings. Then, he evaluaion crieria and he comparison mehods were explained. Accuracy is one of he mos commonly used evaluaion crieria, and represens he proporion of he oal number of posiive and negaive correc samples o he all samples (equaion 3). p n accuracy p n fp fn A. Esimain he parameers 1) The number of sensors preceeding an aciviy To recognize he beginning of he aciviy, a number of differen sensors has been invesigaed (Table 2). Table 2 Esimaing he number of preceeding sensors Home % 97% 92.9% 93.2% 93.1% Home % 96.4% 94.2% 91% 91% Table 2 provides he accuracy of deecing he beginning of an aciviy. The resuls sugges ha by considering hree sensors, he bes accuracy is archived. 2) Threshold Parameer In order o deermine he appropriae hreshold, he value of has been changed from 0.02 o 0.10 and he opimum hreshold was deermined. Threshold values and differen accuracy have been repored in Table 3. Table 3 Esimaing he alpha parameer Home % 62.1% 63% 64.6% Home 2 54% 53% 59% 57% % 56.4% In able 4, he mehods which were used for he comparison wih he proposed model are described briefly. Figure 4 floor plan and sensor layou of home 1 Figure 4 illusraes he smar home archiecure. This figure illusraes fixed moion sensors in red dos and door sensors in blue riangles. Table 1 illusraes a sample of his daase. Table 1 Characerisics of he smar homes Home 1 Home 2 # of moion sensors # of door sensors # of residens 1 person 1 person # of sensor evens colleced Timespan 5 monh 4 monh Table 1 illusraes he general specificaions of home 1 and home 2 daa ses and he used sensors. Aciviy classes include, personal hygiene, ener home, leave home, aking shower, cooking, relaxing on he couch, ake medicine, eaing, housekeeping, sleeping in bed, and bed o oile ransiion. 1 Of daases, 70%, 10% and 20% were used for raining, validaion and esing, respecively. 1 Resuls provided in he validaion secion correspond o he home 1 and home 2 daases, available online a X 1,2. hp://eecs.wsu.edu/~nazerfard/air/daases/daax.zip, where Table 4 Noaion and descripion of differen approaches Noaion Descripion Baseline Baseline approach of fixed lengh Sliding window[20] SWTW Fixed lengh sensors windows wih ime based weighing sensors[20] SWMI Fixed lengh sensors windows wih muual informaion weighing sensors[21] DW Sensor windows where he window size deermined dynamically[22] The resuls of he proposed mehod as well as he resuls of oher mehods are repored in Table 5. The numbers represen he efficiency of he algorihm based on he accuracy crierion. Table 5 Accuracy Resuls Mehod Home 1 Home 2 Proposed mehod 64.6% 59% SWMI 64% 54% SWTW 62% 45% DW 59% 55% Baseline 58% 48% As Table 5 illusraes, our proposed mehod performs beer han he oher approaches. The proposed mehod can also be used o represen an esimae of an aciviy a any given momen.

5 V. CONCLUSIONS AND FUTURE WORK Online aciviy recogniion is one of he mos imporan challenges in he area of smar environmen research. Mos of he previous mehods for aciviy recogniion had problem in deermining he window size. The presen sudy segmened he sensors for each even aciviy regardless of he window size by deermining he beginning and ending of he aciviy. The resuls showed he efficiency of he proposed mehod compared o he available mehods. In general, he accuracy of he proposed mehods for he aciviy recogniion can exceed han a cerain amoun due o he uncerainy of human behavior. Therefore, i is pracical o propose in he smar environmen where aciviy is likely o occur. Because he raining of aciviy recogniion sysems requires a large amoun of daa, i is recommended ha ransiion learning and oher people s informaion should be used o reduce he duraion of collecing daa and raining sysem. REFERENCES [1] D. Cook, M. Schmier-Edgecombe, A. Crandall, C. Sanders, and B. Thomas, Collecing and disseminaing smar home sensor daa in he CASAS projec, CHI Work. Dev. Shar. Home Behav. Daases o Adv. HCI Ubiquious Compu. Res., no. November 2015, [2] D. Cook, MavHome: An agen-based smar home, 2003.(PerCom 2003, no. January, pp , [3] S. Inille and K. Larson, The PlaceLab: A live-in laboraory for pervasive compuing research (video), PERVASIVE 2005 Video Progr., pp. 1 4, [4] B. Kröse, T. van Kaseren, C. Gibson, and T. van den Dool, CARE: Conex Awareness in Residences for Elderly, In. Conf. In. Soc. Geronechnology, pp , [5] C. D. Kidd e al., The aware home: A living laboraory for ubiquious compuing research, in Lecure Noes in Compuer Science (including subseries Lecure Noes in Arificial Inelligence and Lecure Noes in Bioinformaics), 1999, vol. 1670, pp [6] D. J. Cook, Learning Seing- Generalized Aciviy Models for Smar Spaces, IEEE Inell. Sys., vol. 27, no. 1, pp , [7] D. J. Cook and M. Schmier-Edgecombe, Assessing he qualiy of aciviies in a smar environmen, Mehods Inf. Med., vol. 48, no. 5, pp , [8] S. Szewcyzk, K. Dwan, B. Minor, B. Swedlove, and D. Cook, Annoaing smar environmen sensor daa for aciviy learning, Technol. Heal. Care, vol. 17, no. 3, pp , [9] D. Sanchez, M. Tenori, and J. Favela, Aciviy Recogniion for he Smar Hospial, IEEE Inell. Sys., vol. 23, no. 2, pp , [10] A. Fleury, N. Noury, and M. Vacher, Supervised classificaion of Aciviies of Daily Living in Healh Smar Homes using SVM., Conf. Proc.... Annu. In. Conf. IEEE Eng. Med. Biol. Soc. IEEE Eng. Med. Biol. Soc. Annu. Conf., vol. 2009, pp , [11] G. Okeyo, L. Chen, H. Wang, and R. Serri, Dynamic sensor daa segmenaion for real-ime knowledge-driven aciviy recogniion, Pervasive Mob. Compu., vol. 10, no. PART B, pp , [12] X. Hong and C. D. Nugen, Segmening sensor daa for aciviy monioring in smar environmens, Pers. Ubiquious Compu., vol. 17, no. 3, pp , [13] J. Wen, M. Zhong, and Z. Wang, Aciviy recogniion wih weighed frequen paerns mining in smar environmens, Exper Sys. Appl., vol. 42, no , pp , [14] D. J. Cook, N. C. Krishnan, and P. Rashidi, Aciviy Discovery and Aciviy Recogniion: A New Parnership, Cybern. IEEE Trans., vol. 43, no. 3, pp , [15] A. Jurek, C. Nugen, Y. Bi, and S. Wu, Cluseringbased ensemble learning for aciviy recogniion in smar homes, Sensors (Basel)., vol. 14, no. 7, pp , [16] F. T. Sun, Y. T. Yeh, H. T. Cheng, C. Kuo, and M. Griss, Nonparameric discovery of human rouines from sensor daa, 2014 IEEE In. Conf. Pervasive Compu. Commun. PerCom 2014, pp , [17] M. H. Kabir, M. R. Hoque, K. Thapa, and S.-H. Yang, Two-Layer Hidden Markov Model for Human Aciviy Recogniion in Home Environmens, In. J. Disrib. Sens. Neworks, vol. 12, no. 1, p , [18] T. L. M. Van Kaseren, G. Englebienne, and B. J. A. Kröse, An aciviy monioring sysem for elderly care using generaive and discriminaive models, Pers. Ubiquious Compu., vol. 14, no. 6, pp , [19] S. Yan, Y. Liao, X. Feng, and Y. Liu, Real ime aciviy recogniion on sreaming sensor daa for smar environmens, 2016 In. Conf. Prog. Informaics Compu., pp , [20] N. C. Krishnan and D. J. Cook, Aciviy recogniion on sreaming sensor daa, Pervasive Mob. Compu., vol. 10, no. PART B, pp , [21] N. Yala, B. Fergani, and A. Fleury, Feaure exracion for human aciviy recogniion on sreaming daa, 2015 In. Symp. Innov. Inell. Sys. Appl., no. Ocober, pp. 1 6, [22] F. Al Macho, H. C. Mayr, and S. Ranasinghe, A windowing approach for aciviy recogniion in sensor daa sreams, In. Conf. Ubiquious Fuur. Neworks, ICUFN, vol Augus, pp , [23] J. Wan, M. J. O Grady, and G. M. P. O Hare, Dynamic sensor even segmenaion for real-ime aciviy recogniion in a smar home conex, Pers. Ubiquious Compu., vol. 19, no. 2, pp , [24] M. Espinilla, J. Medina, J. Hallberg, and C. Nugen, A new approach based on emporal sub-windows for online sensor-based aciviy recogniion, J. Ambien Inell. Humaniz. Compu., vol. 0, no. 0, pp. 1 13, 2018.