Available online at ScienceDirect. Procedia Computer Science 60 (2015 )

Transcription

1 Available online at ScienceDirect Procedia Computer Science 60 (2015 ) th International Conference on Knowledge Based and Intelligent Information and Engineering Systems Toward application of immunity-based model to gait recognition using smart phone sensors: a study of various walking states Yuji Watanabe* Nagoya City University, 1 Yamanohata, Mizuho-cho, Mizuho-ku, Nagoya , Japan Abstract In our previous study on gait recognition, we developed the application on ios for smart phone to record not only the acceleration but also more data such as the rotation of the phone around the 3 axes. And then we performed preliminary experiments using the application to collect the user-generated acceleration data on walking when the phone was held calling and touching as well as in the. However, the acceleration data were recorded for only 4 subjects on one day. In this study, we carry out additional experiments increasing the number of subjects and adding the data recorded on other day. The results show that although the correctly classified rate when the phone is in the is not yet affected by the number of subjects, the performance in the cases of calling and touching is influenced by the small number of subjects. Interestingly, we observe that the additional data on other day have a different effect on identification performance according to phone holding states. We also examine the influence of various walking states for one subject. Furthermore, we discuss the application of an immunity-based diagnosis model to gait recognition to integrate the identification results from multiple smart phone sensors The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license 2015 The Authors. Published by Elsevier B.V. ( Peer-review under under responsibility responsibility of KES of KES International International. Keywords: Biometrics; Gait recognition; Smart phone; Immunity-based model 1. Introduction For the last decade, mobile devices, such as smart phones, tablets, and wearable computers have been rapidly spread. Mobile devices have a lot of important private information, so that user authentication is increasingly * Corresponding author. Tel.: ; fax: address: yuji@nsc.nagoya-cu.ac.jp The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license ( Peer-review under responsibility of KES International doi: /j.procs

2 Yuji Watanabe / Procedia Computer Science 60 ( 2015 ) necessary to prevent the leak of the private information by unauthorized or careless usage. However, password authentication or biometrics authentications based on fingerprint or face are generally applied at the beginning of use because frequently retyping the password during use is extremely user-unfriendly to annoy the user. As a method of continuous and implicit user authentication without frequent user involvement, behavior-based biometrics authentication is a promising approach. Therefore, we are trying to develop a behavior-based user authentication system to continuously monitor user behaviors on smart phone 1, 2, 3, 4. At first, we focused on operational behaviors on touch screen and then made applications to record fingers history on smart phone 1. And then we reported the performance of touch screen-based biometrics authentication 2. Next, we examined gait authentication during walk using 3 axes accelerometers installed in smart phone 3, 4. This paper is a follow-up report on the acceleration-based gait authentication using smart phone. The identification of gait pattern with accelerometer was first proposed by Ailisto et al. in and followed by Gafurov et al. study 6. High-quality dedicated accelerometers were used to record the acceleration when subjects were walking. Some acceleration-based gait researches 7, 8, 9, 10 using smart phone have recently progressed because popular smart phones are generally equipped with accelerometers. In most former studies, accelerometers were mounted on hip, arm, or ankle, or smart phones were in the or the pouch in fixed manners. In other words, the acceleration data were collected in the same situation. However, when a user walks, smart phone is not only put in the but also used calling or touching on the screen. In these situations, the direction of the phone should be taken into account. Primo et al. 10 also pointed out the position of the phone and then compared the case of placing a Google Nexus in each of the and left hand s with the case of holding two phones (one in each hand). Ngo et al. 11 addressed the practical sensor-orientation inconsistency and showed that the proposed matching algorithm was robust to any initial sensor-orientation. However, they employed dedicated 4 accelerometers fixed at different orientations and locations on the same plate inside a backpack, that is, they do not use smart phone. In our previous study 4, we improved our application developed on ios to record not only the acceleration but also more data such as the rotation of a phone around the 3 axes using the gyroscope and the electromagnetic compass. The user-generated acceleration data in the world coordinate system were collected by the improved application when the phone was carried in the following 3 holding states: 1) in the, 2) holding the phone to the ear, and 3) touching on the screen. The authentication results of preliminary experiment showed that 1.30% false acceptance rate (FAR) at 2.34% false rejection rate (FRR) was obtained when the phone was in the while the FRR was extremely bad as touching on the screen. However, in the preliminary experiment, the acceleration data was recorded for only 4 subjects on one day. Here, there are some questions: how is the precision of gait recognition changed when the number of subject increases or the data recorded for the same subjects on other day are added? How do the walking states, for example, the position of, or left hand, the kind of footwear, and so on, affect gait recognition? In this study, we carry out additional experiments increasing the number of subjects and adding the data recorded about one month later. To examine the influence of various walking states, we also perform other experiments for one subject when he places the phone in the or left pants or shirt, calls or touches in his or left hand, wears slippers or shoes. In the same way as our previous study 4, we extract 43 features from the usergenerated acceleration data in the world coordinate system. And then we determine which user or which state using some classification algorithms from the Weka data mining suite 12. We report the gait recognition results of the additional experiments. Finally, to integrate the identification results from multiple smart phone sensors, we discuss the application of an immunity-based diagnosis model to gait recognition. 2. Acceleration-Based Gait Recognition 2.1. Gait record application In our previous study 4, we improved our ios application for smart phone to record not only the acceleration but also more data such as the rotation of the phone around the 3 axes using the gyroscope and the magnetometer. In this subsection, we explain the gait record application. According to Event Handling Guide for ios 13, the CMDeviceMotion object of the Core Motion uses sensor fusion algorithms to refine the raw accelerometer and gyroscope data and generate information for a device s attitude, its unbiased rotation rate, the direction of gravity on

3 1858 Yuji Watanabe / Procedia Computer Science 60 ( 2015 ) a device, and the user-generated acceleration. We do not need to filter the raw acceleration data because the CMDeviceMotion automatically separates gravity and user-generated acceleration. Additionally, the CMDeviceMotion has enum constants for indicating the reference frames when device motion starts. For example, enum constant CMAttitudeReferenceFrameXMagneticNorthZVertical describes a reference frame in which the Z axis is vertical and the X axis points toward magnetic north in the horizontal plane. In other words, we can get usergenerated acceleration data in the world coordinate system. Therefore, the improved application adopts the CMDeviceMotion and the reference frames to record the logged time, the gravity in unit G, the user-generated acceleration separated from gravity, the rotation rate for each of the three axes in unit radians per second, the magnetic field of each axis in unit micro-tesla, and the three angles (roll, pitch, and yaw) in unit radians. As for data record interval, Kwapisz et al. 7 set the sampling frequency of 50 milliseconds (ms), in other words, they collect about 20 data per second, while Nickel et al. 8 record about 200 data per second. Event Handling Guide for ios 13 mentions we can set the reporting interval to be as small as 10 ms, which corresponds to a 100Hz update rate, so that our application collects approximately 100 data per second. Note that because smart phone simultaneously runs not only our application but also operating system and other applications, the data collection is not perfectly periodic Preprocessing for acceleration We focus on the user-generated acceleration data from all the recorded data set. In each experiment, each subject make one round trip along the corridor with about 50m distance. It takes about one minute. Fig. 1 shows an example of the user-generated acceleration data of the 3 axes in the world coordinate system when a subject walks with smart phone in the pants. In this figure, during about 3 seconds from the beginning of the plot, the subject is putting the phone into the. The subject starts to walk after about 5 seconds. This not walking phase should be removed by a preprocessing before next feature extraction process. By the same reason, the preprocessing also deletes the data before and after turnaround Feature extraction Fig. 1. An example of the user-generated acceleration data of the 3 axes in the world coordinate system. We divide the preprocessed acceleration data for each axis into non-overlapping windows with window size 300. Since acceleration data are collected about 100 times per second, each window has about 3-second interval. The window size in the researches by Kwapisz et al. 7 is 200 for acceleration data which are recorded about 20 times per second, so that the time interval of the window is 10 second. However, it is undesirable to spend such much time on authentication. Therefore, this study shortens the authentication time into 3 second as the same as the study by Nickel et al. 8. In future, we will try to apply 1-second interval which is taken by Gait-ID 9.

4 Yuji Watanabe / Procedia Computer Science 60 ( 2015 ) From the 300 raw accelerometer data of each axis in each window, we extract characteristic behavioral features suitable for user authentication. As the same as the existing studies 3, 4, 7, we extract a total of 43 features which are variations of just 6 basic features. The 43 features are described below with the number of features for each basic feature noted in parentheses, where x i, y i, and z i are i-th acceleration data for each axis in each window: 300 Average for each axis (3): x 1 x / 300 i i 300 i 1 i x 300 x i 1 i x 300 i 1 i i i Standard deviation for each axis (3): ( x )/ 300 Average absolute difference for each axis (3): / 300 Average resultant acceleration (1): x y z / 300 Time between peaks for each axis (3): Time in milliseconds between peaks in the sinusoidal waves associated with most activities. In fact, the first peak is the maximum point among the 300 data, the next peak is the second maximum point, and then at least 3 peaks are found. The times between contiguous peaks are averaged. For example, for the 300 data shown in Fig. 2, the time t 1 between the first peak and the third one and the time t 2 between the third and the second are averaged. Binned distribution for each axis (30): We determine the range of data (maximum - minimum) in the 300 data, and then divide this range into 10 equal sized bins as illustrated in Fig. 2. We record the fraction of the 300 data that fall within each of the bins. Like the study by Nickel et al. 8, instead of feature extraction, Hidden Markov Model or Dynamic Time Warping can be directly applied to raw time series data of acceleration. Deep learning 14 which can automatically acquire features from raw data is also a promising approach. However, these techniques might require high computational resources in smart phone. We will try to compare them in the further work Gait recognition Fig. 2. An example of time between peaks and 10 equal sized bins for given 300 data. To recognition a user or a state, we enter the 43 features of all the users into a classification algorithm which is selected from the Weka data mining suite 12. Kwapisz et al. 7 have used only 2 classification algorithms: decision trees (J48) and Neural Network (NN) from the Weka. In our study 3 which employed all of the 56 classification algorithms in the Weka, we confirmed that Radial Basis Function (RBF) was the best of all the algorithms. Additionally, in our follow study 4, Bayesian Network (BN) and Random Forest (RF) from the Weka also performed

5 1860 Yuji Watanabe / Procedia Computer Science 60 ( 2015 ) better, but decision trees (J48) was the worst for all the case. Therefore, in this study, we employ 4 algorithms, that is, BN, NN, RBF, and RF. For each algorithm, we use the default settings, that is, automatically optimization methods and the ten-fold cross validation. In the ten-fold cross validation, the features of all the windows are randomly partitioned into 10 equal size samples. 9 of the 10 samples are used as training data, and the remaining one is retained as the validation data. The cross-validation process is then repeated 10 times, with each of the 10 samples used exactly once as the validation data. The 10 results can then be averaged to produce a single estimation. To evaluate the recognition performance, we use one metrics: correctly classified rate. 3. Experiments 3.1. Conditions We carry out two types of additional experiments. The first experiments are performed as increasing the number of subjects and adding the data recorded about one month later. The second experiments are conducted to examine the influence of various walking states when a subject places the phone in or left pants or shirt, calls or touches in his or left hand, wears slippers or shoes. The aims of the first and the second experiments are different, that is, the identification of users and states, respectively. For each subject in all the experiments, our gait record application is executed by the same iphone 5 32GB to collect data set, namely, the logged time, the gravity, the user-generated acceleration, the rotation rate, the magnetic field, and the three angles. Each subject in each walking state make one round trip along the corridor with about 50m distance for about one minute. After each subject finishes walking through all the states, the phone is returned to get all the recorded data set into a personal computer through itunes software. In the first experiments, 8 subjects participated on one day, and then 4 subjects among them joined follow-up tests about one month later (we are now going to collect more data from more subjects). They carried the phone in 3 holding states as follows: State 1: in the which contains only the phone. State 2: pretending to call (holding the phone to the ear but not talking). State 3: pretending to touch on the screen (just looking on the screen but not operating the phone). In the second experiments, on one day, a subject attempted 9 different walking states as listed in Table 1. The 9 walking states can be broadly divided into the 3 holding states mentioned above. In the states from 1-1 to 1-4, the locations of the phone are different, and the footwear in the state 1-5 is not slippers but shoes. The phones in the state 2-1 and 3-1 are held by the hand while in the state 2-2 and 3-2 by the left hand. Table 1. 9 different walking states for a subject. State Holding in in in in in calling calling touching touching Location of phone front pants front left pants back pants front left shirt front pants hand left hand hand left hand Footwear slippers slippers Slippers slippers shoes slippers slippers slippers slippers 3.2. Results Our previous study 4 has clarified that when we entered the features from all the 3 holding states of all the subjects to classification algorithm, the subject in different 3 holding states could not be identified as the same person. Fig. 3 illustrates the part of the Z-axis (vertical in the horizontal plane) accelerations for subject A in the 3 holding states. In this figure, we apply raw acceleration data to low pass filter to make it easy to see. From the result, the acceleration data of 3 holding states for the subject are quite different each other. The acceleration of state 1 where the phone is in the periodically oscillates with large amplitude and intense frequency. However, the

6 Yuji Watanabe / Procedia Computer Science 60 ( 2015 ) accelerations of state 2 and 3 where the subject holds the phone to the ear and touches on the screen respectively have small amplitude and low frequency. The reason is probably that the vibration of walking leg can directly transmit to the smart phone in the while the hand carrying the phone suppresses the vibration of walking in state 2 and 3. Fig. 3. Part of the Z-axis accelerations for subject A in the 3 holding states. The raw acceleration data are applied to low pass filter. In the first experiments, we apply each classification algorithm to the features only from each holding state. In future, we plan to automatically separate the holding states by the orientation of the smart phone. Fig. 4 depicts the correctly classified rate (%) of subjects as a function of the number of subjects participated on one day. The results of Fig. 4 (a) and (b) are obtained by Bayesian Network (BN) and Radial Basis Function (RBF), respectively. The remaining 2 algorithms, namely, Neural Network (NN) and Random Forest (RF), have the similar tendency. Generally speaking, in the field of biometrics authentication, identification performance is prone to deteriorate as increasing the number of identified subjects. From the results, although the correctly classified rate in the case of state 1 is not yet affected by the number of subjects, the performance in the cases of state 2 and 3 is influenced by the small number of subjects, only 6. In our previous preliminary experiment for 4 subjects 4, we also observed the deterioration of performance in state 2 and 3 compared with state 1. Fig. 5 depicts the ensemble plots of the Z-axis accelerations for 4 subjects A, B, C, D in (a) state 1: in the and (b) state 3: touching on the screen. We also apply raw acceleration data to low pass filter to make it easy to see. On the one hand, the result of state 1 in Fig. 5 (a) shows that the acceleration data of subjects are different each other, for example, the vibration of subject A is intense while the wave of subject B has a gentle slope. This difference leads to good identification performance in state 1. On the other hand, from the result of state 3 in Fig. 5 (b), the acceleration waveforms of 4 subjects look like similar compared with state 1. This similarity possibly results in the deterioration of performance in state 3. Fig. 4. Correctly classified rate (%) of subjects by 2 algorithms ((a) Bayesian Network, (b) Radial Basis Function) as a function of the number of subjects participated on one day.

7 1862 Yuji Watanabe / Procedia Computer Science 60 ( 2015 ) Fig. 5. Ensemble plots of the Z-axis accelerations for 4 subjects A, B, C, D in (a) state 1: in the and (b) state 3: touching on the screen. The raw acceleration data are applied to low pass filter. Next, Table 2 is the result of correctly classified rate (%) of subjects by 4 classification algorithms when the data recorded for 4 subjects about one month later are added or not. In the table, State 1 means to use only the acceleration data recorded on one day when 8 subjects carried the phone in the, and State 1 + follow-up test data means to use the data logged both on one day for 8 subjects and about one month later for the same 4 subjects. We expect that the additional data on other day may lead to the diversity of subjects gait and cause lower identification performance. The results of state 1 and 2 in Table 2 present the reduction in correctly classified rate for the additional data as we expected. However, it is interesting that the classification rate of the state 3 adding the follow-up data is inversely improved. The reason is yet unclear until we should collect and analyse more acceleration data from many subjects in future. Table 2. Correctly classified rate (%) of subjects by 4 algorithms when the data recorded for 4 subjects about one month later are added or not. Bayesian Network Neural Network Radial Basis Function Random Forest State State 1 + follow-up test data State State 2 + follow-up test data State State 3 + follow-up test data In the second experiments, we apply each classification algorithm to all the features of acceleration data from 9 walking state to decide which state. Table 3 shows correctly classified rate (%) of states by 4 algorithms. From the result in the second row, it is successful to classify the data into 9 states. Furthermore, the result in the third row confirms that to classify into 3 holding states, namely, in the, calling, and touching, achieve better performance. The reason is under investigation. We should also inspect the performance adding the data on other days for the same subject like the first experiments. Table 3. Correctly classified rate (%) of states by 4 algorithms when a subject walks in 9 different states. Bayesian Network Neural Network Radial Basis Function Random Forest To classify into 9 states To classify into 3 holding states

8 Yuji Watanabe / Procedia Computer Science 60 ( 2015 ) Discussion: toward immunity-based gait recognition Generally speaking, an approach to raise the identification accuracy by a single sensor is to use multiple sensors and combine the recognition results from the multiple sensors, that is, multimodal biometrics. In fact, our application can record not only the acceleration of smart phone but also the rotation rate, the magnetic field, and the three angles (roll, pitch, and yaw). The latter sensor values might be useful for gait recognition. After we complete gait recognition for individual sensor on smart phone, we should consider how to combine the identification results from the multiple sensors. We intend to apply an immunity-based diagnosis model to gait recognition using multiple sensors on smart phone. The immunity-based diagnosis model has been proposed by Ishida 15. The diagnostic model, which is a variant of majority vote model, is performed by mutual tests among sensors and dynamic propagation of active states. Each sensor node has the capability of testing the linked nodes, and being tested by the adjacent others as well. Based on the test outcomes, each node calculates its own credibility. We believe the immunity-based diagnosis model can be directly applied to gait recognition using multiple sensors. Fig. 6 schematically shows the application of the immunity-based diagnosis model to multimodal biometrics. Each node, which has its own credibility, is corresponding to single sensor-based identification and final metaidentification. The outcome from sensor-based identification to meta-identification indicated by solid arrows in this figure is identification result. The credibility of meta-identification is the sum of the outcomes weighted by the credibility of single sensor-based identification. If the credibility of meta-identification becomes over a threshold, then it output identified user or state. The test outcome between single sensor nodes indicated by broken arrows will be decided by time, frequent, and accuracy. Like as our study, SenGuard 16 proposed a concept of realizing the user identification on smart phone using multiple sensors, that is, motion, voice, location history, and multi-touch. It tempted to use a majority based meta classifier for the final decision. However, it has not yet clarified detailed sensor fusion technique and results combined by multiple sensors. We intend to differentiate ourselves from SenGuard by introducing the immunitybased diagnosis models to the user identification using multiple smart phone sensors. In addition, a sensor may fall into a temporary abnormality or missing data due to an internal or external change, for example, the magnetic field which is easy to be affected by magnetic substances near the sensor. We believe that the sum of the outcomes weighted by the credibility in the immunity-based diagnosis model makes the final identification more precise. 5. Conclusions Fig. 6. Schematic diagram of the application of the immunity-based diagnosis model to multimodal biometrics. In this study, we carried out the additional experiments increasing the number of subjects and adding the data recorded on other day. The results confirmed that the correctly classified rate when the phone is in the is not yet affected by the number of subjects while the performance in the cases of calling and touching is influenced by the small number of subjects. Interestingly, we observed that the additional data on other day have a different effect on identification performance according to phone holding states. We also examined the influence of various walking

9 1864 Yuji Watanabe / Procedia Computer Science 60 ( 2015 ) states for one subject. Finally, we discussed the application of the immunity-based diagnosis model to gait recognition to integrate the identification results from multiple smart phone sensors. In future, we will collect more gait data from many subjects and then estimate the walking state by the phone orientation to recognize in response to each state using appropriate methods. And then we will achieve the application of immunity-based model to gait recognition using smart phone sensors. Acknowledgements This work was partly supported by a Grant-in-Aid for Scientific Research (C) (15K00188) from the Japan Society for the Promotion of Science and Grant-in-Aid for Research in Nagoya City University. References 1. Watanabe Y, Ichikawa S. Extraction of Operational Behavior for User Identification on Smart Phone. The Seventeenth International Symposium on Artificial Life and Robotics (AROB), 2012, p Watanabe Y, Houryu, Fujita T. Toward Introduction of Immunity-based Model to Continuous Behavior-based User Authentication on Smart Phone. Procedia Computer Science, 22, 2013, p Watanabe Y, Houryu. A Study of Accelerometer-based Authentication on Walking with Smart Phone. International Workshop on Smart Info-Media Systems in Asia (SISA), 2013, p Watanabe Y. Influence of Holding Smart Phone for Acceleration-based Gait Authentication. Proc. of IEEE 2014 International Conference on Emerging Security Technologies (EST), 2014, p Ailisto HJ, Lindholm M, Mäntyjärvi J, Vildjiounaite E, Mäkelä SM. Identifying people from gait pattern with accelerometers. Biometric Technology for Human Identification II, 2005, 5779(1): Gafurov D, Helkala K, Sondrol T. Biometric Gait Authentication Using Accelerometer Sensor. Journal of Computers, 2006, 1(7): Kwapisz JR, Weiss GM, Moore SA. Cell Phone-Based Biometric Identification. Proc. of the 4th IEEE International Conference on Biometrics: Theory Applications and Systems, 2010, pp Nickel C, Busch C, Rangarajan S, Mobius M. Using Hidden Markov Models for Accelerometer-Based Biometric Gait Recognition. IEEE 7th International Colloquium on Signal Processing and its Applications (CSPA), 2011, pp Juefei-Xu F, Bhagavatula C, Jaech A, Prasad U, Savvides M. Gait-ID on the Move: Pace Independent Human Identification Using Cell Phone Accelerometer Dynamics. Proc. of the 5th IEEE International Conference on Biometrics: Theory, Applications and Systems, 2012, pp Primo A, Phoha VV, Kumar R, Serwadda A, Context-Aware Active Authentication Using Smartphone Accelerometer Measurements. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2014, pp Ngo TT, Makihara Y, Nagahara H, Mukaigawa Y, Yagi Y. Orientation-Compensative Signal Registration for Owner Authentication Using an Accelerometer. IEICE Transactions on Information and Systems, 2014, E97-D(3): Witten I, Frank E. Data Mining: Practical Machine Learning Tools and Techniques. Morgan Kaufmann Publishers, Event Handling Guide for ios, 2013/01/28, Hinton, GE, Salakhutdinov R. Reducing the dimensionality of data with neural networks. Science, 313(5786), 2006, pp Ishida Y. Fully Distributed Diagnosis by PDP Learning Algorithm: Towards Immune Network PDP Model. Proc. of IJCNN, 1990, p Shi W, Yang J, Jiang Y, Yang F, Xiong Y. SenGuard: Passive User Identification on Smartphones Using Multiple Sensors. Proc. of IEEE 7th International Conference on Wireless and Mobile Computing, Networking and Communications, 2011, p