Recognition of User Activity for User Interface on a Mobile Device

Transcription

1 Recognition o User Activit oruser Interace on a mobile device Recognition o User Activit or User Interace on a Mobile Device Jonghun Baek Dept. o Inormation and Communication Kungpook National Universit, Daegu, South Korea Sang Tae Kim School o Electrical Engineering and Computer Science Kungpook National Universit, Daegu, South Korea Hun Deok Kim School o Electrical Engineering and Computer Science Kungpook National Universit, Daegu, South Korea Jae-Soo Cho School o Internet-Media Engineering Korea Universit o Technolog and Education, Cheonan, South Korea Boung-Ju Yun School o Electrical Engineering and Computer Science Kungpook National Universit, Daegu, South Korea aqb@korea.com Abstract This paper presents a novel user interace that recognizes a user phsical activit to supplement the lacking user interace o mobile devices. The interaces o mobile applications are oten un-natural due to limited resources and miniaturized input/output controls. In particular, users ma encounter such problem in applications like elearning. Thereore, we developed novel user interaction to complement the poor user interace o mobile devices when a student studies on it through the network.. Introduction Advances in mobile computing and MEMS (Micro-Electro-Mechanical Sstems) enable us to embed various sensors into handheld device such as cellular phones, PDAs (Personal Digital Assistants), PMP (Portable Media Plaer), and other portable devices. Thereore, contetual inormation can be captured through various sensors and recognition technologies. Speciicall, an accelerometer can be easil embedded in a handheld device because an accelerometer has advantages such as being lightweight, small, and inepensive. For this reason an accelerometer is one o the most widel used sensors in diverse areas, such as counting steps [], medical engineering [], novel user interace [3], and contet awareness computing []. Man new applications or an accelerometer are now being considered b handheld device designers, and in act handheld device models with an accelerometer are Proceedings o the th South East Asia Regional Computer Conerence, November 8-9, 7, Bangkok, Thailand.

2 Jonghun Baek, Sang Tae Kim, Hun Deok Kim, Jae-Soo Cho, and Boung-Ju Yun available in the market. A handheld device, though its computing power is increasing, provides a limited computing environment when compared with a PC or an embedded sstem with a digital signal processor. In addition, a handheld device cannot dedicate its ull computing power to auiliar applications when its primar role is a communications device. In particular, an accelerometer application to estimate user activit estimation, which should run all the time, cannot demand too much CPU time. This is not onl a matter o saving computing power but also a matter o preserving the limited batter power in a handheld device. Thereore, it is beneicial to develop a light-weight signal processing algorithm in order to estimate user s activit. eicientl estimated the user activit in various positions b using it. A series o eperiment or testing the eectiveness o the proposed method have been perormed and the results are presented. In this paper, the aim is to implement the accelerometer application such as an intelligent multimedia control interace (e.g., static: audio/video, walking: audio, and running: pause) or a handheld device. Such the accelerometer application can be used in ubiquitous learning environments that provide education inormation at the right time b estimating activit o a user (student) who uses a handheld device. We, thereore, tr to estimate i the user is in a static, walking, or running states. Man research groups have tried to estimate a user s activit as contet inormation. Most studies use a signal accelerometer attached to the speciic part o a user s bod (e.g., waist, wrist, thigh, ankle, and spine) or multiple accelerometers [5]- []. This might, however, make it diicult to estimate the user s activit as well as inconveniencing a user in real lie, because an accelerometer is ied on part o the bod []. In contrast, this paper attempts to minimize inconveniencing the user and to create a more practical, real lie, implementation. An accelerometer was embedded in a handheld device and the various positions in which it can be carried b a user in their dail lie: ront pocket, hip pocket, shirt pocket, backpack, waist, handhold, and so on, were considered. In this case, the gravit acceleration component o an accelerometer s signal varies with the handheld s position and orientation and thus the user activit cannot be easil estimated. In an eort to design a light-weight and eective signal processing method or eliminating the gravit acceleration, we ound that a second-order Butterworth highpass ilter (SHPF) was appropriate and =g, =-g =g, =-g =g, =g =g, =g =g, =-g =.8g, =-.7g =.5g, =.9g (a) (b) (c) (d) (e) () Fig.. Data collection protocol: (a) ront pocket, (b) hip pocket, (c) waist, (d) shirt pocket, (e) backpack, () handhold in the standing and walking, running, respectivel.. Data collection protocol Various positions and orientations o an accelerometer are shown in Fig.. A dierent initial orientation o the accelerometer in each position was used to assess the usabilit o the SHPF or estimating the user activit, regardless o the orientation o the accelerometer or individual s characteristics. The training data set is collected rom a single run o accelerometer data, which is a two-dimensional time series collected or about minute rom subjects ( males, age 5 to 3) at a sampling rate o samples/s rom the accelerometer when it is in three activities, such as standing, walking, and running, o a user at the various positions o an accelerometer. Special Issue o the International Journal o the Computer, the Internet and Management, Vol.5 No. SP, November, 7.

3 Recognition o User Activit oruser Interace on a mobile device Fig. shows the initial directions o an accelerometer at the various positions o a handheld device in three activities o a user. The numbers in Fig. are the average o the output signal or X- and Y-ais o the accelerometer measured during minute or each orientation o an accelerometer in three activities o a user. 3. Acceleration time-series normalization The eisting method or estimating a user s activit is as ollows: a eature etractor calculates a set o statistics o the data, which in turn orms a eature vector to be input to a classiier. The output o the classiier is one o the labeled classes, as shown in Fig.. Accelerometer data Feature etractor Classiier Labeled classes Standing Sitting Ling Upstairs Downstairs so on Fig.. Traditional signal processing steps or estimating a user activit. The selection o good eatures rom time series data is a undamental step in statistical pattern recognition and is a ver important task, because i the selected eatures are etracted based on individual s characteristics and individual s speciic situation, the addition o a new person would necessitate more training b designing a new model or b etracting a new eature. In this stud, the orientations o an accelerometer are dierent rom each other in various positions o an accelerometer and an individual s characteristics such as bod, clothe, and bag. In addition, the orientation o an accelerometer is continuousl changing while walking and running. For these reasons, the gravit acceleration component o the accelerometer s signal is also changing and we cannot appl it to the same eature in all positions and orientations o a handheld device. In order to minimize these eects, the acceleration time-series is normalized b the SHPF. An accelerometer signal, a, can be written as, a = a g + am, where, a g and am are the gravit and motion acceleration components, respectivel. In the dnamic states, we deine gravit acceleration, which is a low-requenc signal, as noise (DC signal) because gravit acceleration is not the same in all positions and orientations o a handheld device. In other words, to classi the user s activit at the various positions o a handheld device, we should implement a ilter which completel eliminates the noise component and has no eect on the motion acceleration component. To develop a light-weight signal processing algorithm or estimating user activit on a handheld device, we used a SHPF and investigated its requenc characteristics. Fig. 3a shows the requenc response o the SHPF, and Fig. 3b shows the pole-zero plot in the z-plane. We observed the requenc response b moving the poles. The requenc response curve had its peak value at a speciic requenc component when the pole vale was a comple number. I the pole value was a real number and the pole was moved to the let hal plane in the z-plane, then the gain o the SHPF was decreased. For these reasons, the SHPF had an eect on the motion acceleration component and the noise was also not eliminated. However, when the pole was moved to right plane, the skirt characteristic o the SHPF was better and had a maimum lat response at the pass band. Thus, the SHPF can eliminate onl the gravit acceleration component (see Fig. 3). We selected p = p =. 95 as the value or both the poles. These vales were selected through the eperimentation or the threshold analses to classi a user s activit. Fig. 3b Proceedings o the th South East Asia Regional Computer Conerence, November 8-9, 7, Bangkok, Thailand.3

4 Jonghun Baek, Sang Tae Kim, Hun Deok Kim, Jae-Soo Cho, and Boung-Ju Yun shows all poles inside the unit circle and thereore, implies a stable ilter. H(ω).8.6. p =.7, p =.3 p =p =.. p =p =.95 p =p = ω Imaginar Part z =z = p =p = Real Part (a) (b) Fig. 3. Characteristics o the SHPF: (a) requenc response curve and (b) pole-zero plot on the z-plane. The transer unction and gain (K) the SHPF are given b: H j ( ) ( z z )( z z ) z e = K K = ( z p )( z p ) ( + p )( + p ) o = ω, () This transer unction is transormed the ollowing recurrence equation: [] n = ( p + p ) a [ n ] p pa [ n + K( a[] n a[ n + a[ n ] ) a ] () where, a is the raw signal o the accelerometer and a is the signal ater iltering. An eperiment was conducted to assess the usabilit o the SHPF b equation (). Fig. contains the training data set and shows the eperiments results or the representative 3 o 8 contets (3 activit levels o a user 6 position levels o a handheld device). In the Fig., all the acceleration time-series data iltered b the SHPF were distributed at near zero g in all the contets. Thereore, the SHPF eliminated onl the gravit acceleration component and ver nearl had no eect on the motion acceleration component in all the contets. As a result, the acceleration time-series was normalized b the SHPF, and the three activities o a user can be classiied b their handheld device. Raw Signal Filtered Signal Standing gravit acceleration - 6 Acceleration(g) Number o samples Fig.. Raw signal and iltered signal or Y-ais o the accelerometer in the ront pocket.. User activit estimation Fig. 5 shows the proposed algorithm in order to estimate a user s activit in various contets. We investigated the parameters, such as the estimation time and overlapping proportion o acceleration time-series data or the window lengths, or classiing the activit using the training data set. All acceleration eatures were computed on windows o,, 3,, and 5 samples. The windows corresponded to,, 3,, and 5 seconds, respectivel. As a result o the eperiments, the perormance o the classiication was ound to be relativel insensitive to variations in time interval. For the overlapping proportion (%, 5%, 5%, and 75%) at all estimation time, the nonoverlapping was easier to classi, thereore, we selected the window size o samples. To ind generic threshold values, a series o Static YES Raw signals Accelerometer X-ais Magnitude Average Standard deviation Y-ais NO signals ater iltering nd order High-pass ilter Dnamic Magnitude Fig. 5. Raw signal and iltered signal or Y-ais o the accelerometer in the ront pocket. threshold analses tests were done b using the training data set. From the results o the threshold analses, the thresholds or dierentiating among static, walking and running activities were ound, and annotated on the plot in Fig. 6. YES Average NO Special Issue o the International Journal o the Computer, the Internet and Management, Vol.5 No. SP, November, 7.

5 Recognition o User Activit oruser Interace on a mobile device Standard Deviation Dnamic zone Static zone Time Ten subjects participated in evaluating the classiication accurac o the generic threshold values ound b the training data set. Two methods were used to collect the test data. The irst method was using the data collection protocol. The second method was b reel carring a handheld device in various positions and orientations. Each method had 5 test subjects ( males, age to 35). A subject conducted each activit in various contets or minute in a linear hallwa. The eperimental results are summarized in Table. There are two numbers in parentheses; the irst number is the estimated number o times b the proposed algorithm whereas the second number is actual number o times observed b the supervisor. The results were good enough to demonstrate the eectiveness o the SHPF to estimate user activit in various contets. 5. Conclusion :, O:. zone E F G A B C D H I J K L Time To estimate a user s activit in real lie, we attached an accelerometer to a handheld device and considered the various positions in which a user could carr it. In an eort to implement a user activit estimation sstem, we studied the method normalizing the acceleration time-series rom an Mean zone :, O: Fig. 6. Thresholds or classiing activit state: (a) distinguish static rom dnamic states and (b) distinguish walking rom running. Smbols in the igure: A and G or ront pocket, B and H or hip pocket, C and I or shirt pocket, D and J or waist, E and K or backpack, F and L or handhold in walking and running, respectivel. accelerometer and ound that the SHPF was appropriate, and developed the user activit estimation sstem on the PDA which is usable in real lie. The proposed algorithm, which is ver light-weight, can estimate the user s activit regardless o the orientation o the accelerometer or dierent test user s and is shown to be as eective as a tried-and-true method. TABLE I Classiication results: results rom using data collection protocol and reel carring a handheld device in various contets. Subject Data Collection Protocol Freel Carring (%) Averag (%) (%) A B C D E F G H I J Static S S S3 S S5 S6 S S S S S S Activit Smbols in the table: S or ront pocket, S or hip pocket, S3 or shirt pocket, S or backpack, S6 or handhold, A ~ J or subjects participated in this stud, respectivel. Reerence [] Lee, G., and Baek, J. (6): Counting steps using impact oscillator, Electronics Letters,, (7), pp [] Bazzarelli, M., et al.. (3): A wearable computer or phsiotherapeutic scoliosis treatment, IEEE Transactions on Instrumentation and measurement, 5, (), pp. 6 9 [3] Baek, J., Jang, I., and Yun, B (6): Recognizing and analzing o user s continuous action in mobile sstems, IEICE Transactions on Inormation and Sstem, E89-D, (), pp Proceedings o the th South East Asia Regional Computer Conerence, November 8-9, 7, Bangkok, Thailand.5

6 Jonghun Baek, Sang Tae Kim, Hun Deok Kim, Jae-Soo Cho, and Boung-Ju Yun [] Siewiorek, D., et al. (5): SenSa: a contet-aware mobile phone, Proc. o IEEE Int. Sm. Wearable Computers, pp. 8 9 [5] Parkka, J., et al. (6) : Activit classiication using realistic data rom wearable sensors, IEEE Transactions on Inormation Technolog in biomedicine, 53, (), pp [6] Bao, L., and Intille, S. S. (): Activit recognition rom userannotated acceleration data, Proc. Int. Con. Pervasive Computing, LNCS 3, pp. - [7] Mantjarvi, J., Himberg, J., and Seppanen, T. (): Recognizing human motion with multiple acceleration sensors, Proc. o the IEEE Int. Con. Sstems, Man, and Cbernetics,, pp [8] Lee, S. W., and Mase, K (): Activit and location recognition using wearable sensors, IEEE Pervasive Computing,, (3), pp. -3 [9] Baek, J., et al. (): Accelerometer signal processing or user activit detection, Knowledge-Based Intelligent Inormation and Engineering Sstems, LNAI 35, pp [] Farringdon, J., et al. (999): Wearable sensor badge and sensor jacket or contet awareness, Proc. Int. Sm. Wearable Computers, pp. 7-3 [] Yi, J. S., et al. (5): Contet awareness via a single device-attached accelerometer during mobile computing, Proc. MobileHCI 5, pp Special Issue o the International Journal o the Computer, the Internet and Management, Vol.5 No. SP, November, 7.6