Labels Quantified Self App for Human Activity Sensing

Transcription

1 Labels Quantified Self App for Human Activity Sensing Christian Meurisch TU Darmstadt (Telecooperation) Hochschulstraße Darmstadt, Germany Benedikt Schmidt TU Darmstadt (Telecooperation) Hochschulstraße Darmstadt, Germany Michael Scholz TU Darmstadt (Telecooperation) Hochschulstraße Darmstadt, Germany Immanuel Schweizer TU Darmstadt (Telecooperation) Hochschulstraße Darmstadt, Germany Max Mühlhäuser TU Darmstadt (Telecooperation) Hochschulstraße Darmstadt, Germany Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. Ubicomp/ISWC 15 Adjunct, September 07-11, 2015, Osaka, Japan Copyright is held by the owner/author(s). Publication rights licensed to ACM. ACM /15/09...$15.00 DOI: Abstract An in-depth understanding of human activity is essential for building personalized systems like Google Now to support users in everyday life. For that understanding, a comprehensive user-annotated human activity corpus is highly relevant. However, getting a personal dataset for data mining tasks is always a big challenge for researchers. Our approach is Labels, a self-tracking mobile application to collect a human activity corpus with ground-truth data. Providing a user interface users are able to annotate automatically collected sensor data from mobile, desktop and social with valuable metadata (ground-truth), e.g., their performed activities. In this paper, we will present Labels, its functionalities and quality assurance mechanisms for annotated data. We evaluated this app within a four-week field study with 163 participants. The collected dataset with over 43 thousand manual annotated data will also be presented. Author Keywords Personal Tracking, Quantified Self, Human Activity Sensing, Heterogeneous Information Sources, Annotated Dataset ACM Classification Keywords H.1.2 [Models and Principles]: User/Machine Systems

2 Introduction Personal assistance systems like Google Now or Apple s Siri, which support users in their daily lives, become more and more popular. For example, automatic event reminders considering access route and traffic situations are currently suitable for daily use. However, building assistance applications and services, which support the user in a more proactive way and act on user s behalf, requires an even better understanding of human activity. To acquire such knowledge of human behavior researchers need a comprehensive dataset with ground-truth annotations. This always poses a challenge, especially when users need to release and manually annotate their personal data. A big demand for self-tracking systems with minimal user interactions exists. State of the art applications are already able to annotate collected sensor data [18, 7]. However, most approaches only collect datasets focussing on individual research questions [10, 17], which is great to answer them. But investigating the next research question the collection of a new dataset is required again, i.e., building a new tracking app and acquiring participants to conduct a user study. Moreover, in the most cases only a small amount of - mainly physical - sensors are collected [17]. To get an indepth understanding of human activity we consider data from multiple heterogenous data sources as crucial. In this paper, we present our approach, Labels, a selftracking mobile application for collecting human activity sensing corpus. Labels collects and fuses automatically tracked sensor data and user-annotated metadata (groundtruth). Integrated into Kraken.me, our multi-device user tracking suite for mobile, desktop and social [16], personal data are automatically tracked from (1) physical (e.g., accelerometer, location) and (2) virtual sensors (e.g., call log, software interaction) as well as (3) social networks (e.g., friends list). Moreover, Labels provides a user interface with a time slot representation of user s daily routines. These time slots are created on the basis of places and transitions. For that user-supportive feature, we proceed on the assumption that a user performs one activity at one place, e.g., eating in a restaurant, or during one transition, e.g., going by bus. By manually annotating these time slots automatically tracked data are enriched with valuable metadata of human activities. Quality assurance mechanisms support the annotation process to ensure a high quality dataset. In summary and to the best of your knowledge, Labels is the first user self-tracking app that combines a supportive user interface for data annotation with a fully-automated multidevice user tracking suite for mobile, desktop and social. To demonstrate the enormous potential of this app, test it in the field and get a first comprehensive human activity sensing corpus we have conducted a four-week field study with 163 participants. The resulting heterogeneous dataset is checked and filtered in view of data quality. Analyzing this large-scale data collection obtained only by one user study offers researchers a great possibility to answer various research questions regarding human activity at once, e.g., what does the user do and with whom does he interact at particular places? Even, predictive questions, like what will the user do at specific day and time, can be addressed now. The remainder of this paper is organized as follows. First, we introduce Labels, the architecture, its functionalities, the quality assurance mechanisms as well as describe tracking capabilities to get a human activity corpus. Second, we report the study design, the participants and the resulting annotated dataset. After reporting, the study results are described. The paper closes with an overview of related work and conclusion.

3 Quantified Self App Labels is a freely available Android app 1 for collecting annotated human activity corpus. In detail, Labels collects and fuses automatically tracked sensor data and user-annotated metadata (ground-truth). Main challenge by designing and implementing this app was, providing users the possibility to annotate multiple heterogeneous sensor data, which are collected with high frequency, in an easy way. To address this challenge and simplify annotating Labels provides a user interface with a rough time slot representation of user s daily routines. User s task is to annotate appropriate time slots with his related activity by selecting predefined labels or adding custom tags. The predefined labels are customizable by researchers in view of the study purpose. By that annotation process automatically tracked sensor data are provided with crucial ground-truth data necessary for machine learning tasks [1, 12]. Architecture Figure 1 shows the architecture of Labels integrated into the Kraken.me 2 infrastructure, our multi-device user tracking suite for mobile, desktop and social [16]. Mobile apps can simply add the KrakenSDK, our Android library, to integrate into the Kraken.me infrastructure and have access to energy-efficient tracking suite capabilities [15]: (1) physical (e.g., accelerometer, location) and (2) virtual sensor data (e.g., call log, software interaction) as well as (3) social network information (e.g., friends list). Currently, we are supporting seven famous social networks including Facebook, Twitter or LinkedIn. The library also offers authentication through these online social networks as well as communication with the Kraken.me server via API, e.g., login or 1 informatik.tk.studentsecretary.labels 2 Smartphone (Android App) Cloud (Amazon EC2) Web Annotating Labels Tracking KrakenSDK Login + Data upload API Analysis Kraken.me Server Storage Authentication + Crawling API Visualization Kraken.me Dashboard Online Social Networks (e.g., Facebook) Figure 1: Simplified architecture of Labels integrated into the Kraken.me infrastructure automatic data upload. The collected data are first stored on the device and are pushed to the Kraken.me server on a regular basis. JSON is used as data exchange format. The central server hosted on Amazon EC2 Cloud 3 is responsible for storing and processing data as well as crawling social network information (e.g., friends list of a user by requesting Facebook Graph API 4 ). On the webbased Kraken.me dashboard, users are able to view their collected and processed personal data. Functionalities In this section we describe the two main functionalities of Labels: the automatic time slot detection and the annotation process. Time slot detection. Labels supports the user with a rough structure of his daily routines (see Figure 2a). These time slots are created on the basis of user s places and transitions. For that user-supportive feature, we proceed on

4 (a) Main screen (b) Labeling screen (c) Context menu: editing (d) Merging screen (e) Settings screen Figure 2: Graphical user interface of Labels (Android app) the assumption that a user performs one activity at one place, e.g., eating in a restaurant, or during one transition, e.g., going by bus. If users perform more than one activity within a detected place-related time slot, they are able to split it into multiple time slots and vice versa: merging multiple time slots into one time slot. They can also create, edit and remove time slots manually. Places or transitions are revealed by periodically running DBSCAN on user s locations [6]. Whereby, the periodic interval depends on user s movements utilizing physical activities from Google Play Services 5 to save battery. However, these regular calculations of time slots drain the battery faster than our energyefficient sensor data tracking [15]. In the next app version of Labels, we will switch to a more energy-efficient calculation model: on-demand calculations are performed when 5 the user opens the app. As fallback, i.e., the user does not open the app for a long period, we perform periodic calculations with low frequency again. Annotation process. Each time slot has to be annotated manually by users. Figure 2a shows the main screen of the app with daily detected time slots and already assigned labels on the left. By pressing the tiles on the left side of each time slot a popup dialog opens (see Figure 2b). The grey tiles are predefined labels users can select to annotate the related time slot. These predefined labels are configurable in view of the study purpose. Optionally, users can also enter arbitrary custom tags (text input field below the tiles). After storing that metadata and closing the popup the main screen will be refreshed. In addition to automatically detected time slots users are also able to add own time slots

5 by pressing plus action button on the bottom right. Longpress on time slots enables the context menu where users have the option to edit or remove the selected time slot (see Figure 2c). Splitting and merging time slots (see Figure 2d) are assistant functionalities to fit time slots. Finally, Figure 2e shows the setting screen with some control options: server synchronization, daily annotation reminder or detection service controlling. Statistics give the user feedback about his annotation behavior, especially data quality that we will discuss in the next section. Quality assurance Data quality is an important part for user-annotated data. To ensure high quality of data already during user studies, Labels permanently measures data quantity and quality based on metrics reported in the following. Participants are informed about quality and quantity with status bars using traffic lights (see Figure 2e). Data quantity is checked through two metrics: covered time and labels per time slot. Whereby, covered time describes the ratio of sum of all time slot durations to time passed since app installation that is equal to the study start point. Labels per time slots is defined as ratio of count of all annotated time slots to count of all time slots. Status bars are green when the ratio value is greater than 0.85, red in case of smaller than 0.5 and otherwise orange. Data quality is more complex to capture. Nevertheless, we try to define it through tagging behavior, human reaction time and powers of recollection. First, we simply assume that users, who have bothered to add optional custom tags to a time slot, also selected the correct label for this time slot. Second, we always display available labels to users in random order. Based on that we infer data quality from user s reaction time. Utilizing Model Human Processor (MHP) we approximate the minimal time of msec required to perceive (perceptual processor), choose (cognitive processor) and select (motor processor) the correct label [3]. Third and last quality indicator is powers of recollection, i.e., the longer the performed activity is ago, the more difficult to recall a specific activity and reconstruct the daily routines. We empirically assume users are able to perfectly recall the activity up to 12 hours later. After 24 hours the accuracy of recall is not longer sufficient accounted for by forgetting [14]. Between these constraints we linear decrease the resulting quality score. In addition to that, we introduce rule-based selectable labels to further increase data quality, i.e., labels are checked with regard to their plausibility. For example, users are not able to select a transition label, if we detect that they still stay in their home clusters inferring from their sensor data. Human Activity Sensing Corpus The resulting human activity sensing corpus collected by Labels integrated into the Kraken.me infrastructure [16] contains the following heterogeneous data: (1) physical (i.e., accelerometer, location, light, proximity, gyroscope, loudness, wifi scans) and (2) virtual sensor data (i.e., call log, software interaction, calendar, contacts, s, ringtone mode, browsing history, network traffic, battery usage, charging state), and (3) social network information from seven famous providers like Facebook, Twitter or LinkedIn (e.g., friends list, profile, posts, messages, etc.) 6 as well as (4) user-annotated human activity data. Whereby, predefined labels are customizable by researchers in view of the study purpose. This resulting comprehensive dataset can then be used in supervised machine learning approaches [1, 12] to get a better understanding of human behavior. 6 Depending on user s permissions given to Labels we can theoretically crawl all contents from user s social networks.

6 SUS (System Usability Scale) is a simple, ten-item scale to measure subjective usability from a user [2]. UEQ (User Experience Questionnaire) is a 26 item questionnaire for measuring user experience of a product regarding six aspects: Attractiveness, Perspicuity, Efficiency, Dependability, Stimulation, and Novelty [13]. Field Test To test the app in the field as well as get a first comprehensive human activity sensing corpus we have conducted a four-week twenty-four-seven diary study. We have prepared Labels by predefining labels of students daily routines and enabling appropriated sensors for tracking. Moreover, we have evaluated Labels with a questionnaire regarding helpfulness of app features within a diary study, usability (SUS) and user experiences (UEQ) of the app. Participants For the study 195 students of Technische Universität Darmstadt were recruited during winter term 2015, whereby 163 students completed the study and filled out the questionnaire. The demographics of completed participants are as follows: 76% participants are male and 24% are female, 90% Masters students and 10% Bachelors students of computer science. The average age of participants is 25.4 years. The participants stated to have an average technical skills of 3.7 on a 5-point rating scale. Collected User-annotated Dataset The resulting user-annotated dataset collected by Labels is shown in Table 1; center column displays the total collected time slots (52,609), whereby 88.3% were auto- and 11.7% were user-created. Looking at the location context, 73.9% (38,891) were stationary time slots at particular places and 26.1% (13,718) were transitions between places. Among these, 43,393 (82.5%) are annotated and useful. Moreover, 31,998 (60.8%) time slots are also custom tagged. Considering the total number of tags (39,908), tagged time slots are assigned by 1.25 custom tags on average. Applying our strict quality definitions we get a resulting high quality user-annotated human activity sensing corpus (see right column of Table 1). This dataset could be directly used for supervised or unsupervised machine learning approaches Data type Count (raw) Count (filtered) Time slots (total) 52,609 35,416 - auto-created 46,440 32,096 - user-created 6,169 3,320 - stationary 38,891 26,877 - transition 13,718 8,539 Time slots w/ labels 43,393 35,416 - w/ tags 31,998 31,998 Tags (total) 39,908 39,908 Table 1: User-annotated dataset collected by Labels app [1, 12] to get a better understanding of human behavior, e.g., recognition of complex activities or revealing user s routines. Study Results The results in this section are grounded on 163 participants in our four-week twenty-four-seven field study. Participants rated the app with 3.37 ± 0.98 on a 5-point rating scale, which means a slightly positive assessment with a relative high dispersion. In the following, we have a look at different evaluation aspects for an application. System Usability Scale The resulting SUS score over all participants for Labels is ± which means that the usability is okay. Downward outliers are still in the "ok" range, while upward outliers are already in the "excellent" range of usability. User Experience Questionnaire Figure 3a shows the overall results of the questionnaire. All six categories of user experience are consistently positive. Outstanding is the result of 1.71 ± 0.20 for the scale "Perspicuity" that shows it is quite easy to understand how to use the app. All other rating aspects are in the upper neu-

7 A(rac,veness Perspicuity Efficiency Dependability S,mula,on Novelty - 2 Auto detec-on Manual crea-on Merging / spli8ng Labeling Tagging (a) UEQ results (scale -3 to +3) (b) Rating of app features (scale -2 to +2) Figure 3: Study result regarding app ratings tral range: "Attractiveness" is the general positive impression of users towards the app (0.90 ± 0.22). "Efficiency" captures a fast and an efficient app usage by the users (0.79 ± 0.21). "Dependability" result confirms users feel in control of the interactions (0.70 ± 0.20). "Stimulation" result approves a interesting and exciting app (0.69 ± 0.23). Finally, "Novelty" makes a point to innovative and creative app design (0.68 ± 0.19). App features Figure 3b shows the evaluation of particular app features in view of helpfulness and assisting on a 5-point rating scale. All features are consistently positive rated. Outstanding are the merging and splitting features to adjust auto-created time slots (1.02 ± 0.21). These are absolutely required assistant features, because users could perform more than one activity at a single place. As expected, labeling / annotating (0.60 ± 0.18) and tagging (0.59 ± 0.18) are rated as helpful in view of the fact that they are core features of an app like Labels. Whereas, manual creation of time slots (0.49 ± 0.23) is perceived as more helpful as our supportive feature, auto-detection of time slots (0.21 ± 0.19), by the user. Nevertheless, both app features are rated positive. Overall result In summary, it can be stated that users rated the app consistently positive in both user experience, especially usability, and app features. The results show that, on the one hand, the apps is well-implemented and, on the other hand, user benefits from the app and its features in diary studies. Related Work Labels realizes a self-tracking application for human activity sensing. It collects both user-annotated and automatic trackable data from mobile, desktop and social. Whereby, physical and virtual sensors are integrated. In the following we give a brief overview of related work on those aspects. Recently, mobile phones feature an increasing amount of sensors [10] providing the perfect platform for human activity sensing. Simple activity recognition support is now build into most modern operating systems. State of the art approaches detect simple activities, e.g., standing, walking, etc., with an accuracy above 90% relying on physical sensor data only (e.g., accelerometer) [9]. Other approaches utilizing human mobility patterns to better understand human behaviors. Since sequences of place visits encode and reveal, at a coarse resolution, most daily activities [5].

8 Labels also utilizes this assumption of place-related activities for supporting users by the annotation process. However, only using physical sensor data is insufficient to infer more complex activities and get an understanding of human behavior. Lane et al. [11] exploit community behavior and virtual sensors (e.g., phone call, application usage) to further increase results in human activity sensing. Kraken.me, our multi-device user tracking suite [16], combines all available data sources, i.e., physical and virtual sensors from mobile and desktop as well as social network information. Inspired by [4], the social graph is further refined by mining co-location information. State of the art applications are already able to annotate collected sensor data [18]. In [8], the authors introduce the idea of "mission" that is a sequence of selecting activity class and device position as well as performing the activity. They also gathered over 35,000 activity data by a largescale experiment with more than 200 participants over 13 months. However, only sensor data from accelerometer of smartphones are considered. Wang et al. [17] instrument surveys for that purpose. However, these approaches are still limited in the amount of data sources and only collect focused datasets. In [7], the authors even present an annotation system using multi-sensory stream for daily activity. It segments each day only in a few ( 10) meaningful events which the user has to annotate with multiple tags categorized by activity, place and people, e.g., eating in a restaurant with friends. However, the system has only been evaluated by one volunteer. The overview shows that the main contribution of Labels in comparison to the related works is the combination of a full-automated tracking suite for mobile, desktop and social [16] with manual annotation mechanisms of these automated-tracked personal data. Conclusion In this paper, we introduced Labels, a self-tracking mobile application for sensing human activities. Integrated into Kraken.me, our multi-device user tracking suite for mobile, desktop and social [16], Labels collects physical and virtual sensor data as well as social network information. Providing a user interface, users are able to annotate these automatic collected data by their related activity within specific placerelated time slots (ground truth). Moreover, we have evaluated Labels within a four-week field study with 163 participants and collected a dataset with over 43 thousand manual annotated data. The results show that participants annotated about 82.5% of their place-related time slots with their performed activities. While consistently positive ratings of user experience and good app usability (SUS 70), Labels also supports the user by various positive rated features (e.g., time slot detection). The resulting human activity sensing corpus consists of (1) physical (e.g., accelerometer, location) and (2) virtual sensor data (e.g., call log, software interaction), (3) social network information (e.g., friends list) as well as (4) userannotated human activity data. In our future work, we will use this dataset to recognize student s activities (e.g., learning, attending a lecture, sleeping) using supervised machine learning approaches [1, 12]. We will also provide apps for other operating systems (e.g., ios) to cover a wide range of user devices. Moreover, we will enhance and use our tracking tools in field studies for getting an in-depth understanding of human behavior. We plan to share our tools and cooperate with the community. Acknowledgements This work has been funded by the LOEWE initiative (Hessen, Germany) within the NICER project.

9 REFERENCES 1. Christopher M Bishop Pattern Recognition and Machine Learning. springer. 2. John Brooke SUS - A Quick and Dirty Usability Scale. Usability Evaluation in Industry 189, 194 (1996), Stuart K Card, Thomas P Moran, and Allen Newell The Model Human Processor: An Engineering Model of Human Performance. Handbook of Human Perception 2 (1986). 4. Justin Cranshaw, Eran Toch, Jason Hong, Aniket Kittur, and Norman Sadeh Bridging the Gap Between Physical Location and Online Social Networks. In 12th Intl. Conf. on Ubiquitous Computing (UbiComp 10). ACM, Trinh-Minh-Tri Do and Daniel Gatica-Perez The Places of Our Lives: Visiting Patterns and Automatic Labeling from Longitudinal Smartphone Data. Technical Report. 6. Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu A Density-based Algorithm for Discovering Clusters in Large Spatial Databases with Noise. In 2nd Intl. Conf. on Knowledge Discovery and Data Mining (KDD 96). ACM, Jihun Hamm, Benjamin Stone, Mikhail Belkin, and Simon Dennis Automatic Annotation of Daily Activity from Smartphone-based Multisensory Streams. In Mobile Computing, Applications, and Services. Springer, Yuichi Hattori, Sozo Inoue, and Go Hirakawa A Large Scale Gathering System for Activity Data with Mobile Sensors. In 15th Intl. Symp. on Wearable Computers (ISWC 11). IEEE, Jennifer R Kwapisz, Gary M Weiss, and Samuel A Moore Activity Recognition Using Cell Phone Accelerometers. ACM SigKDD Explorations Newsletter 12, 2 (2011), Nicholas D Lane, Emiliano Miluzzo, Hong Lu, Daniel Peebles, Tanzeem Choudhury, and Andrew T Campbell A Survey of Mobile Phone Sensing. Communications Magazine, IEEE 48, 9 (2010), Nicholas D Lane, Li Pengyu, Lin Zhou, and Feng Zhao Connecting Personal-scale Sensing and Networked Community Behavior to Infer Human Activities. In 16th Intl. Conf. on Pervasive and Ubiquitous Computing (UbiComp 14). ACM, Oscar D Lara and Miguel A Labrador A Survey on Human Activity Recognition Using Wearable Sensors. Communications Surveys & Tutorials, IEEE 15, 3 (2013), Bettina Laugwitz, Theo Held, and Martin Schrepp Construction and Evaluation of a User Experience Questionnaire. Springer. 14. Gertrude Raffel The Effect of Recall on Forgetting. Journal of Experimental Psychology 17, 6 (1934), Immanuel Schweizer, Roman Bärtl, Benedikt Schmidt, Fabian Kaup, and Max Mühlhäuser Kraken.me Mobile: The Energy Footprint of Mobile Tracking. In 6th Intl. Conf. on Mobile Computing, Applications and Services (MobiCASE 14). IEEE,

10 16. Immanuel Schweizer and Benedikt Schmidt Kraken.me: Multi-device User Tracking Suite. In 16th Intl. Conf. on Pervasive and Ubiquitous Computing (UbiComp 14): Adjunct Publication. ACM, Rui Wang, Fanglin Chen, Zhenyu Chen, Tianxing Li, Gabriella Harari, Stefanie Tignor, Xia Zhou, Dror Ben-Zeev, and Andrew T Campbell StudentLife: Assessing Mental Health, Academic Performance and Behavioral Trends of College Students Using Smartphones. In 16th Intl. Conf. on Pervasive and Ubiquitous Computing (UbiComp 14). ACM, Lijuan Marissa Zhou, Cathal Gurrin, and Zhengwei Qiu Zhiwo: Activity Tagging and Recognition System for Personal Lifelogs. In 3rd Intl. Conf. on Multimedia Retrieval (ICMR 15). ACM,