A Spatiotemporal Approach for Social Situation Recognition

Transcription

1 A Spatiotemporal Approach for Social Situation Recognition Christian Meurisch, Tahir Hussain, Artur Gogel, Benedikt Schmidt, Immanuel Schweizer, Max Mühlhäuser Telecooperation Lab, TU Darmstadt

2 MOTIVATION Personal Assistance Systems (e.g., Google Now) Human Behavior Recognition / Characterization Personal Data This talk: Social Situation Recognition from Spatiotemporal Data 2

3 CHALLENGES Need of various data sources e.g., mobile sensor data, web data Fusion and processing of heterogenous sensor data Instrumentation Some approaches need additional (static) hardware e.g., special wearables, smart home sensors Dataset Willingness of users to collect and share their personal data Continuous tracking of data 3

4 (SOCIAL) SITUATION User s situation is represented by place, time, environment, social presence and (complex) activity e.g., usera trains with userb in gym on Monday 6 p.m. Goal Reducing complex recognition problem and required data Guidelines Using as few as possible data Only considering highly available device (e.g., smartphones) This talk (work in-progress showcase) Recognition of social situation only from mobility pattern (i.e., locations) 4

5 CONTRIBUTIONS Our Approach Social situation recognition from spatiotemporal data Self-tracking Dataset 163 students, 4 weeks (24/7), 24M locations Social Situation Recognizer Random forest 100 trees, /- 23.3% F1-measure 5

6 CONTRIBUTIONS Our Approach Social situation recognition from spatiotemporal data Self-tracking Dataset 163 students, 4 weeks (24/7), 24M locations Social Situation Recognizer Random forest 100 trees, /- 23.3% F1-measure 6

7 OUR APPROACH Situation recognition only with mobility patterns (i.e., locations) Data model social spatial temporal We assume that one information type can be inferred by using the two others 7

8 OUR APPROACH Situation recognition only with mobility patterns (i.e., locations) Data model social spatial temporal This talk We assume that one information type can be inferred by using the two others 8

9 OUR APPROACH Low-level spatial context High-level spatial context High-level spatialtemporal context High-level spatialtemporal social context User u1 Location traces l1 Places p1 Place visits v1 User u2 User u3 Location traces l2 Location traces l3 Places p2 Places p3 Place visits v2 Place visits v3 Social interactions i Social situation s User un Location traces ln Places pn Place visits vn Features Class label 9

10 OUR APPROACH Place and Place Visits Place stationary geographical location where a user stays for an amount of time place detection [kang2005] distance threshold = 25m temporal threshold = 15min four meaningful places [zhou2007] home, workplace, university, other place Place visits when and how long a user visits his specific places i.e, add temporal aspect: date/time + duration 10

11 OUR APPROACH Social Interactions Meeting of at least two persons who are temporal co-located over a specific amount of time > 5min = granularity of our data model Detection temporal and spatial overlapping of users place visits using DBSCAN (minpts=2, eps=20m) as clustering algorithm Limitations only instrumented people are considered central processing or personal data from others are required hard to solve :( 11

12 OUR APPROACH Social Situation Recognition A social situation in our approach is represented by social, place and time. The approach recognizes social interactions by place and time Outlook Utilizing well-researched next-place detection place (and time) is foreseen, thus, it predicts the exact (!) user group a user will meet 12

13 CONTRIBUTIONS Our Approach Social situation recognition from spatiotemporal data Self-tracking Dataset 163 students, 4 weeks (24/7), 24M locations Social Situation Recognizer Random forest 100 trees, /- 23.3% F1-measure 13

14 CONTRIBUTIONS Our Approach Social situation recognition from spatiotemporal data Self-tracking Dataset 163 students, 4 weeks (24/7), 24M locations Social Situation Recognizer Random forest 100 trees, /- 23.3% F1-measure Proof of concept 14

15 SELF-TRACKING STUDY 195 students of TU Darmstadt were recruited 163 students completed the four-week study Gender: 76% male, 24% female Age: ø 25.4 years Course of study: 90% Master, 10% Bachelor Practical phase of seminar course Requirements: Android 4.x phone, Facebook account ~16,4 % dropout rate # Quantified Self 24/7 over 4 weeks " Questionnaires ! Labels - Quantified Self App for Human Activity Sensing Christian Meurisch { Telecooperation, TU Darmstadt } 15

16 Kraken.me SELF-TRACKING STUDY User study tools Multi-device user tracking suite [1] mobile (Android, ios) desktop (Windows, Linux, Mac) social (Facebook, Twitter, ) Sensors [2] physical (e.g., acceleration, location) virtual (e.g., calendar, call logs) social connector (e.g., friends list) Automatic tracking of sensor data [1] UbiComp 14 Workshop: HASCA [2] MobiCase 14! Labels - Quantified Self App for Human Activity Sensing Christian Meurisch { Telecooperation, TU Darmstadt } 16

17 SELF-TRACKING STUDY Quantified Self Dataset TIME PERIOD PARTICIPANTS SENSOR DATA 28d M! Labels - Quantified Self App for Human Activity Sensing Christian Meurisch { Telecooperation, TU Darmstadt } 17

18 SELF-TRACKING STUDY Quantified Self Dataset 24/7 self-tracking over 4 weeks ( ) TIME PERIOD PARTICIPANTS SENSOR DATA 28d M! Labels - Quantified Self App for Human Activity Sensing Christian Meurisch { Telecooperation, TU Darmstadt } 18

19 SELF-TRACKING STUDY Quantified Self Dataset Students of TU Darmstadt TIME PERIOD PARTICIPANTS SENSOR DATA 28d M! Labels - Quantified Self App for Human Activity Sensing Christian Meurisch { Telecooperation, TU Darmstadt } 19

20 SELF-TRACKING STUDY Quantified Self Dataset Sensor type Sensor data Sampling method (trigger) Count Physical sensor Virtual sensor TIME PERIOD PARTICIPANTS SENSOR DATA 28d M Acceleration Periodic (system) 15,655,893 Location Periodic (system) 24,679,844 Light On changes (system) 148,382,777 Network connections On changes (system) 1,120,029 Ringtone mode Periodic (manual) 9,005,776 Browsing history Periodic (manual) 3,060,571 Application usage On changes (system) 4,832,596 Call logs On changes (system) 910,614 Calendar events On changes (system) 78,559 Phone contacts On changes (system) 68,470 Google activities Periodic / on changes (system) ( 69,236,646 Total 277,031,775 Social Networks Facebook Twitter Google+ Foursquare Microsoft Live LinkedIn Xing Additional sensors Power status Network traffic WiFi Scans Psychological data BFI-44 (Personality) SSI-90 (Social Skills)! Labels - Quantified Self App for Human Activity Sensing Christian Meurisch { Telecooperation, TU Darmstadt } 20

21 SELF-TRACKING STUDY Quantified Self Dataset Sensor type Sensor data Sampling method (trigger) Count Physical sensor Virtual sensor TIME PERIOD PARTICIPANTS SENSOR DATA 28d M Acceleration Periodic (system) 15,655,893 Location Periodic (system) 24,679,844 Light On changes (system) 148,382,777 Network connections On changes (system) 1,120,029 Ringtone mode Periodic (manual) 9,005,776 Browsing history Periodic (manual) 3,060,571 Application usage On changes (system) 4,832,596 Call logs On changes (system) 910,614 Calendar events On changes (system) 78,559 Phone contacts On changes (system) 68,470 Google activities Periodic / on changes (system) ( 69,236,646 Total 277,031,775 Social Networks Facebook Twitter Google+ Foursquare Microsoft Live LinkedIn Xing Additional sensors Power status Network traffic WiFi Scans Psychological data BFI-44 (Personality) SSI-90 (Social Skills)! Labels - Quantified Self App for Human Activity Sensing Christian Meurisch { Telecooperation, TU Darmstadt } 21

22 CONTRIBUTIONS Our Approach Social situation recognition from spatiotemporal data Self-tracking Dataset 163 students, 4 weeks (24/7), 24M locations Social Situation Recognizer Random forest 100 trees, /- 23.3% F1-measure Proof of concept 22

23 CONTRIBUTIONS Our Approach Social situation recognition from spatiotemporal data Self-tracking Dataset 163 students, 4 weeks (24/7), 24M locations Social Situation Recognizer Random forest 100 trees, /- 23.3% F1-measure Proof of concept 23

24 SOCIAL SITUATION RECOGNIZER Feature Extraction 24

25 SOCIAL SITUATION RECOGNIZER Classifier Features 59,412 instances, i.e., / per user Personalized classifiers for each user, i.e., we trained 163 classifier Supervised machine learning 26 different algorithms w/ various configurations 10-fold cross-validation using WEKA (data mining software) Random Forest w/ 100 trees 25

26 SOCIAL SITUATION RECOGNIZER Preliminary Results Personalized classifier Average classifier F 1 measure [%] Users In total, /- 23.2% F1-measure 26

27 SOCIAL SITUATION RECOGNIZER Preliminary Results Personalized classifier Average classifier F 1 measure [%] % Users 18% of users ranging from 70% to 99.3% F1-measure 27

28 SOCIAL SITUATION RECOGNIZER Preliminary Results Personalized classifier Average classifier F 1 measure [%] % Users For over 38% of users we can correctly detect exact social context in every second situation 28

29 SOCIAL SITUATION RECOGNIZER Preliminary Results Personalized classifier Average classifier F 1 measure [%] % Users However, it is quite challenging for the rest of users 29

30 DISCUSSION & FUTURE WORKS Social interactions Exact (!) users were detected/predicted Future works Considering sub-groups for detection Assigning unique persons a probability of attendance Overlapping of location traces for non-stationary detection Supervised machine learning Conventional classifiers were used Future works Using deep learning approaches Other possible recognition use cases Place detection (from social and temporal) User routines (from social and spatial) 30

31 SUMMARY & CONCLUSIONS Motivation Building smart personal assistance systems (e.g., Google Now) Knowledge about human behavior is essential Results in complex recognition tasks e.g., required data sources, instrumentations, models, Our approach Goal: reducing complexity of situation recognition Social situation recognition only from location traces Generic data model (social, temporal, spatial) Results 4-week self-tracking data set with 163 students collected over 24M locations Recognition of exact (!) social context by /- 23.3% F1-measure 31

32 $ THE END Thanks for Your Attention!