Wearable Sensing for Understanding, Forecasting and Assisting Human Activity. Kris Kitani Assistant Research Professor Carnegie Mellon University

Transcription

1 Wearable Sensing for Understanding, Forecasting and Assisting Human Activity Kris Kitani Assistant Research Professor Carnegie Mellon University

2 camera Wearable sensing camera smartphone

3 camera Wearable sensing camera smartphone

4 What s so unique about wearable cameras?

5 Table' tennis Features of FPV Near-View Environment Chat Intimate Social Sensing Chat traditional view Chat near-view Closer view of the environment (especially hands-object interactions) Chat Up close observations of interactions (faces, gestures, eye contact, touch) True Visual Experience On-the-Go Direct access to a person s visual experience (saliency, visual motifs) Platform is mobile (large scale visual analysis)

6 Research Directions Object Recognition Action Recognition Video Summarization [Piriavash, Ramanan CVPR 2012] [Fathi, Faharadi, Rehg, ICCV 2011] [Lee, Gnosh, Grauman. CVPR 2012] Social Saliency Gaze Analysis Privacy Preservation [Park, Jain, Sheikh. NIPS 2012] [Li, Fathi, Rehg. ICCV 2013] [Hoyle et al. UBICOMP 2014] Activity Prediction Social Interactions Affordance Estimation [Ryoo et al. HRI 2015] [Fathi, Hodgins, Rehg. CVPR 2012] [Rhinehart, Kitani. CVPR 2016]

7 wearable sensing for Human Activity Understanding Forecasting Assisting

8 Learning Action Maps of Large Environments via First-Person Vision Nicholas Rhinehart, Kris M. Kitani. Learning Action Maps of Large Environments via First-Person Vision. Conference on Computer Vision and Pattern Recognition (CVPR), 2016.

9 When we observe a scene we know how we can act in that environment

10 When we observe a scene walk walk we know how we can act in that environment

11 When we observe a scene sit sit sit we know how we can act in that environment

12 When we observe a scene open open open we know how we can act in that environment

13 Can we teach a computer to understand scene functionality?

14 Input: Captured visual experience

15 Places to Open Door Point Cloud Localization

16 Activity Detections In one or more scenes Scene 1 Detections Scenes Scene 1 Scene 2 Regularized WNMF Modeling and Matrix Completion Cross-location similarities Appearance Info Object Detection Features Actions Locations Scene 1 Sit Action Map Action Maps Scene 2 Sit Action Map Scene Classification Features

17 Places to Sit Point Cloud

18 Understanding wearable sensing for Human Activity Forecasting Assisting

19 First-Person Activity Forecasting with Online Inverse Reinforcement Learning Nicholas Rhinehart, Kris M. Kitani The Robotics Institute, Carnegie-Mellon University Best Paper Honorable Mention

20 Can we teach a computer to predict the future? (using a wearable camera)

21

22

23 Continuously predict future goals using first-person video

24 1. Track state with SLAM and activity detection

25 1. Track state with SLAM and activity detection 2. Discover goals with stop-detection Living room Dining room Bathroom Bedroom Exit Kitchen Office

26 1. Track state with SLAM and activity detection 2. Discover goals with stop-detection 3. Forecast future goals Dining room Living room Bathroom Bedroom Exit Kitchen Office

27 Understanding Forecasting wearable sensing for Human Activity Assisting

28 Wearable Sensing via Smartphones for Assisting Indoor Navigation for Blind People

29 Our Vision

30 NavCog: Turn-by-turn navigational instruction

31

32 Turn-by-turn Navigation Providing voice & visual instructions at each corner Re-routing capability Similar to car navigation systems High-precision localization via Bluetooth beacon network and particle filtering Turn right. Approaching. Sound and vibratio 9 meters ahead, turn right.

33 Newel Simon Hall: 2 floors, 6800m2 120 beacons, 200 poi, 156 destinations Installation type: manual Installation time: weeks

34 Gate Hillman Center: 6 floors, 22, 000m2 450 beacons, 487 poi, 396 destinations Installation type: crowdsourcing (beacons), cobot (fingerprints) Installation time: 1 week (beacons) + 2 days (fingerprints)

35 Wean Hall: 8 floors, 30, 000m2 300 beacons, 668 poi, 626 destinations Installation type: admin app (beacons), lidar (fingerprints) Installation time: 1 day (beacons) + 1 day (fingerprints)

36 Coredo Muromachi: 3 buildings, 4 floors, 21, 000m2 218 beacons, 200 POI, 92 destinations Installation type: manual* (beacons), lidar (fingerprints) Installation time: 2 weeks

37 Greentree DoubleTree Hotel 2017

38 Pittsburgh International Airport 2018

39 Allegheny General Hospital 2018

40 Nihonbashi Muromachi Pilot (Feb. 1 Feb. 28, 2017) (21,000m 2 ) Target users: Blind Wheelchair Foreign visitors Source: MITSUI FUDOSAN ( Mitsui Shopping Park Urban ( IBM Corporation

41 Conversational Concierge + Navigation Current location Conversational restaurant/shop search Route Navigation 2017 IBM Corporation

42 2017 IBM Corporation

43 Wearable Sensing for Understanding, Forecasting and Assisting Human Activity Kris Kitani Assistant Research Professor Carnegie Mellon University