Lifelog-Style Experience Recording and Analysis for Group Activities

Transcription

1 Lifelog-Style Experience Recording and Analysis for Group Activities Yuichi Nakamura Academic Center for Computing and Media Studies, Kyoto University

2 Lifelog and Grouplog for Experience Integration entering room take something camera in glasses captures First Person Vision leaving personal memory working integrating life logs of many people showing experiences on a large display

3 LifeLog through FPV RECORDING AND RECALLING PERSONAL EXPERIENCE head-mount camera: First Person View videos Digitized and structured

4 Grouplog: capturing activities through First person Vision It must be Why? I think so! How d think a it?

5 First Person Vision Videos The personal view records Advantages Rich information (compared to texts, audio, or photos) Record what the user sees (good trigger of memory) Disadvantages long and redundant data bad quality (shaky, etc.)

6 a b c d e

7 Summary of Personal View Records WHERE WHAT Objects or Scenes to which a user paid attention

8 Example of Summary (1) 2002/12/11 9:30-10:25 laboratory Meeting 2002/12/11 10:35-11:00 experimental lab Display the video by projector 2002/12/11 12:05-12:40 home Today s menu is Mexican. This is delicious!

9 Scenes of Attention indices for personal view records Active motion for gazing at Active Attention Stay looking at Passive Attention

10 Detecting Scenes of Attention our previous method geometric transformation parameters The image is converted to the image taken at the viewpoint of image T at time T- T frame. significant difference subtraction if the movement is small, the image make passive attention. Detect Difference the changed image region

11 Detect Scenes of Attention geometric transformation parameters The image is converted to the image taken at the viewpoint of image T at time T- T frame. significant difference. hybrid model subtraction (2D and 3D model) 2. screening of false detections if the movement is small, the image make passive attention. Detect the changed region

12 a b c d

13 a b c d

14 e f

15 The result of Detecting Scenes of Attention

16 The result of Detecting Scenes of Attention

17 Link between a surveillance view and a personal view Personal view analyze a user s behavior detect a scene of attention Linking Surveillance view track a human in the view detect a user s location

18 Use of surveillance camera

19 GUI Browser Results of integration with the personal view where and what the user paid attention what the user did

20 Group Log

21 What is Group Log? Multimedia Log of Experimental and Educational Group Activities Field work study, Outdoor activity, educational trip, etc. I think so! It must be How do you think about it? Why?

22 Why Group Log? Review group activities so as to make good use of the obtained knowledge for future experiences For participants : inspire their metacognition For organizers : give valuable feedback about the activity Summarize own experiences Review and discuss conducted activities

23 Collective First person View Video A First person view (FPV) video records Scene, Objects, Events that a person looked Head motions and moving events Collective FPV video Consists of FPV video logs of all participant The FPV videos interact for each other Person A Person D Person B Person E camera Person C Person F

24 Issues on Collective FPV Video Not easy to review Unstable camera work, e.g. rapid shaking Difficulty in simultaneously reviewing multiple FPV videos Long time video log of all participants

25 Integrating experiences outdoor school for an elementary school same events different view points with different behaviors various aspect of the same events

26 personal experience when, what, and where each person did

27 Reflection B saw what A was doing... subjective objective => comprehensive

28 differences and diversity different points of view among persons at the same place

29 finding organization two groups were generated without supervising

30 Vision research on FPV Videos re framing, gazing estimation, summarization

31 Re framing of FPV Video Re framing for easy review Stabilize Zoom into and track a particular interesting object Zoom out to understand a whole scene Re framed video may have lack on the view Input video Re framed video

32 Idea : Interpolation How to fill the lack Temporal interpolation : use scenes captured at different time Spatial interpolation : estimate outer region Actual camera work lack Temporal Spatial Space lack Requested Re framing Time

33 Processing flow Space time Over drawing Temporal Spatial Background Image Re framing output

34 Temporal interpolation Algorithms Image stitching with correspondences of local feature points Spatial interpolation Image inpainting

35 Results Input video Proposed Method With lacks

36 Tech: Estimation of gazes Estimate a gaze through scene construction Visual SLAM(Simultaneous Localization and Mapping) Unify coordinate systems of multiple gazes Use a visual marker as a reference coordinate system y x z

37 Visualizing Lines of Sight Relationship between lines of sight (LoS) give Concentration, diffusion, translation of interest Difference of interest in multiple person Los distribution view

38 Algorithm Head mounted camera Visual marker (AR toolkit) t i View lines of B and C on A s view Person A t 1 t 2 Person B Person C Capture ptam AR toolkit Rendering : frame including the visual marker : frame not including the visual marker run ptam Used for ptam init. run ptam Person A Person B Merge coordinate systems t

39 Summarization of FPV video Summarize portions of a FPV video with synthesized image boards to interpret situations at a glance Summarize about gazing action Summarize about conversation Summarize about using tool Etc. Summary boards representation t FPV video Gazing Action Conversation Using Tool

40 Summary board about gazing action Captured FPV video Synthesized summary board

41 Grouplog: capturing activities through First person Vision It must be Why? I think so! How d think a it?

42 Thank you for your attention!