Virtual Co-Location for Crime Scene Investigation and Going Beyond

Transcription

1 Virtual Co-Location for Crime Scene Investigation and Going Beyond Stephan Lukosch Faculty of Technology, Policy and Management, Systems Engineering Section Delft University of Technology Challenge the future

2 Introduction

3 Introduction current challenges Time needed for 3D reconstruction: data capture è alignment è data clean-up è geometric modeling è analyses Time freeze: data capture is often conducted after a scene has been contaminated Complexity: situations differ significantly Expertise: special skills required to secure evidence, expert availability, critical timing issues 3 26

4 Approach Virtual co-location Virtual co-location relies on augmented reality creates spaces in which people and other objects are either virtually or physically present allows users to engage in spatial remote collaboration allows remote users to interact with local users as if being there in person allows users to be virtually present at any place of the world Support virtual co-location of experts and investigators on the crime scene 4 26

5 Approach Virtual co-location 5 26

6 Requirements R1: Lightweight head-mounted display (HMD) R2: Contactless augmentation alignment (no markers on the crime scene) to keep the crime scene as uncontaminated as possible R3: Bare hands gestures for user interface operation to have free hands to physically interact with the crime scene R4: Remote connection to and collaboration with experts to guide a novice investigator through the crime scene 6 26

7 Lightweight head-mounted display (R1) Video see-through of a modified Carl Zeiss Cinemizer OLED 2 Microsoft HD-5000 webcams record at ~30hz in 720p Webcams render 720p stereoscopic images to the Cinemizer Weight of the HMD is ~180 grams 7 26

8 Contactless augmentation alignment (R2) 3D pose estimation: necessary to render virtual objects correctly on required positions stereo camera setup 3D natural feature matching and estimation sparse metric map for other system modules Dense 3D Map Maker: creates a detailed copy of the crime scene moving user creates continuous stream of disparity maps each new disparity map is combined to construct the 3D map point clouds are used for occlusion and collision checks for snapping digital objects to physical locations 8 26

9 Pose estimation and map making 9 26

10 Dense map making (Akman, 2012) Akman, O.: Robust Augmented Reality, Ph.D. thesis, Delft University of Technology, The Netherlands, December

11 Bare hand gestures (R3) Hand tracking: stereo camera rig to detect the hand movements in 3D Gesture-based interaction: left hand thumb-up, left hand thumbdown, right hand thumb-down and moving hand forward Menu items: recording the scene, tools, call colleagues, placing restricted area ribbons, map making, placing tags, bullet trajectories 11 26

12 Remote connection to and collaboration with experts (R4)! Client/server architecture based on OGRE render engine! Sharing of! image streams! camera track pose estimations! dense maps! interactions with the scene! Spatial-oriented collaboration! adding or removing artifacts! synchronous but turn-based! HMD wearer! stereoscopic first person view! 3D graphical user interface! hand gestures! Remote clients! 3D navigable view on the scene! mouse and keyboard 12 26

13 Remote connection to and collaboration with experts (R4) 13 26

14 Evaluation setup Setup (2 separated rooms) staged crime scene layman uses HMD observation room expert views crime scene on a large screen observers view layman and expert Subjects 3 observers, 1 expert, 1 layman Experimental procedure layman and expert can communicate via VoIP layman and expert jointly conduct 3 spatial tasks 1. tagging a specific part of the scene 2. using barrier tape and poles to secure the body 3. bullet trajectory analysis with ricochet 14 26

15 Evaluation results Visual feedback provides mutual understanding of the analysis state, expert statement: there is not much arguing about something you can both see Information exchange was very picture oriented, e.g.: behind the table, don t cross the uttermost right ribbon, if you look slightly to the left Ambivalent feedback on recording and remote observation, e.g. monitoring tool vs. possibility to capture best practices Presence of the expert is not experienced Experts would like to see the location with their own eyes Use system for training purposes, observer question: Can you inflict virtual injury on a dummy or actor? 15 26

16 Conclusions Virtual co-location allows remote users to collaborate with the wearer of the HMD in shared augmented space builds a 3D map of the environment in real-time uses bare hand gestures to operate the 3D user interface Evaluation shows that the system supports remote spatial interaction with the physical scene supports collaboration in shared augmented space tackles current issues in crime scene investigation 16 26

17 FUTURE WORK 17 26

18 Future work Application and test of the system in real settings Improve algorithm for map making and hand detection Explore new models for interaction in AR User interface Collaboration support Explore effect of different HMDs Provide situational awareness in AR Interaction in augmented reality Perceived and witnessed presence of experts Validity of the data Data fusion (heat cameras, existing models, databases) 18 26

19 Upcoming head-mounted device(s) AR headset MARTY based on the Sony HMZ-T1 designed by Niels Mulder (Studio RAAR) on behalf of AR Lab 19 26

20 Upcoming sensor(s) - heart-beat detection 20 26

21 Upcoming augmented interfaces (Swart, 2012) Swart, T.: Design of a gesture controlled graphic interface for Head Mounted Displays for CSI The Hague, Delft University of Technology, The Netherlands,

22 FUTURE APPLICATION SCENARIOS 22 26

23 Future application scenario I Topic: Information visualization, sharing and fusion to stimulate shared situational awareness between local and remote investigators Partner: Dutch Police, Netherlands Forensic Institute (NFI), Dutch Fire Department On the Spot: Augmented Reality ten behoeve van een gedeeld (veiligheids)beeld 23 26

24 Future application scenarios III Topics:! Collaboration in AR for ISS maintenance! Training astronauts for future experiments Partner: ESA 24 26

25 Future application scenarios IV Topic: Collaborative design and engineering in the brown field Partners: Dow Chemicals, Grontmij, RealToDesk, ApplyCapnor 25 26

26 Thank you for your attention! Virtual Co-Location for Crime Scene Investigation and Going Beyond Stephan Lukosch Faculty of Technology, Policy and Management, Systems Engineering Section Delft University of Technology Challenge the future