Vision-based Localization and Mapping with Heterogeneous Teams of Ground and Micro Flying Robots

Transcription

1 Vision-based Localization and Mapping with Heterogeneous Teams of Ground and Micro Flying Robots Davide Scaramuzza Robotics and Perception Group University of Zurich All videos in this presentation can be found at: YouTube Channel:

2 My Research Group rpg.ifi.uzh.ch

3 Current Research: Computer Vision applied to Autonomous Navigation of Micro Flying Robots Vision-based Navigation of Flying Robots Air-ground collaboration Event-based Vision Vision-based Control for aggressive flight

4 Outline Autonomous GPS-denied Navigation Multi-robot systems Homogeneous systems (air-air) Heterogeneous systems (air-ground)

5 Outline Autonomous GPS-denied Navigation Multi-robot systems Homogeneous systems (air-air) Heterogeneous systems (air-ground)

6 Autonomous, Vision-based Navigation in GPS-denied Environments

7 Why not GPS? Does not work indoors Even outdoors it is not a reliable service?

8 Why not GPS? Does not work indoors Even outdoors it is not a reliable service Satellite coverage Multipath problem

9 Visual Odometry How does it work? What are good features to track? Image 1 Image 2 R, T

10 Keyframe-based Visual Odometry Keyframe 1 Keyframe 2 Current frame New keyframe Initial pointcloud New triangulated points [Forster, Pizzoli, Scaramuzza, «SVO: Semi Direct Visual Odometry», ICRA 14, RSS 15]

11 Visual-Inertial Fusion Fusion is solved as a non-linear optimization problem (no Kalman filter): Increased accuracy over filtering methods IMU residuals Reprojection residuals [Forster, Carlone, Dellaert, Scaramuzza, IMU Preintegration on Manifold for efficient Visual-Inertial Maximum-a-Posteriori University of Zurich Robotics Estimation, and Perception RSS 15] Group - rpg.ifi.uzh.ch

12 Visual Odometry Accuracy: 0.1% of the travel distance [Forster, Carlone, Dellaert, Scaramuzza, IMU Preintegration on Manifold for Efficient Visual-Inertial University of Zurich Maximum-a-Posteriori Robotics and Perception Estimation, Group RSS 15] - rpg.ifi.uzh.ch

13 SmartPhone computer for image analysis Quad Core Odroid (ARM Cortex A-9): used in Samsung Galaxy S4 phones Inertial sensors 170 grams! HD camera Global shutter HDR 90 fps

14 Autonomous Vision-based Flight (no GPS) [Forster, Pizzoli, Scaramuzza, «SVO: Semi Direct Visual Odometry», ICRA 14]

15 Bayesian Estimation of the Depth Uncertainty Initialize a depth-filter for every new feature Recursive Bayesian depth estimation Measurement Likelihood models outliers: [Forster, University Pizzoli, of Zurich Scaramuzza, Robotics and «SVO: Perception Semi Group Direct - Visual rpg.ifi.uzh.ch Odometry», ICRA 14]

16 Localization and Mapping with Swarms of Micro Aerial Vehicles

17 C. Forster, S. Lynen, L. Kneip, D. Scaramuzza, Collaborative Monocular SLAM with Multiple Micro Aerial Vehicles, IROS 13 University ] of Zurich Robotics and Perception Group - rpg.ifi.uzh.ch

18 C. Forster, S. Lynen, L. Kneip, D. Scaramuzza, Collaborative Monocular SLAM with Multiple Micro Aerial Vehicles, IROS 13 University ] of Zurich Robotics and Perception Group - rpg.ifi.uzh.ch

19 C. Forster, S. Lynen, L. Kneip, D. Scaramuzza, Collaborative Monocular SLAM with Multiple Micro Aerial Vehicles, IROS 13 University ] of Zurich Robotics and Perception Group - rpg.ifi.uzh.ch

20 C. Forster, S. Lynen, L. Kneip, D. Scaramuzza, Collaborative Monocular SLAM with Multiple Micro Aerial Vehicles, IROS 13 University ] of Zurich Robotics and Perception Group - rpg.ifi.uzh.ch

21 Motivation Map faster an unknown environment Know relative position between robots Robust against single-robot failure This problem is known in robotics as Multi-Robot Visual SLAM (Simultaneous Localization and Mapping)

22 Comparison with State-of-the-Art Co-SLAM [Danping and Ping, PAMI 12] Synchronized Known initial positions Fully centralized Our proposed Solution [IROS 13] Asynchronous Unknown initial positions Distributed pre-processing Cam 1 Cam 2 VO 1 VO 2 Frame Handler Frame Handler Map Map Place Recognizer

23 System Overview MAVs VO 1 VO 2 Ground station Map Map Distributed processing: Each MAV runs an onboard visual odometry and streams point features and relative poses (1 Mbit/s instead of 90 Mbit/s for full frames at 30 Hz) The ground station computes local maps for each MAV, detects overlaps, and merges different maps into global map

24 System Overview MAVs VO 1 VO 2 Ground station Map Map Place Recognizer Distributed processing: Each MAV runs an onboard visual odometry and streams point features and relative poses (1 Mbit/s instead of 90 Mbit/s for full frames at 30 Hz) The ground station computes local maps for each MAV, detects overlaps, and merges different maps into global map

25 System Overview MAVs VO 1 VO 2 Ground station Map Map Place Recognizer Distributed processing: Map Each MAV runs an onboard visual odometry and streams point features and relative poses (1 Mbit/s instead of 90 Mbit/s for full frames at 30 Hz) The ground station computes local maps for each MAV, detects overlaps, and merges different maps into global map

26 System Overview MAVs VO 1 VO 2 Ground station Map Map Place Recognizer Distributed processing: Map Each MAV runs an onboard visual odometry and streams point features and relative poses (1 Mbit/s instead of 90 Mbit/s for full frames at 30 Hz) The ground station computes local maps for each MAV, detects overlaps, and merges different maps into global map

27 Mapping on the Groundstation MAV R, t MAVs VO 1 Groundstation Ground station Map C. Forster, S. Lynen, L. Kneip, D. Scaramuzza, Collaborative Monocular SLAM with Multiple Micro Aerial Vehicles, IROS 13 University ] of Zurich Robotics and Perception Group - rpg.ifi.uzh.ch

28 Mapping on the Groundstation MAV R, t MAVs VO 1 Groundstation Ground station Map C. Forster, S. Lynen, L. Kneip, D. Scaramuzza, Collaborative Monocular SLAM with Multiple Micro Aerial Vehicles, IROS 13 University ] of Zurich Robotics and Perception Group - rpg.ifi.uzh.ch

29 Mapping on the Groundstation MAV R, t Groundstation R, t Use motion estimate from Visual Odometry as prior C. Forster, S. Lynen, L. Kneip, D. Scaramuzza, Collaborative Monocular SLAM with Multiple Micro Aerial Vehicles, IROS 13 University ] of Zurich Robotics and Perception Group - rpg.ifi.uzh.ch

30 Mapping on the Groundstation MAV R, t Groundstation R BA, t BA Refine pose w.r.t map with Bundle Adjustment g2o [Kümmerle et al., ICRA 11]

31 Place Recognition MAVs VO 1 VO 2 Map Map Place Recognizer 1. Appearance-based Detection Bag of Words image retrieval 2. Geometric Verification [Sivic et al., 2005] 3-point RANSAC for point-cloud alignment Ground station 3-point algorithm [Kneip & Scaramuzza,CVPR 11]

32 Loop-Closure (single robot) Indoor flight 7 DoF pose-graph optimization based on [Strasdat et al., RSS 2010] C. Forster, S. Lynen, L. Kneip, D. Scaramuzza, Collaborative Monocular SLAM with Multiple Micro Aerial Vehicles, IROS 13 University ] of Zurich Robotics and Perception Group - rpg.ifi.uzh.ch

33 Map Merging (multiple robots) Indoor flight C. Forster, S. Lynen, L. Kneip, D. Scaramuzza, Collaborative Monocular SLAM with Multiple Micro Aerial Vehicles, IROS 13 University ] of Zurich Robotics and Perception Group - rpg.ifi.uzh.ch

34 Map Merging (multiple robots) Outdoor flight C. Forster, S. Lynen, L. Kneip, D. Scaramuzza, Collaborative Monocular SLAM with Multiple Micro Aerial Vehicles, IROS 13 University ] of Zurich Robotics and Perception Group - rpg.ifi.uzh.ch

35

36 Multi-robot Dense Reconstruction [Scaramuzza, Achtelik, Weiss, Fraundorfer, et al., Vision-Controlled Micro Flying Robots: from System Design to Autonomous Navigation and Mapping in GPS-denied Environments, IEEE RAM, 2014]

37 Air-Ground Collaboration

38 Robots for Search and Recue? Search and rescue missions can benefit from robotic technologies (Fukushima, Gotthard rock slide, Italy earthquake) Most robots move on the ground and are teleoperated by trained professionals Fukushima Nuclear Power Plant

39 Fukushima Nuclear Power Plant

40 Collaboration between Ground and Aerial Robots Air-ground exploration Air-ground collaborative grasping C. Forster, M. Pizzoli, D. Scaramuzza, Air-Ground Localization and Map Augmentation Using Monocular Dense Reconstruction, IROS 13] Mueggler, Faessler, Fontana, Scaramuzza, Aerial-guided Navigation of a Ground Robot among Movable Obstacles,, University SSRR 14] of Zurich Robotics and Perception Group - rpg.ifi.uzh.ch

41 Air-Ground Localization

42 How can we achieve Mutual Localization? Two distinct approaches Relative observations - Ground to air - Air to ground Common scene observation

43 Air-Ground Robot Localization based on Infrared LEDs 5 infrared LEDs Camera with infrared filter

44 Air-Ground Robot Localization based on Infrared LEDs Schwabe, Faessler, Mueggler, Scaramuzza, A Monocular Poese Estimation System based on Infrared LEDs, ICRA 14]

45 Air-Ground Robot Localization based on Common Scene Observation Camera Laser Forster, Pizzoli, Scaramuzza, Air-Ground Localization and Map Augmentation Using Monocular Dense Reconstruction, University of Zurich IROS 13 Robotics and Perception Group - rpg.ifi.uzh.ch

46 Air-ground Collaborative Mapping Forster, Pizzoli, Scaramuzza, Air-Ground Localization and Map Augmentation Using Monocular Dense Reconstruction, IROS 13

47 Challenge Radically different view-points! Different sensors

48 Standard Feature Matching Appearance-based relative localization with feature matching fails! Some matches in locally planar scenes Affine SIFT [Morel, SIAM 09] No matches Our Solution: First build dense maps with each robot and then align the maps

49 Ground Robot Mapping with Kinect Trajectory of the ground-robot Dense point-cloud [Forster et al., «Semi-direct Visual Odometry», submitted to ICRA 2014]

50 Ground Robot Mapping with Kinect Trajectory of the ground-robot Dense point-cloud [Forster et al., «Semi-direct Visual Odometry», submitted to ICRA 2014]

51 Aerial Mapping with Single Camera Red points: ground-robot map [Forster [M. University Pizzoli, et al., C. of Forster, «Semi-direct Zurich D. Scaramuzza, Visual Odometry», submitted to ICRA 2014] Robotics and REMODE: Perception Probabilistic, Group - rpg.ifi.uzh.ch Monocular Dense Reconstruction in Real Time, ICRA 14]

52 Initial Guess from Heightmap Correlation Aerial heigh map Ground heigh map Correlation Zero Mean Sum of Squared Differences (ZMSSD) cost: [Forster et al., «Semi-direct Visual Odometry», submitted to ICRA 2014]

53 Map Augmentation

54 Air-Ground Collaborative Grasping

55 Aerial Guided Navigation among Movable Obstacles Winner of KUKA Innovation Award at AUTOMATICA

56 Aerial Guided Navigation among Movable Obstacles Winner of KUKA Innovation Award at AUTOMATICA Target location Obstacle to remove Optimal path

57 Aerial-Guided Navigation among Movable Obstacles Winner of KUKA Innovation Award at AUTOMATICA Mueggler, Faessler, Fontana, Scaramuzza, Aerial-guided Navigation of a Ground Robot among Movable Obstacles, University SSRR 14] of Zurich Robotics and Perception Group - rpg.ifi.uzh.ch

58 All videos in this presentation can be found at: Software and datasets:

59 Thanks! Questions? Software: Funding