Virtual Worlds for the Perception and Control of Self-Driving Vehicles

Transcription

1 Virtual Worlds for the Perception and Control of Self-Driving Vehicles Dr. Antonio M. López

2 Index Context SYNTHIA: CVPR 16 SYNTHIA: Reloaded SYNTHIA: Evolutions CARLA Conclusions

3 Index Context SYNTHIA: CVPR 16 SYNTHIA: Reloaded SYNTHIA: Evolutions CARLA Conclusions

4 Our Mission as CVC/UAB group 4 Forming students (undergraduate, master and PhD) in the fields of Computer Vision, Machine Learning, and Artificial Intelligence for Autonomous Systems, in particular, Cars. Basic Research producing high impact papers in top-level conferences and Q1 journals. Technological transfer & Innovation developing prototypes, demonstrators and products jointly with the industry. Dissemination doing an effort to bring our research and its applications to the general public. antonio@cvc.uab.es //

5 // 5

6 Research: ML for Vision 6 I m bored, let s labelling data for fun! antonio@cvc.uab.es //

7 // 7

8 Index Context SYNTHIA: CVPR 16 SYNTHIA: Reloaded SYNTHIA: Evolutions CARLA Conclusions

9 // 9

10 // 10

11 // 11

12 // 12

13 Semantic segmentation results The SYNTHIA Dataset: A Large Collection of Synthetic Images for Semantic Segmentation of Urban Scenes, G Ros, L. Sellart, J. Materzynska, D. Vázquez, A.M. López, CVPR antonio@cvc.uab.es //

14 Data Publicly Released at 14 Image generator to acquire thousands of data with several kinds of ground truth. RGB & Per pixel: depth, semantic class (CamVid), instance ID We simulated different weather and illumination conditions, as well as four seasons We simulated a camera setting for covering 360º >300,000 images with their ground truth available antonio@cvc.uab.es //

15 DPM to assess Photo-Realism 15 vehicle detection SYNTHIA GTA-V //

16 Back to DPM to assess Photo-Realism: 16 vehicle detection From Virtual to Real World Visual Perception using Domain Adaptation -- The DPM as Example, A.M. López, J. Xu, J.L. Gómez, D. Vázquez, G. Ros, arxiv: To appear in Domain Adaptation in Computer Vision Applications, Springer Series: Advances in Computer Vision and Pattern Recognition, Edited by Gabriela Csurka //

17 // 17

18 Change Detection 18 //

19 Summary of the Research 19 //

20 Index Context SYNTHIA: CVPR 16 SYNTHIA: Reloaded SYNTHIA: Evolutions CARLA Conclusions

21 // 21

22 // 22

23 // 23

24 // 24

25 // 25

26 // 26

27 // 27

28 Best Industrial Paper at BMVC slope horizon Stereo + Horizon Line + Road Slope Stereo Images Semantic Stixels Semantic segmentation antonio@cvc.uab.es //

29 Original Stixels 29 Slanted Stixels New dataset: SYNTHIA-San Francisco publicly available soon //

30 Index Context SYNTHIA: CVPR 16 SYNTHIA: Reloaded SYNTHIA: Evolutions CARLA Conclusions

31 // 31

32 Adding 360º LIDAR with Semantics 32 //

33 // 33

34 Image-to-Image Domain Adaptation: case study on traffic sign recognition Assumptions: 34 1) Real world: missing classes 2) Virtual world: easy to generate samples of any class Real world data Proposal: Virtual world data We want: A new real-world classifier that takes into account the missing classes, but with minimum annotation effort. 1) Train a deep network that knows to transform the virtual images to look like the real ones, using only the intersection classes for training this network. 2) Use the virtual world to generate many examples of the missing classes. 3) Transform the virtual samples according to the learned network. Known classes 4) Train the real-world classifier using the real-world samples (of previous classes) and the transformed samples (of new classes). antonio@cvc.uab.es // New classes

35 Image-to-Image Domain Adaptation: case study on traffic sign recognition Traffic signs types (the ones of Tsinghua dataset). ~ images generated per day. We force variability: light, background, viewpoint, etc. It is simple to add new traffic signs types. antonio@cvc.uab.es //

36 // 36

37 Image-to-Image Domain Adaptation: case study on traffic sign recognition 37 S T S T Known Classes S T S T antonio@cvc.uab.es //

38 Image-to-Image Domain Adaptation: case study on traffic sign recognition 38 New Classes //

39 Physics-based Rendering in SYNTHIA 39 //

40 Video Analytics towards Vision Zero Project City of Bellevue, Washington, USA 40 Video Analytics towards Vision Zero, Franz Loewenherz, Victor Bahl, Yinhai Wang, ITE Journal, Vol. 87, n. 3, March Keys: Analytics at intersections. Training of neural networks required. Crowdsourcing of volunteers for collecting ground truth data. Unity & CVC/UAB Rendered data //

41 // 41

42 // 42

43 // 43

44 Augmented Reality 44 //

45 Augmented Reality 45 //

46 Augmented Reality 46 //

47 Augmented Reality 47 //

48 Index Context SYNTHIA: CVPR 16 SYNTHIA: Reloaded SYNTHIA: Evolutions CARLA Conclusions

49 49 More photo-realism and ground truth: New datasets Vision Zero project //

50 50 More photo-realism and ground truth: New datasets Vision Zero project Car Learning to Act: Interactive simulator Open-source spirit //

51 51 Server Physic simulations Rendering Ground truth Privileged information Client Data recording Environment settings control Vehicle control AI //

52 52 Features So far two towns from the scratch Different weather/daytime conditions Sets of cameras attached to the vehicle Depth, semantic classes, 3D bounding boxes Speed, traffic infractions, collisions Synch / Asynch modes Based on own assets or free available ones We will open source our C++ code Publicly available soon antonio@cvc.uab.es //

53 // 53

54 // 54

55 55 We compare: (1) Modular pipeline; (2) Imitation learning; (3) Reinforcement learning //

56 56 Conditional Imitation Learning //

57 // 57

58 // 58

59 Index Context SYNTHIA: CVPR 16 SYNTHIA: Reloaded SYNTHIA: Evolutions CARLA Conclusions

60 60 Simulation of perception and control methods are essential for designing, training and testing AI drivers; both datasets and interactive simulations are key, as SYNTHIA and CARLA. Virtual- to real-world domain adaptation is an essential topic, both for pure perception and for sensorimotor models. SYNTHIA: generating more photorealistic datasets and eventually training deep networks to control the parameters of the image generation (render and composition, augmented reality). CARLA: add more sensors and content, as well as external interaction models. //

61 Many thanks for attending!!! Many thanks to the many people of the CVC/UAB that has been contributing to this work, especially to Jose A., Felipe, Marc, Fran, Xisco, Néstor, Fran2, Alberto, Iris, Mario, Ignazio, Juan, Daniel, Laura, Juan Carlos, Toni, David, etc. etc. As well as to people from different companies I cannot name (confidentiality), and others I can name: Vladlen, Alexey, Germán, JoseD, Diana, Renaldas, Uwe, David, Dough, etc. etc.