Virtual Worlds for the Perception and Control of Self-Driving Vehicles Dr. Antonio M. López antonio@cvc.uab.es
Index Context SYNTHIA: CVPR 16 SYNTHIA: Reloaded SYNTHIA: Evolutions CARLA Conclusions
Index Context SYNTHIA: CVPR 16 SYNTHIA: Reloaded SYNTHIA: Evolutions CARLA Conclusions
Our Mission as CVC/UAB group 4 Forming students (undergraduate, master and PhD) in the fields of Computer Vision, Machine Learning, and Artificial Intelligence for Autonomous Systems, in particular, Cars. Basic Research producing high impact papers in top-level conferences and Q1 journals. Technological transfer & Innovation developing prototypes, demonstrators and products jointly with the industry. Dissemination doing an effort to bring our research and its applications to the general public. antonio@cvc.uab.es // www.cvc.uab.es/~antonio
antonio@cvc.uab.es // www.cvc.uab.es/~antonio 5
Research: ML for Vision 6 I m bored, let s labelling data for fun! antonio@cvc.uab.es // www.cvc.uab.es/~antonio
antonio@cvc.uab.es // www.cvc.uab.es/~antonio 7
Index Context SYNTHIA: CVPR 16 SYNTHIA: Reloaded SYNTHIA: Evolutions CARLA Conclusions
antonio@cvc.uab.es // www.cvc.uab.es/~antonio 9
antonio@cvc.uab.es // www.cvc.uab.es/~antonio 10
antonio@cvc.uab.es // www.cvc.uab.es/~antonio 11
antonio@cvc.uab.es // www.cvc.uab.es/~antonio 12
Semantic segmentation results The SYNTHIA Dataset: A Large Collection of Synthetic Images for Semantic Segmentation of Urban Scenes, G Ros, L. Sellart, J. Materzynska, D. Vázquez, A.M. López, CVPR 2016 13 antonio@cvc.uab.es // www.cvc.uab.es/~antonio
Data Publicly Released at 14 www.synthia-dataset.net Image generator to acquire thousands of data with several kinds of ground truth. RGB & Per pixel: depth, semantic class (CamVid), instance ID We simulated different weather and illumination conditions, as well as four seasons We simulated a camera setting for covering 360º >300,000 images with their ground truth available antonio@cvc.uab.es // www.cvc.uab.es/~antonio
DPM to assess Photo-Realism 15 vehicle detection SYNTHIA GTA-V antonio@cvc.uab.es // www.cvc.uab.es/~antonio
Back to DPM to assess Photo-Realism: 16 vehicle detection From Virtual to Real World Visual Perception using Domain Adaptation -- The DPM as Example, A.M. López, J. Xu, J.L. Gómez, D. Vázquez, G. Ros, arxiv:1612.09134 To appear in Domain Adaptation in Computer Vision Applications, Springer Series: Advances in Computer Vision and Pattern Recognition, Edited by Gabriela Csurka antonio@cvc.uab.es // www.cvc.uab.es/~antonio
antonio@cvc.uab.es // www.cvc.uab.es/~antonio 17
Change Detection 18 antonio@cvc.uab.es // www.cvc.uab.es/~antonio
Summary of the Research 19 antonio@cvc.uab.es // www.cvc.uab.es/~antonio
Index Context SYNTHIA: CVPR 16 SYNTHIA: Reloaded SYNTHIA: Evolutions CARLA Conclusions
antonio@cvc.uab.es // www.cvc.uab.es/~antonio 21
antonio@cvc.uab.es // www.cvc.uab.es/~antonio 22
antonio@cvc.uab.es // www.cvc.uab.es/~antonio 23
antonio@cvc.uab.es // www.cvc.uab.es/~antonio 24
antonio@cvc.uab.es // www.cvc.uab.es/~antonio 25
antonio@cvc.uab.es // www.cvc.uab.es/~antonio 26
antonio@cvc.uab.es // www.cvc.uab.es/~antonio 27
Best Industrial Paper at BMVC 17 28 slope horizon Stereo + Horizon Line + Road Slope Stereo Images Semantic Stixels Semantic segmentation antonio@cvc.uab.es // www.cvc.uab.es/~antonio
Original Stixels 29 Slanted Stixels New dataset: SYNTHIA-San Francisco publicly available soon antonio@cvc.uab.es // www.cvc.uab.es/~antonio
Index Context SYNTHIA: CVPR 16 SYNTHIA: Reloaded SYNTHIA: Evolutions CARLA Conclusions
antonio@cvc.uab.es // www.cvc.uab.es/~antonio 31
Adding 360º LIDAR with Semantics 32 antonio@cvc.uab.es // www.cvc.uab.es/~antonio
antonio@cvc.uab.es // www.cvc.uab.es/~antonio 33
Image-to-Image Domain Adaptation: case study on traffic sign recognition Assumptions: 34 1) Real world: missing classes 2) Virtual world: easy to generate samples of any class Real world data Proposal: Virtual world data We want: A new real-world classifier that takes into account the missing classes, but with minimum annotation effort. 1) Train a deep network that knows to transform the virtual images to look like the real ones, using only the intersection classes for training this network. 2) Use the virtual world to generate many examples of the missing classes. 3) Transform the virtual samples according to the learned network. Known classes 4) Train the real-world classifier using the real-world samples (of previous classes) and the transformed samples (of new classes). antonio@cvc.uab.es // www.cvc.uab.es/~antonio New classes
Image-to-Image Domain Adaptation: case study on traffic sign recognition 35 174 Traffic signs types (the ones of Tsinghua dataset). ~ 260000 images generated per day. We force variability: light, background, viewpoint, etc. It is simple to add new traffic signs types. antonio@cvc.uab.es // www.cvc.uab.es/~antonio
antonio@cvc.uab.es // www.cvc.uab.es/~antonio 36
Image-to-Image Domain Adaptation: case study on traffic sign recognition 37 S T S T Known Classes S T S T antonio@cvc.uab.es // www.cvc.uab.es/~antonio
Image-to-Image Domain Adaptation: case study on traffic sign recognition 38 New Classes antonio@cvc.uab.es // www.cvc.uab.es/~antonio
Physics-based Rendering in SYNTHIA 39 antonio@cvc.uab.es // www.cvc.uab.es/~antonio
Video Analytics towards Vision Zero Project City of Bellevue, Washington, USA 40 Video Analytics towards Vision Zero, Franz Loewenherz, Victor Bahl, Yinhai Wang, ITE Journal, Vol. 87, n. 3, March 2017. Keys: Analytics at intersections. Training of neural networks required. Crowdsourcing of volunteers for collecting ground truth data. Unity & CVC/UAB Rendered data antonio@cvc.uab.es // www.cvc.uab.es/~antonio
antonio@cvc.uab.es // www.cvc.uab.es/~antonio 41
antonio@cvc.uab.es // www.cvc.uab.es/~antonio 42
antonio@cvc.uab.es // www.cvc.uab.es/~antonio 43
Augmented Reality 44 antonio@cvc.uab.es // www.cvc.uab.es/~antonio
Augmented Reality 45 antonio@cvc.uab.es // www.cvc.uab.es/~antonio
Augmented Reality 46 antonio@cvc.uab.es // www.cvc.uab.es/~antonio
Augmented Reality 47 antonio@cvc.uab.es // www.cvc.uab.es/~antonio
Index Context SYNTHIA: CVPR 16 SYNTHIA: Reloaded SYNTHIA: Evolutions CARLA Conclusions
49 More photo-realism and ground truth: New datasets Vision Zero project antonio@cvc.uab.es // www.cvc.uab.es/~antonio
50 More photo-realism and ground truth: New datasets Vision Zero project Car Learning to Act: Interactive simulator Open-source spirit antonio@cvc.uab.es // www.cvc.uab.es/~antonio
51 Server Physic simulations Rendering Ground truth Privileged information Client Data recording Environment settings control Vehicle control AI antonio@cvc.uab.es // www.cvc.uab.es/~antonio
52 Features So far two towns from the scratch Different weather/daytime conditions Sets of cameras attached to the vehicle Depth, semantic classes, 3D bounding boxes Speed, traffic infractions, collisions Synch / Asynch modes Based on own assets or free available ones We will open source our C++ code Publicly available soon antonio@cvc.uab.es // www.cvc.uab.es/~antonio
antonio@cvc.uab.es // www.cvc.uab.es/~antonio 53
antonio@cvc.uab.es // www.cvc.uab.es/~antonio 54
55 www.carla.org We compare: (1) Modular pipeline; (2) Imitation learning; (3) Reinforcement learning antonio@cvc.uab.es // www.cvc.uab.es/~antonio
56 Conditional Imitation Learning antonio@cvc.uab.es // www.cvc.uab.es/~antonio
antonio@cvc.uab.es // www.cvc.uab.es/~antonio 57
antonio@cvc.uab.es // www.cvc.uab.es/~antonio 58
Index Context SYNTHIA: CVPR 16 SYNTHIA: Reloaded SYNTHIA: Evolutions CARLA Conclusions
60 Simulation of perception and control methods are essential for designing, training and testing AI drivers; both datasets and interactive simulations are key, as SYNTHIA and CARLA. Virtual- to real-world domain adaptation is an essential topic, both for pure perception and for sensorimotor models. SYNTHIA: generating more photorealistic datasets and eventually training deep networks to control the parameters of the image generation (render and composition, augmented reality). CARLA: add more sensors and content, as well as external interaction models. antonio@cvc.uab.es // www.cvc.uab.es/~antonio
Many thanks for attending!!! Many thanks to the many people of the CVC/UAB that has been contributing to this work, especially to Jose A., Felipe, Marc, Fran, Xisco, Néstor, Fran2, Alberto, Iris, Mario, Ignazio, Juan, Daniel, Laura, Juan Carlos, Toni, David, etc. etc. As well as to people from different companies I cannot name (confidentiality), and others I can name: Vladlen, Alexey, Germán, JoseD, Diana, Renaldas, Uwe, David, Dough, etc. etc.