Combining ROS and AI for fail-operational automated driving Prof. Dr. Daniel Watzenig Virtual Vehicle Research Center, Graz, Austria and Institute of Automation and Control at Graz University of Technology VIRTUAL VEHICLE 1
COMET K2-Center AUTOMOTIVE RAIL AEROSPACE Gegründet: 2002 Mitarbeiter: 204 Umsatz: Standort: Website: 20,3 Mio. EUR Graz www.v2c2.at Shareholder: Dr. Jost Bernasch Geschäftsführer Prof. Hermann Steffan Wissenschaftlicher Leiter December 2017 Software Defined Vehicles VIRTUAL VEHICLE 2
Our automated driving research activities Embedded control and software functions Real-time sensor fusion Sensor self-diagnostics and fail-operational architectures Dependable computing and reliable in vehicle control (strong multi-core expertise in both SW and HW) Functional safety analyses (ISO PAS 21448) Virtual prototyping of automated driving functions Validation of real driving scenarios (real-time co-simulation) Traffic simulation (micro, macro) / infrastructure integration AI in both function development and vehicle operation December 2017 Software Defined Vehicles VIRTUAL VEHICLE 3
Outline Motivating example The challenge fail-operational ROS and AI integration Simulation, visualization, and communication AD demo car AI example for active safety design Summary December 2017 Software Defined Vehicles VIRTUAL VEHICLE 4
Motivating example Virtual Vehicle Research Center in Graz (about 200 employees) Expertise in simulation and experimental verification Focus on Automated Driving Dependable Systems Group Functional safety Verification and validation December 2017 Software Defined Vehicles VIRTUAL VEHICLE 5
Motivating example Motivating example Uber car Safety is top priority for user acceptance of automated driving http://orf.at/stories/2384924/2384925/ December 2017 Software Defined Vehicles VIRTUAL VEHICLE 6
Motivating example Motivating example consequences Achieve user acceptance Avoid accidents and hazards (see example) Keep development and verification efforts reasonable Easy to use simulation environment during development Verify software on different test platforms MiL, SIL, HiL, and vehicle Simple goals, hard to achieve December 2017 Software Defined Vehicles VIRTUAL VEHICLE 7
Technical challenges Reference architecture Perceived objects and events Value judgement Plan evaluation Situation evaluation Plan results Sensory processing Update Predicted Input Environment model Plan State Behaviour generation Sensor inputs Actions Sensors Actuators ANSI Reference architecture of intelligent systems Information and data uncertainties! December 2017 Software Defined Vehicles VIRTUAL VEHICLE 8
Technical challenges Vehicle technologies Sensory processing Camera, RADAR, LiDAR, Value judgement and environment model High performance computing (HPC) Segmentation (using AI) Camera HPC ECU Accelaration/Brake Situation identification RADAR Behaviour estimation of other traffic participants LiDAR Steering Behaviour generation Electronic Control Units (ECU) Vehicle control algorithms December 2017 Software Defined Vehicles VIRTUAL VEHICLE 9
Limits of sensors As effective sensors are, they have some drawbacks Limited range Performance is susceptible to common environmental conditions (rain, fog, varying lighting conditions) False positives Range determination not as accurate as required The use of several sensor types can ensure a higher level of confidence in target detection and characterization Robust sensors and sensor self-diagnosis Redundancy in HW and SW ( fail-operational ) Sensor fusion (raw data? objects? hybrid?) December 2017 Software Defined Vehicles VIRTUAL VEHICLE 10
Vehicle detection: false positives Use of heat maps (buffer where each pixel of successfully detected window adds 1 ) Average the buffer from the last 10-20 frames Use only pixels that survived a certain amount of frames Extract component bounding blocks vehicle(s) December 2017 Software Defined Vehicles VIRTUAL VEHICLE 11
Enhancement of view Austrian test region ALP.Lab December 2017 Software Defined Vehicles VIRTUAL VEHICLE 12
Enhancement of view December 2017 Software Defined Vehicles VIRTUAL VEHICLE 13
System architecture automated driving unpredictable conditions fuse all input to one model plan safe actions under current conditions Radar LIDAR Cameras Infrared Ultrasonic V2X Reliable sensor data processing and fusion Raw data analysis of all sensors Deliver consistent environmental model Scene understanding, driver monitoring, decision making, and planning Situation and behaviour prediction Planning of provably safe trajectories Handover/takeover planning Other Sensors System Performance and Driver Monitoring Out-of-position Warning Intervention Measure reliability and uncertainty Detection and decision on function availability System degradation Sensor self-diagnosis Ensuring failoperational behaviour steer functions and vehicle Fail-operational X-by-wire actuation(low level control) [Source: based on ECSEL Project RobustSense, 2015] December 2017 Software Defined Vehicles VIRTUAL VEHICLE 14
Towards fail-operational: e.g. power steering Fail-safe (what we have now) No emergency operation necessary Safe state: system off, driver immediately in control loop High-availability Safe state: system off, driver immediately in control loop Emergency operation is desirable but not required Minimize hazardous situation in case of potential misuse Fail-operational Emergency operation is required (10-15s) Eyes-off, brain-off Achieved by adding measures to all vital parts [Source: Steindl, Miedl, Safetronic, 2015] December 2017 Software Defined Vehicles VIRTUAL VEHICLE 15
Homogeneous vs. diversity concept Homogeneous redundancy uses a minimum of two equal instances in parallel the effort in development can be reduced due to the identical components because of the equality, this approach only protects against random faults caused by aging, deterioration or bit flips probability for a complete system crash is higher than in approaches with diverse components Redundancy by diversity (avionics) The calculating components are heterogeneous, e.g. from different manufacturers System SW of each unit is different or uses at least different HW components this system SW diversity complements functional diversity of the application SW Different implementations result in a lower probability of failure for the system due to the lower probability that the diverse components show the same misbehavior at the same time December 2017 Software Defined Vehicles VIRTUAL VEHICLE 16
Fail-operational architecture EC Project PRYSTINE (2018 to 2021, Programmable systems for intelligence in automobiles) Infineon, Scania, Ford, BMW, Virtual Vehicle, TU Graz December 2017 Software Defined Vehicles VIRTUAL VEHICLE 17
Fail-operational architecture Massive redundancy will not be the solution (2x and 3x, STANAG 4626) We need smart solutions! Mechanism for HW fault detection (e.g. BIST, built-in self tests) HW extensions for predictive diagnosis Reconfiguration strategies (isolation of faulty function and shift of functions ) Degradation strategies (critical vs. non-critical functions) December 2017 Software Defined Vehicles VIRTUAL VEHICLE 18
PRYSTINE December 2017 Software Defined Vehicles VIRTUAL VEHICLE 19
ROS Introduction December 2017 Software Defined Vehicles VIRTUAL VEHICLE 20
ROS Introduction Robot Operating System (ROS) Open-source, meta-operating system for robots Originally developed by Stanford AI Lab and Willow Garage in 2007 Maintained by the Open Source Robotics Foundation (OSRF) Runs on top of e.g. Ubuntu/Linux Designed for many kinds of robots Provides tools for building/running code Including software libraries Hardware abstraction Allows for low-level device control Provides communication system December 2017 Software Defined Vehicles VIRTUAL VEHICLE 21
ROS Introduction ROS Communication Peer-to-peer communication Central service registration Supports UDP and TCP Advantages Flexible configuration Fast communication Simple distribution of functions on computation platforms December 2017 Software Defined Vehicles VIRTUAL VEHICLE 22
ROS Introduction Integration of common solutions Sensor data processing Library for 2D/3D image and point cloud processing Filtering, feature detection, Tracking on camera data Motion planning Support of multiple planning algorithms 3D simulation and visualization Dynamics, kinematics, sensors, Data transformation Coordination system transformations for sensor data December 2017 Software Defined Vehicles VIRTUAL VEHICLE 23
ROS Introduction ROS Simulation 3D environment Sensor simulation Laser scanner built in Camera available Extension possible Sensor simulation Vehicle simulation Position Orientation Speed Steering December 2017 Software Defined Vehicles VIRTUAL VEHICLE 24
ROS Introduction Driving Simulation Detailed street layouts Number of lanes Street markings Traffic management Traffic signs and signals Different weather conditions Sunshine, snow, rain, Sensor simulation Multiple sensor types available Komplexe Straßenverläufe Änderungen im Straßenverlauf https://vires.com/vtd-vires-virtual-test-drive/ Ändernde Umweltbedingungen Unterschiedliche Straßenbeschaffenheit Extension possible December 2017 Software Defined Vehicles VIRTUAL VEHICLE 25
AI Integration Simulation Exchangeability of simulation and real environment Replacement of sensor data format Software deployment can be freely chosen ROS Simulation ROS messages ROS Function Implementation Driving Simulation Sensor data ROS Converter Decision making Sensor data processing available Segmentation (AI function) available Behaviour generation Real world data HPC #1 HPC #2 ECU December 2017 Software Defined Vehicles VIRTUAL VEHICLE 26
AI Integration AI function control flow Standard Software - Read image - Copy to GPU memory - Read result - Send message call return result AI inference - Matrix operations (multiply and add) - Mass non-linear function (ReLU, sigmoid, ) CPU GPU ROS specific part can be implemented as usual December 2017 Software Defined Vehicles VIRTUAL VEHICLE 27
Our automated drive test vehicle ROS Deployment Use ROS function implementation in vehicle No change of SW needed Integrated sensors Six radar sensors Four long range, two short range 6 Cameras Front, two side, back One Mobile Eye Virtual Vehicle Automated Drive test vehicle Full vehicle control Steering, acceleration, brake, Virtual Vehicle AD test vehicle trunk December 2017 Software Defined Vehicles VIRTUAL VEHICLE 28
Active safety design using ML techniques December 2017 Software Defined Vehicles VIRTUAL VEHICLE 29
Motivation: Development process active safety systems Problem: Complexity and high variation of accidents Safety functions for every combination of accident type and cause Type: Cause: speeding, slippery road, etc First 50%: 26 types and causes of accidents Last 50%: 5287 types and causes of accidents Function design based an quantitative (e.g. DESTATIS) and qualitative accident databases (e.g. GIDAS Pre-Crash-Matrix) An exponential effort in the development of individual active safety systems is required to cover only significant increase in the addressing of accidents! December 2017 Software Defined Vehicles VIRTUAL VEHICLE 30
Function development based on real world data Method analysis: From the accident to the active safety system Accident recording by highly automated vehicles (SAE Level 3) [BMW] [Here] [Kostal] Backend 360 environment recognition High-precision digital maps incl. localization Driver monitoring Backend communication Data recording: Course of the ego vehicle and other traffic participants road geometry, driver behavior 1) Evaluation of active safety systems with recorded traffic data 2) Learning the function behavior of active safety systems directly from recorded data [Source: BMW and Virtual Vehicle, cooperative R&D project, 2017] December 2017 Software Defined Vehicles VIRTUAL VEHICLE 31
Vision: Optimization based on real world traffic data 360 sensors High-precision maps Driver monitoring Backend Project use case: Crossing pedestrian (75% of all pedestrian accidents) Generated pedestrian scenarios from the effectiveness analysis Crossing scenarios Total scenarios Accidents Training data 1840 242 Test data 1829 243 December 2017 Software Defined Vehicles VIRTUAL VEHICLE 32
Function development active safety system December 2017 Software Defined Vehicles VIRTUAL VEHICLE 33
Simulation results: False positives vs. speed reduction Variation by the algorithm evaluation: 1)Feature set(featurevariation) 2) Training data(reduced speed range pedestrian) 100 90 80 Random Forest 100 90 80 Neural Network Speed reduction Reference Implementation 73.7% Random Forest 83.4% Neural Network 92.0% Speed reduction [%] 70 60 50 40 30 Reference algorithm 20 RF RF (Extended Features) 10 RF (Reduced Training Data) RF (Extended Features / Reduced Training Data) 0 0 5 10 15 20 25 30 35 40 False positives Speed reduction [%] 70 60 50 40 30 Reference algorithm 20 NN NN (Extended Features) 10 NN (Reduced Training Data) NN (Extended Features / Reduced Training Data) 0 0 5 10 15 20 25 30 35 40 False positives December 2017 Software Defined Vehicles VIRTUAL VEHICLE 34
AI-based systems challenges to be faced Requirements engineering (Machine Learning algorithms) Identification of Key Performance Indicators for ML algorithms e.g. accuracy(fault tolerances but also mis-detection), speed, etc. HW requirements for different DNN structures Identification of relevant safety analyses according to ISO 26262 and ISO PAS 21448 Determination of safety measures, time tolerances for detection etc. How to measure code and structure coverage of DNNs? Selection of training and test data Work out criteria based on statistical methods, quality, and versatility SW unit tests and integration test Verification and validation of SW safety requirements December 2017 Software Defined Vehicles VIRTUAL VEHICLE 35
Summary Automated driving requires redundancy (SW and HW) Fail-operational architectures ISO PAS 21448 Minimum redundancy but maximum reliability Homogeneous redundancy vs. redundancy by diversity trade-off AI functions have already proven their usefulness Object detection and localization (segmentation) End-to-End driving vs. modular approach AI in function development ROS is already widely used Sound basis for rapid prototyping Seamless transition from simulation to real hardware ROS is a suitable development and test environment for AI functions December 2017 Software Defined Vehicles VIRTUAL VEHICLE 36
Thank you for your attention. Prof. Dr. Daniel Watzenig Email: daniel.watzenig@v2c2.at December 2017 Software Defined Vehicles VIRTUAL VEHICLE 37