Autonomous Automation: How do we get to a Million Miles of testing? Jace Allen Business Development Manager Simulation, Test, and EEDM dspace Inc. 50131 Pontiac Trail Wixom, MI 48393 USA 1
Agenda 1. Intro to V&V for ADAS/AV/HAD Changing Environment of AV/HAD ADAS V&V Process and ISO26262 2. Testing Toolchain for ADAS/AV/HAD Models, Scenarios, and Sensors HIL Testing and Sensor Fusion Needs SIL Testing and Cluster Simulation 3. Testing Process and Autonomous Automation Testing Methods and Tools Real-time Testing and Observers Test Management and Automation Optimizing Testing
Challenges Testing autonomous driving in real traffic The real world is complex The real world is unpredictable The real world is hazardous Source: nik/pkb Unlimited number of real-life traffic scenarios Many unknown factors, human driver no longer as a fallback how to validate system robustness? Exponential growth in testing effort. hundreds of millions of test kilometers required. 4
Validate System Behavior with Simulation MBD Testing = Simulation at All Levels Advantages of simulation: reproducibility, test beyond performance/endurance limits and dangerous situations ISO 26262 recommends MIL/SIL/HIL simulation for conducting the software safety requirements verification What changes with testing Autonomous Vehicles? 5
ADAS/AV development process Requirements Specification System concept MIL (traffic simulation) Test drives in real traffic Homologation Driving simulator Test drives on prov. ground Prototyping ADAS HIL (closed-loop) SIL (closed-loop) Component HIL (closed-loop) Target implementation SIL (closed-loop) Algorithm ADAS/Sensor ECU(s) under Test 6 dspace-internal
ADAS/AV development process Requirements Specification System concept MIL (traffic simulation) Test drives in real traffic Homologation Driving simulator Test drives on prov. ground Machine learning Prototyping ADAS HIL (closed-loop) SIL (open-loop, data playback) SIL (closed-loop) Component HIL (closed-loop) Component HIL (open-loop) Target implementation SIL (closed-loop) SIL (open-loop) Algorithm ADAS/Sensor ECU(s) under Test 7 dspace-internal
ADAS/AV development process Requirements Specification Measurement data (Sensor and vehicle data) Cloud System concept MIL (traffic simulation) Driving simulator Test scenarios/cases Model-, parameter-, scenario management Test drives on prov. ground Test drives in real traffic Homologation Machine learning Prototyping ADAS HIL (closed-loop) SIL (open-loop, data playback) PC cluster SIL (closed-loop) Component HIL (closed-loop) Component HIL (open-loop) Target implementation Algorithm ADAS/Sensor ECU(s) under Test SIL (closed-loop) Cloud PC cluster SIL (open-loop) 8 dspace-internal
Challenges Changing validation process Vehicle, driver, sensors Road, road networks Traffic, roadside structures Environmental conditions (weather, ) Driving maneuvers Data & test management Models, parameters, tests, test results, Traffic scenarios (critical, representative, ) Scenario databases, Support of open file formats and standards Virtual ECUs Automated test execution, test evaluation and control Defined scenarios Stochastic parameter variation Test control to detect critical scenarios based on metrics and evaluation criteria Requirements-based testing not sufficient Real ECUs PC cluster 9
TESTING TOOLCHAIN FOR ADAS/AV/HAD
dspace Solutions: A Powerful ADAS Toolchain Simulate Models Test Scenarios Visualize? Maneuver Control & Experiment Test Management Tests 11
Automotive Simulation Models (ASM) for ADAS and autonomous driving Vehicle Dynamics Environment MotionDesk Animation Vehicle simulation Vehicle Dynamics Drivetrain Soft-ECU network Driver model Maneuver Road networks Roads and intersections Lane support Artificial/real world roads Road import Roadside structures Traffic Objects Static and dynamic objects Vehicles, trucks, pedestrians Traffic signs, traffic lights, parking vehicles, Environment Sensors 2-D/3-D sensors Camera, radar, lidar, Line, lane and traffic sign recognition Object list simulation Traffic ModelDesk Parameterization 12
Test Scenario Definition ASM Traffic Scenario definition ASM Simulated Traffic flow and Tool automation Independent definition of fellow scenarios Demo scenarios for standards (Euro NCAP) Open API for Automation Road Import OpenDRIVE, OpenCRG Measurement data/gnss Here/ADAS RP OSM, Google Earth Scenario Import Manually defined based on expert knowledge GIDAS database OpenSCENARIO (planned) ModelDesk API Stochastic, etc. 13
dspace HIL simulation technology for ADAS and autonomous driving Sensor models Road, environment, driving maneuvers Automotive Simulation Models (ASM) Virtual driver Real sensors and ADAS ECUs Soft ECUs Vehicle dynamics Brake/engine torque, steering angle, Real-time HIL simulator 14
dspace HIL simulation technology for ADAS and autonomous driving Sensor system Sensor models Road, environment, driving maneuvers Automotive Simulation Models (ASM) Virtual driver Real sensors and ADAS ECUs Soft ECUs Vehicle dynamics Real-time HIL simulator 15
ASM Traffic and sensor models HIL or VEOS Simulink Traffic environment Idealized sensor data Error model (Noise, false positive/negative detections, ) Sensor data with realistic errors ASM 2-D contour sensor 3-D object sensor Sensor or customer specific implementation Integration of sensor-specific error models in Simulink possible due to open Automotive Simulation Models (ASM) Attribute sensor 16
Network Management Options for testing Sensor ECUs Opt. 5: Simulation over-the-air (OTA) OTA-Device Option 1 & 2 Technology independent approach Provide ideal ground truth based information Part of ASM Calculated on SCALEXIO CN (CPU) Option 3 & 4 Physics-based approach More related to the measurement principle of a sensor Calculated on GPU Option 5 Test Bench with real Sensor ECU 1) Automotive Real-Time Radar Scene Generator Opt. 4: Insert raw data Opt. 3: Insert target list Opt. 2: Insert Object list Opt. 1: Restbus Vehicle network Sensor Frontend Preprocessing Raw data Detection, Data Proc. Target list Object tracking Object list Application logic (Trajectory planning, Motion control) Sensor ECU Confidential, Information are subject to change without notice
Network Management Options for testing Sensor ECUs Opt. 5: Simulation over-the-air (OTA) OTA-Device Sensor Frontend Sensor ECU Option Camera Radar Lidar Ultrasonic 1 2 3 4 Under development n/a n/a Under development 3D point cloud 5 ARSG 1) n/a n/a Opt. 4: Insert raw data Opt. 3: Insert target list Opt. 2: Insert Object list Opt. 1: Restbus Preprocessing Raw data Detection, Data Proc. Target list Object tracking Object list Application logic (Trajectory planning, Motion control) 1) Automotive Radar Scene Generator Vehicle network Confidential, Information are subject to change without notice
Raw Sensor Data Generation Environment Sensor Interface Unit Powerful high-end FPGA Xilinx Zynq UltraScale+ (MPSoC ZU9) Synchronous output for >8 sensors (Cameras/Radar 1 /Lidar 1 ) with 1 ESI Unit 2 Up to 15.9 Gbit/s aggregated data rate FPGA model (partly) open for customers (VHDL, Verilog, XSG, and HLS) FMC modules Flexible adaption of video interfaces via plug-in modules (FMC) Direct camera I/F (GMSL, FPD-Link III, Ethernet AVB 1 & GigE Vision 1 ) Customer-specific interfaces ESI Unit Plug-on Device (POD) for short range video interfaces parallel, HiSPI, CSI2, LVDS, etc. High dynamic range (HDR) support: Up to 20 bit 1 Feedback channel to ECUs (e.g. field of view, exposure time, ) from ESI Unit Both Open loop and closed loop testing with the ESI 1 Under Development 2 Depending on customer setup Confidential
Camera Image Sensor Lens Imager Image processing Bayer Pattern (RGGB) Register Interface I²C Source: https://en.wikipedia.org/ Source: http://www.onsemi.com/
ESI Unit: FPGA Firmware Overview Environment Sensor Interface Unit (Xilinx Zynq ZU9EG FPGA) Ethernet 4x ARM Cortex A53 Linux Pixel Pipeline Control HDMI Input Video Input Bayer Pattern Pixel Pipeline Aurora Gain FIU Aurora Output ESI POD Camera ECU Video Data Configuration & Control Confidential
ESI Unit: FPGA Firmware Overview Environment Sensor Interface Unit (Xilinx Zynq ZU9EG FPGA) Ethernet 4x ARM Cortex A53 Linux HDMI Input Video Input Pixel Pipeline Aurora Output Aurora ESI POD Camera ECU Video Data Configuration & Control Confidential
ESI Unit: FPGA Firmware Overview Environment Sensor Interface Unit (Xilinx Zynq ZU9EG FPGA) Ethernet 4x ARM Cortex A53 Linux Radar Pipeline GMSL Output GMSL Radar ECU HDMI Input Video Input Video Splitter Pixel Pipeline Aurora Output Aurora ESI POD Camera ECU Pixel Pipeline GMSL Output GMSL Camera ECU Radar Data Video Data Configuration & Control Confidential
Raw Data Generation for Cameras MotionDesk and multiple sensor models (Camera, Radar, Lidar) PREVIEW ModelDesk ASM Front camera Rear camera Control data Laser scanner Sensor composition combining multiple sensor outputs incl. meta-data ESI Unit HDMI Ethernet PC with graphics card and MotionDesk Rest bus simulation Vehicle network Confidential
Raw Data Generation for Cameras MotionDesk and multiple sensor models (Camera, Radar, Lidar) PREVIEW ModelDesk ASM Front camera Rear camera Control data Laser scanner Sensor composition combining multiple sensor outputs incl. meta-data CN+GPU SensorSim HDMI Ethernet ESI Unit Rest bus simulation Vehicle network Confidential
Product ECU spec. project Network Management Overview Option 3 & 4 HIL - today ESI Unit PODs FPGA Firmware Environment Sensor Interface Unit (FPGA) Environment Sensor Interface POD (FPGA) Sensor Frontend Preprocessing Sensor ECU Raw data Ethernet HDMI Detection, Data Proc. Target list High-end PC and GPU (Windows) MotionDesk Host Ethernet / IOCNet Environment Sensor Simulation Automotive Simulation Models (ASM, SCALEXIO CN) Object tracking Object list Application logic (Trajectory planning, Motion control) 29 Confidential
VEOS Realistic simulations of ADAS and automated driving functions on standard PCs 35
Sensor simulation HIL and SIL PREVIEW Integrated toolchain for SIL and HIL use cases Many sensor technologies are adressed Five options to test a sensor toolchain Toolchain with low latency and synchronization Use of several GPUs is planned API to integrate custom models is planned 38
TESTING PROCESS AND AUTONOMOUS AUTOMATION
AutomationDesk Testing and Test Tools Various methods of Test Development Signal-based testing, XML and xil-api Open Standards Manage all tools in the ADAS Testing Process fit for purpose for developing safety related software according to IEC 61508 and ISO 26262. pre-qualified for all ASILs according to ISO 26262 40
Real-Time Testing Standard PC Real-time test programming via Python scripts (with use of specific RTT Libraries) Real-Time Test management (download, start, stop, pause...) Real-time platform or VEOS Execution/Scheduling of Python real-time tests Synchronization between real-time tests and Simulink model 41
Observers and the RTT Observer Library Coupling of BTC Embedded Specifier and dspace Systems Requirements (textual) Test specification (informal, structured) EmbeddedSpecifier Observer Download Observer Passed/Failed Req-Coverage Simulator (VEOS / HIL) Textual requirements => Formalized requirements => Simulation based formal verification (~ISO26262) Permanent verification of safety-critical requirements with observers, e.g. in parallel to execution of classic AutomationDesk tests Drastic increase in test depth and coverage for safety-critical functions Many additional benefits for test automation users Requirements (formalized) RTT-Observer 42
Test Management for ADAS: Keeping Track of Testing Activities Traceability and Coverage Traceability from requirement to test result and overall requirements coverage For all types of requirements, e.g. safety, functional, performance or robustness requirements Test Scenario Traceability Which test scenario is tested by which tests? Has a given test scenario been tested successfully? Monitor progress across multiple test platforms and different test tools For tests by means of simulation (MIL/SIL/HIL) as well as real-world tests Test reports, results overviews and test evaluation Test Stimuli Traceability Which parameters and inputs were used for which tests during a test execution? 43
SYNECT Test Management Manage all MIL/SIL/HIL Testing 1. Requirement integration and coverage analysis - Connect to PLM/ALM Tools 2. Off-the-shelf integration with common test tools such as AutomationDesk, Simulink, and BTC EmbeddedTester 3. Open interface to connect other test tools or custom test solutions 4. Monitor, analyze, visualize test results during the test process 5. Dynamic Test Parameterization handle ADAS model/test functionality 44
Integrated Tool Chain for Testing by Means of Simulation Euro NCAP AEB Use Case Automatically execute Euro NCAP tests and generate score results Automated parameterization, execution and evaluation of Euro NCAP tests Example AutomationDesk NCAP AEB Test Demo available online Solutions for all NCAP tests available as an engineering service 45
Test Automation (TA) Framework Test Cases Test Management Project Test Steps Manager SYNECT Project Navigator 46
Test Automation (TA) Framework 47
PC cluster simulation with Virtual ECUs Driving millions of kilometers on your PC Testing at an early development stage Highly scalable due to virtual ECUs Deterministic and reproducible test execution High test throughput through fast test execution Simulation Cluster open to Integrate with test generation methods Cloud computing options Simulation Cluster leverages SIL tool chain in general (VEOS, xil-api, ASM) SYNECT Test Management Real Time Testing 50
Cluster Test Management with dspace SYNECT 51
Parallel Testing and Execution Management Plan and schedule test cases and assign to specific cores (efficiency of multi-core testing) Offload test analysis from Test resources (test resource efficiency) Future auto-optimization and API for schedule/sequence customization 53
Optimizing HIL Testing Time Master Data Manager SYNECT Execution Iteration Test Case - 1 2 mins 2 mins HIL Time HIL PC Test case 1 Test case 2 Test case 3 Captured Data Results EXECUTION Test case 4....mat /.mf4 1.mat /.mf4 2 Passed Failed 4 mins EVALUATION Offline PC Post-Processing Offline PC Iteration Test case 1 Test case 2.mat /.mf4 3.mat /.mf4 4 Passed Passed...... Test case 3 Test case 4... 54
SUMMARY
One Tool Chain for ADAS/AV Testing ISO 26262 ready. Prequalified for all ASILs 56
Sensors Algorithms dspace - The Right Partner for Autonomous Driving Prototyping Simulation Validation Virtual test drives Exhaustive testing 57 dspace-internal
Thanks for listening! Copyright 2017, dspace Inc. All rights reserved. Written permission is required for reproduction of all or parts of this publication. The source must be stated in any such reproduction. This publication and the contents hereof are subject to change without notice. Benchmark results are based on a specific application. Results are generally not transferrable to other applications. Brand names or product names are trademarks or registered trademarks of their respective companies or organizations. 58