Next-generation automotive image processing with ARM Mali-C71 Chris Turner Director, Advanced Technology Marketing CPU Group, ARM ARM Tech Forum Korea June 28 th 2017
Pioneers in imaging and vision signal processing 2
What is an ISP? The Image Signal Processor (ISP) processes raw pixel data into great-looking images for display, encoding and further processing by computer vision algorithms 3
Automotive needs: viewability and reliability 4
Automotive needs: viewability and reliability 5
Automotive needs: viewability and reliability Pedestrian protection system Driver drowsiness detection Night vision Low latency Reliability Resistant to random and systematic faults Imagery feeds into computer vision engine Viewability Dynamic range Low-light and low-noise performance Imagery close to human eyes and at times beyond 6
Automotive ADAS growth Camera volume rising very rapidly, around 20% CAGR By 2023 (expected average): Mid-range (high volume) cars: three cameras Luxury sedans: <ten cameras 7
Advanced vision requires centralised processing ISP per camera sensor Less flexible Higher BOM costs Centralised sensor fusion SoC Centralised processing Flexible architecture Sensor fusion SoC 8
And there are standards!! A number of functional safety standards exist: ISO 26262 Road vehicles IEC 61508 Electrical, electronic programmable systems DO 254 Electronics that fly e.g. in airplanes, helicopters 9
ISO 26262-2011 road vehicles functional safety Relates to electrical and electronic systems used in automobiles Qualitatively assesses risk of hazardous operational situations Aims to avoid or control systematic failures and detect or control random hardware failures, or mitigate their effects 10
ISO 26262: goals Provides an automotive safety lifecycle and supports tailoring the necessary activities during these lifecycle phases Covers functional safety aspects of the entire development process Provides an automotive-specific risk-based approach for determining risk classes (Automotive Safety Integrity Levels, ASILs) Provides requirements for validation and confirmation measures to ensure a sufficient and acceptable level of safety is being achieved 11
High-performance ISP for automotive: ARM Mali -C71 Ultra WDR Multi-camera input Designed for safety Ultra-wide dynamic range 24 stops Every detail captured Simultaneous support for human display and computer vision Virtual ISPs: <4 real-time camera inputs Pleasing and natural for human vision; predictable and reliable for computer vision Every pixel reliable Real-time safety for ADAS applications Enabling system-level ASIL D and SIL3 compliance 300+ fault detection circuits, built-in self-test (BIST), CRC on data paths, every pixel tagged 12
Mali-C71 block diagram Image sensor sources Colour/data plane outputs 13
Enabling system level ASIL D compliance Random faults Systematic faults Safety features 300+ dedicated fault detection circuits Built in continuous self-test Fault interrupt controller Pixel consistency image plane tagging reliability of every pixel Detection of sensor and hardware faults CRC checks to all data paths including memories and configurations Processes Developed within robust requirements, tracing and validation framework Safety manual Failure modes and effects analysis (FMEA) Development interface report Providing support for system level: SIL 3 / IEC 61508 ASIL D / ISO 26262 With a standards-agnostic approach 14
ARM Mali-C71 Image Signal Processor Designed to enable system-level certification Safety Element out of Context (SEooC) ISP has no direct control authority System integrating ARM s ISP needs to meet the requirements ISO26262 (ASIL D) IEC 61508 SIL3 Standard product Integrated into system (SoC) by our customers Specific use cases unknown to ARM Has to detect faults, perform self-tests Has a responsibility to provide timely, accurate and reliable information to the system Required for the system to be safe Road vehicles functional safety This flows down to the components of the system 15
Supporting output for human and machine vision Human display Computer vision 16
Ultra-wide dynamic range 2^24 2^8 Highlights clipped Mid-tones not revealed 0 2^8 Shadows details lost Preserved highlights As perceived by the human eye Enhanced mid-tones 0 0 Revealed shadows 17
Dynamic range management: iridix More than just the world s best local tonemapping engine Precise model of human retina contrast adaption Precisely simulates illumination by white light Basis of Nikon D-Lighting (2005) Shipped in over 2 billion devices iridix lets cameras see like the human eye 18
Dynamic range iridix engine: pixel-by-pixel tone mapping Contrast in highlights enhanced Contrast in midtones preserved iridix calculates a unique tone curve for each pixel of each frame Contrast in shadows enhanced 19
Management of dynamic range 20
Management of dynamic range 21
A complete, high-performing solution Performance Software Tuning 1.2 Giga-pixels/sec of processing performance Supports up to 4 real-time camera inputs and 16 camera streams with a single pipeline Control the ISP, sensor, auto-white balance and auto-exposure Roadmap to complete automotive software designed for ASIL compliance Full set of tools provided Ecosystem support for tuning and bringing up specific use-cases and sensors 22
Mali-C71: the next step in automotive vision First automotive ISP with ultra-wide dynamic range Supports both human display and computer vision Enables safer, more comfortable driving 23
The trademarks featured in this presentation are registered and/or unregistered trademarks of ARM Limited (or its subsidiaries) in the US and/or elsewhere. All rights reserved. All other marks featured may be trademarks of their respective owners. Copyright 2017 ARM Limited