ADAS COMPUTER VISION AND AUGMENTED REALITY SOLUTION

Transcription

1 ENGINEERING ENERGY TELECOM TRAVEL AND AVIATION SOFTWARE FINANCIAL SERVICES ADAS COMPUTER VISION AND AUGMENTED REALITY SOLUTION Sergii Bykov, Technical Lead TECHNOLOGY AUTOMOTIVE

2 Product Vision

3 Road To Autonomous Driving Source: Intel

4 Representation For The Driver Output is an extendable metadata which describes all the augmented objects/hints and supports natural features ON/OFF

5 Computer Vision and Augmented Reality Applications City Driving Pattern Active Park Search Pattern Recognition Maths Image Processing Augmented Navigation Next Gen of Adaptive Cruise COMPUTER VISION Arfificial Intelligence Physics Help in low visibility mode Signal Processing Signs recognition Infographics

6 Key Challenges Of Bringing AR In The Vehicle Usability augmented reality subsystem should not disturb driver as it is continuously observed Hardware limitations computational, power consumption, zero latency (HUD) Requirements for precise environmental model estimation for occlusion avoiding Dependency on inaccurate map and navigation data Distributed HW architectures, platform flexibility requirements High precision absolute and relative positioning requirements Components synchronization and latency avoidance Embedded memory usage limitations, different memory models Algorithms should be both configurable and efficient Specific rendering requirements, not covered by general purpose frameworks Variety of inputs under different platforms Out-of-vehicle simulation (does not support natural simulation like classical navigation)

7 Framework Concept

8 Augmented Navigation Structure We offer a Unique Solution capable to create Augmented, mixed visual Reality for drivers and passengers based on Computer Vision, vehicle sensors, map data, telematics, navigation guidance using Data Fusion technique. Sensors/CAN Telematics/V2X Automotive Cameras Navigation System/Map Data CVNAR Solution Vehicle displays Projection on wind shield Smart Glasses VR devices

9 CVNAR Features Road scene recognition and objects tracking Road boundaries Lane detection Vehicle detection and tracking Distance and time to collision estimation Pedestrian detection and tracking Facade recognition and texture extraction Road signs recognition Parking slot recognition Positioning Precise relative and absolute positioning Flexible data fusion and smooth map matching Automotive constrained SLAM Integration and Fusion Sensor data External positioner data (optional) External recognition engine integration Pupil tracking Telematics: V2I and V2V Natural Augmented Reality Basic vehicle data Lanes and road boundaries Road signs and cautions Navigation data and hints Facades highlights Parking places Narrow street infographics Street names and complex junctions boards POI and OEM specific information Highlights CPU, GPU; OS: Linux, QNX HW: Intel, NVidia, TI, Renesas Extrapolation engines for latency avoidance Machine learning and deep learning

10 Hardware Approach: Computer Vision Box Quick-install demonstration solution Platform for CVNAR (allows to be portable) Integration with Head Units Integration with vehicle networks Using of own sensors if needed Live data from vehicle: - CAN data, Sensors - Video stream Navigation data, preprocessed sensor data, etc. CVB ADAS CVNAR Engine Head Unit Video Stream with augmented objects CVNAR Engine CVB Data Layer Web Interface SW Update Configuration Diagnostic HUD/LCD Control/Settings

11 Hardware Approach: Automotive Stack Own scalable and robust Automotive Stack aimed to minimize time of project start and integration RTOS (OSEK, Micrium, mtron), Microcontrollers (Renesas RH850/V850) Hardware Abstraction Layer (HAL), Operation System Abstraction Layer (OSAL) Trace Server/Client, WatchDog, IPC, Drivers (SPI, I2C, UART, Timers, etc.), SW Update Pre-Integrated Vector CAN/Diagnostic Stack Vehicle Bootloader for Renesas Microcontrollers Domains and areas: System development and integration with Automotive Networks and ECUs Drivers development and peripheral support: Video Cameras, Automotive Sensors, external HW HW brings-up (BSP development) Automotive grade technologies supported by team: Networks: CAN, LIN, MOST, BroadReach, Ethernet RTOS: OSEK, mtron, Micrium, embos, mtron, QNX, Linux, VxWorks Microcontrollers: Renesas (RH850, V850), Freescale (Bolero, i.mx6), TI (OMAP, Jacinto, MSP430) CAN Stacks: Vector, KPIT, own tinycan Audio/Video processing Media Bus: LVDS, FPDLink-III, APIX2, USB, Ethernet, BroadReach

12 Perception Concept

13 Sensor Fusion: Data Inference Optimal fusion filter parameters adjustment problem statement and solution developed to fit different car models with different chassis geometries and steering wheel models/parameters. Features: Absolute and relative positioning Dead reckoning Fusion with available automotive grade sensors GPS, steering wheel, steering wheel rate, wheels sensors Fusion with navigation data Rear movements support Complex steering wheel models identification. Ability to integrate with provided models GPS errors correction Stability and robustness against complex conditions tunnels, urban canyons

14 Sensor Fusion: Advanced Augmented Objects Positioning Solving map accuracy problems Placing: Road model Vehicles detection Map data Position clarification: Camera motion model: Video-based gyroscope Positioner Component Road model Objects tracking

15 Sensors Fusion: Comparing Solutions Our solution Reference solution Update frequency ~15 Hz (+extrapolation with any fps) Update frequency ~4-5 Hz

16 Lane Detection: Adaptability and Confidence

17 Lane Detection: 3D-scene Recognition Pipeline Low level invariant features Single camera Stereo data Point clouds Structural analysis Probabilistic models Real-world features Physical objects 3D scene reconstruction Road situation 3D space scene fusion (different sensors input) Backward knowledge propagation from high levels

18 Lane Detection: Additional Information Features data base Low level screen (3D) features to refine position Points clouds Marking details, Road borders High level structural elements and real world objects Junctions, Facades, Signs, etc. Features Collection Existing map providers Real-time feature extraction and understanding from video sensors Satellite-view photos analysis Map database updates Routes offline processing and upload

19 Lane Detection: HD Map Potential Content Simplified and advanced geometry for roads, traffic lanes, lanes boundaries etc. Application Precise on-road vehicle positioning Different weather, traffic situations Map matching and Path planning Maneuver suggestions Cable navigation Junction assistance Possible junction maneuvers Traffic lights position Useful information: Road border geometry and type Traffic signs (position and type); Traffic lights (position and type); Type and quality of roadbed; Roadside POIs (gas station, store, café etc.); Any other additional features which can be useful for vehicle positioning or driver.

20 Lane Detection: Robustness in Normal Conditions Graphs show error in meters between recognized lanes (curved model) and recognized road marking (distance to detected features + features accuracy) in different distance range and for different road conditions. Figure 1. Regular weather, highway, slightly curved road with lane changes Figure 2. Rainy weather, highway, straight road with lane changes

21 Lane Detection: Robustness in Difficult Conditions Figure 3. Bright sun, highway with secondary roads, straight road with turns and lane changes Figure 4. Hard rain, highway, straight road with lane changes

22 Vehicle Detection Convolutional neural network for vehicle detection GPU Acceleration CUDA Running real-time on NVidia Jetson TK1 Inference speedup on embedded (TK1) GPU vs CPU is ~3x Training speedup on desktop GPU vs CPU is ~20x Classifier accuracy (about 50k, 960x540, ~55-60 deg HFOV): Positive: 99.65% Negative: 99.82% Size of detection down to 30 pix, detection range of about 60 m Figure Vehicle detection examples

23 Road Scene Semantic Segmentation Deep fully convolutional neural network for semantic pixel-wise segmentation Road scene understanding use cases: model appearance, shape, spatial-relationship between classes Inference speedup GPU vs CPU is ~3x Figure Road scene segmentation examples

24 Driver Monitoring: Haar/LBP Cascades Object Detection using Haar or LBP feature-based cascade classifiers is an effective method for face detection It is a machine learning based approach where a cascade function is trained from a lot of positive and negative images. It is then used to detect objects in other images. Applications can achieve consistent face detection in different environments (e.g. vehicle driver, etc.) Can be efficiently implemented for low power embedded devices Figure Haar features Speedups on mobile GPU vs CPU are 20x for Haar 3x for LBP Figure LBP features

25 HMI Concept

26 Rendering Component Structure CVNAR Rendering Head Up Display Client of rendering framework (controller) HMI/scene update commands Frame Renderer HMI Renderer Augmentation Renderer Frame Buffer rendered frames LCD Window System Resource Management Renderer System

27 Renderer Output Layers Frame renderer Augmentation renderer Final image HMI renderer

28 Augmented Objects Primitives. Part 1 Barrier Lane Line Traffic Sign & POI Lane Arrow

29 Augmented Objects Primitives. Part 2 Fishbone Facade Vehicle Street Name

30 Demo Application Screenshot (LCD)

31 Head Up Display Concept. HUD vs LCD Hardware limitation HUD devices are rarely available on market FOV and object size Timings Zero latency Driver eye position Driver perception Virtual image distance Information balance

32 Summary: Key Technology Advantages Proved understanding of pragmatic intersection and synergy between fundamental theoretical results and final requirements Formal mathematical approaches are complemented by deep learning Solid GPU optimization expertise Automotive grade solutions integrated with all the data sources in vehicle data fusion approaches High robustness in various weather and road conditions, confidence is estimated for efficient fusion Closed loops designed and implemented to enhance speed and robustness of each component Integration with V2X and Navigation system System architecture supports distributed HW setup and integration with existing in-vehicle components if required (environmental model, objects detection, navigation, positioner etc.) Hierarchical Algorithmic Framework design highly optimizes computations on embedded platforms Collaboration with scientific groups to integrate cutting edge approaches

33 THANK YOU Sergii Bykov Technical Lead