Visione per il veicolo Paolo Medici 2017/2018 02 Visual Perception
Today Sensor Suite for Autonomous Vehicle ADAS Hardware for ADAS
Sensor Suite Which sensor do you know? Which sensor suite for Which algorithms are required for 1. Driving in the desert 2. Driving in simplified urban environment 3. Driving in normal urban environment
Sensor suite Sensor Suite for Autonomous driving: Stanley Boss BRAIVE VIAC KIT Google Tesla LIDARs for Autonomous Driving ADAS Cameras for Autonomous Driving
Stanley - 5 LIDAR - 1 camera - 1 RADAR - GPS/IMU http://isl.ecst.csuchico.edu/docs/darpa2005/darpa%202005%20stanley.pdf
Boss
Boss http://www.fieldrobotics.org/users/alonzo/pubs/papers/jfr_08_boss.pdf
Boss Applanix POS-LV 220/420 GPS/IMU (APLX) Submeter accuracy with Omnistar VBS corrections Tightly coupled inertial/gps bridges GPS outages SICK LMS 291-S05/S14 LIDAR (LMS) 180/90 deg 0.9 deg FOV with 1/0.5-deg angular resolution 80-m maximum range Velodyne HDL-64 LIDAR (HDL) 360 26-deg FOV with 0.1-deg angular resolution 70-m maximum range Continental ISF 172 LIDAR (ISF) 12 3.2 deg FOV 150-m maximum range IBEO Alasca XT LIDAR (XT) 240 3.2 deg FOV 300-m maximum range Continental ARS 300 Radar (ARS) 60/17 deg 3.2 deg FOV 60-m/200-m maximum range Point Grey Firefly (PGF) High-dynamic-range camera 45-deg FOV
The BRAiVE sensing technology Front sensing 4 cameras (2 graylevel, 2 color)
The BRAiVE sensing technology Lateral sensing
The BRAiVE sensing technology Rear sensing
The BRAiVE sensing technology Back sensing Stereo cameras
BRAiVE all-round vision coverage
The BRAiVE sensing technology Single plane laser scanners 2 frontal, 1 backward
The BRAiVE sensing technology Multiplane laser scanner
The BRAiVE sensing technology 16 Laser beams
The BRAiVE sensing technology DGPS + IMU
BRAiVE s processing BRAiVE s data processing is performed by 4 PCs Each PC is in charge of specific sensing areas One PC is in charge of vehicle control
VIAC The Sensing Suite 7 cameras 4 laserscanners GPS V2V radio + Additional devices
KIT Two stereo rigs (1392 512 px, 54 cm base, 90 opening) Velodyne HDL-64E laser scanner GPS+IMU localization
KIT 2 PointGray Flea2 gray scale cameras(fl2-14s3m-c), 1.4 Megapixels, 1/2 Sony ICX267 CCD, global shutter 2 PointGray Flea2 color cameras(fl2-14s3c-c), 1.4 Megapixels, 1/2 Sony ICX267 CCD, global shutter 4 Edmund Optics lenses, 4mm, opening angle 90, vertical opening angle of region of interest (ROI) 35 1 Velodyne HDL-64Erotating 3D laser scanner, 10 Hz, 64 beams, 0.09 degree angular resolution, 2 cm distance accuracy, collecting 1.3 million points/second, field of view: 360 horizontal, 26.8 vertical, range: 120 m 1 OXTS RT3003inertial and GPS navigation system, 6 axis, 100 Hz, L1/L2 RTK, resolution: 0.02m / 0.1
All heights wrt. road surface 1.60 m Wheel axis (height: 0.30m) 0.06 m 0.54 m All camera heights: 1.65 m Cam 1 (gray) 0.06 m Cam 3 (color) Cam-to-Cam Rect Velodyne laserscanner & CamRect (height: 1.73 m) Cam 0 (gray) Cam 2 (color) z 1.68 m 0.80 m -to-image x y x 0.27 m z y Velo-to-Cam IMU-to-Velo GPS/IMU x (height: 0.93 m) 0.81 m 0.05 m z y 0.32 m 0.48 m 2.71 m Figure :Sensor Setup. dimensions and mounting positions of the sensors (red) with respect to the vehicle body. Heights above ground are marked in green and measured with respect to the road surface. Transformations between sensors are shown in blue. Jan 12, 2016 CSC 2541: 01-Introduction
What s the problem of using so many sensors? One has to Calibrate and Registered them Different 3D locations Different capture times Different types of capture: instantaneous vs scanning
Google Car
Google Car Play Video
Velodyne LIDAR
Velodyne HDL64 LIDAR
Different Velodyne LIDARs
IBEO LIDARs
Tesla Sensor Suite
Tesla Sensor Suite A forward radar A forward-looking camera 12 long-range ultrasonic sensors positioned to sense 16 feet around the car in every direction at all speeds GPS A high-precision digitally-controlled electric assist breaking system Autopilot is on the Market on Model S
ADAS Active Park Assist Lane Departure Warning Traffic Sign Recognition Adaptive Cruise Control / ACC Stop & Go Forward Collision Warning / Emergency Breaking Blind Spot Detection Intelligent HeadLamp Control Pedestrian Detection
ADAS
ADAS
ADAS
ADAS
ADAS
Camera Sensor Resolution Pitch Size Technology Sensitivity Lens FOV Aperture Automotive Hardware problem
Autonomotive Hardware AEC-Q100 standard Operating temperature Grade 0: -40 C to +150 C Grade 1: -40 C to +125 C Grade 2: -40 C to +105 C Grade 3: -40 C to +85 C Grade 4: 0 C to +70 C Storage Temperature (higher!) Mechanical Shock Vibration
CCD Good Sensitivity Optical Blooming!
CCD vs CMOS
Rolling Shutter
Rolling vs Global
Rolling Shutter
Rolling Shutter
Rolling Shutter Each image row (pixel) has different time Pixels of dewarped image have a complex time equation Precise disparity on rectified image is impossible
Aperture vs Shutter f/ lens aperture: Depth of Field Light = Lens Aperture = proportional to 1/f 2 Shutter (Exposure) time: light acquired by pixel proportional to shutter
Light Conversion Shutter: light vs motion blur light α shutter Aperture: light vs depth of field light α 1/f 2 Pixel size (Pitch size): light vs resolution light α pitch 2 Sensitivity/Capacity
Dynamic Range problem DAY: >10^5 lux NIGHT: <10^-1 lux Dynamic Range: ~120db
Dynamic Range 8 12 bit ADC: 8 bit 256:1 48db 10bit 1024:1 60db 12bit 4096:1 72db... - 20bit 1M:1 120db Hardware vs Multiple Shot Non Linear Mapping Local Mapping (ToneMapping )
HDR Hardware
HDR MultiShot Images copyright Vislab/Ambarella Inc.
HDR Blending X1 =X1+noise X2 =X2+noise A = T2/T1 X1 = A * X1 X =X2 *f2+x1 *f1
Video Sensors Good Image Quality: High Sensitivity over a wide Spectrum and Wide Dynamical Range broad temperature range: -40degC... +105degC Some applications also require Color Global Shutter = Expensive! Rolling Shutter = Distortion!
WindShield distortion Lens distortion model is radial Windshild distortion is not radial! Spline?
Additional Issue Thermal stability: lens parameters, calibration parameters can change during time due to temperature changes. Real Time Calibration? Autocalibration
Hardware for ADAS Last Challenge: Energy Efficiency AlphaGo: 1920 CPUs and 280 GPUs, $3000 electric bill per game on mobile: drains battery on data-center: latency? increases TCO