Silicon radars and smart algorithms - disruptive innovation in perceptive IoT systems Andy Dewilde PUBLIC
Fietser in levensgevaar na ongeval met vrachtwagen op Louizaplein Het Laatste Nieuws 16/06/2017 Meisje (12) sterft na ongeval met vluchtmisdrijf, dader nog steeds op de vlucht Italian cyclist Michele Scarponi killed after being hit by van during training De Morgen 28/10/2015 2 The Telegraph 22/04/2017
28% of deadly casualties on cross roads are cyclists Truck (light) Truck Other Car 30% Pedestrian 18% Motorcycle 18% Cyclist 28% Motorcycle (light) 4% 3 Bron: BIVV statistisch rapport 2015
Cycling injuries in traffic accidents are increasing Road safety campaigns have positive impact on the other forms of travelling 4 Bron: BIVV Verkeersveiligheidsbarometer 2016
Dangerous situations in traffic for cyclist 1. Cannot see/hear cars 2. No separation of bike lane and motor lanes 3. Obstacles (corners, trucks, cars) block field of view 4. Blind spots of trucks at junctions 5.... 5
comfortable & safe transportation Radar technology and sensor fusion can help reduce casualties
Garmin Varia Bike Radar System detects approaching cars for cyclists Radar tech acquired from ikubu, crowdsourcing Radar on seat pole detects approaching cars (100+ m) Warning indicator on steering bar Number of cars Speed Flashing tail light to warn drivers 7
Radar based technologies available today that support cyclists Clearview intelligence M100BR Build in pavement surface radar Detects presence of bicycles Used to control intelligent traffic lightning Priority to bikers Optimal flow control 8
From 2018, CAP and EuroNCAP, will give extra points to manufacturers that include cyclist-detection systems Radar & camera Automatic braking 9
Radar + heat sensor equipped traffic light to optimize flow control Installed in London as experiment 10
Radar technology detects approaching cyclists and motorists and warns car drivers before opening doors Radar in the rear bumper Detect bicycles 10..15 m away Indication inside car light or door handle vibration 11
TNO smart bike Radar at the front, camera at back 12
OK, lets go for radar then... Radars are a good choice for people and object detection Accurate range, speed and angular information extremely robust, mm-wave radars operate at all weather conditions though... 13
radar cannot see lane markings and traffic signs
soda can might have disproportionately large radar reflection
... pity... maybe go for camera then? Radars are a good choice for people and object detection Accurate range, speed and angular information extremely robust, mm-wave radars operate at all weather conditions though... Cameras are good choice High resolution, Machine vision is more around, still... 16
camera is blinded by harsh lighting
camera is blinded by bad weather
No golden solution? That almost never happens ;-) Why not combine them? Radars are a good choice for people and object detection Accurate range, speed and angular information extremely robust, mm-wave radars operate at all weather conditions though... Cameras are good choice High resolution images, Very good angular resolution, Machine vision is more around, still... Sensor data fusion: ultra-reliable and robust detection/classification combine advantages of different sensors, compensating their respective disadvantages 19
Machine vision = imaging-based recognition True machine perception = multi-sensor based classification, interpretation and prediction
21
imec is the world-leading R&D and innovation hub in nanoelectronics and digital technology.
APPLICATION DOMAINS SMART HEALTH SMART MOBILITY SMART CITIES SMART INDUSTRIES SMART ENERGY SEMICONDUCTOR & SYSTEM TECHNOLOGIES DIGITAL TECHNOLOGY PLATFORMS CORE CMOS SENSOR TECHNOLOGY FLEXIBLE TECHNOLOGY NETWORKING DIGITAL PRIVACY & SECURITY SOFTWARE & DATA MANAGEMENT SKILLS PATTERNING LOGIC TECHNOLOGY TECHNOLOGY MEMORY TECHNOLOGY INTERCONNECT TECHNOLOGY 3D INTEGRATION OPTICAL I/O
Imec is an independent R&D center focus healthcare, smart cities and mobility, logistics and manufacturing, and energy Close to 3500 researchers Collaborations with industry and academia ~500 M revenue HQ in Belgium activities/offices in Netherlands, US, Taiwan, India, China, Japan 24
Radar and sensor fusion at imec 79GHz radar mmwave Sensing Programs SENDA Sensors & Data Fusion 140GHz radar
Radar: radio detection and ranging A radar is an object detection system uses radio waves to determine range, altitude, direction, or speed of objects
How does a radar work? High frequency carrier Commonly sends short pulses (not necessarily) Distance proportional to time delay between the time of transmission and time the reflection
Doppler effect When a target moves toward the incident wave it literally compresses the waves, resulting in a reflected frequency higher than the incident The opposite happens when the target moves away form the source
Radar resolution (smaller is better) Speed (improves with higher carrier) voxel Range (improves with wider bandwidth) Angle (improves with larger antenna) 24 GHz 77 GHz 79 GHz 140 GHz
3 imec radar programs: 79 GHz, 140 GHz, sensor fusion PHADAR WiSens SENDA 79 GHz, automotive, demo operational in lab, Infineon public partner 140 GHz, vital signs monitoring, TX 2016, TRX 2017 first sensor in 2018, Panasonic public partner Classification Prediction Sensor fusion Focus on machine learning algorithms Imec and off-the-shelf sensors, focus on vulnerable road users
fixed Radar evolution mobile yesterday automotive today tomorrow imec 79 GHz road safety imec 140 GHz with antenna-on-chip
mmwave sensors are extremely robust noise & vibration snow & rain fog smoke dust lighting heat dirt
mmwave sensors are very small small form factor including antennas
SENDA architecture SENSORS Radar PLATFORM CPU/GPU HW LEARNING Trained model based on data Recordings Camera CLASSIFICATION based on trained model
Machine learning & Sensor Fusion From Nvidia webp In the last years Machine Learning has been developed and applied mainly on video camera applications
Machine learning & Sensor Fusion Few cameras and other independent sensors From Nvidia webp Machine learning based on fully perceptive sensor- and datafusion systems more cameras LIDARs Others (Ultrasonics, GPS, etc) more radars
Machine Learning Not only target classification
Using machine learning, radar data features can be used for classification Pedestrians Bikers 38
Combining all data from sensor fusion quickly becomes challenging 39
... or was it?
Conclusions Imec s mmwave program focusses on radar and sensor fusion for a safer environment Radar technology will help create a safer environment self-driving cars, allow cyclist have a better situational view, infrastructure awareness Sensor fusion and machine learning needed build more intelligence into the systems improve performance in harsh environment Imec s mmwave sensing technology allows for high resolution, small form factor and low power low cost by using CMOS technologies performance gain using sensor fusion improved intelligence from machine learning
PUBLIC