Intelligent Vehicles and ADAS (Advanced Driving Assistance Systems) Ph. Bonnifait Lab Heudiasyc CNRS, Université de Technologie de Compiègne FRANCE

Transcription

1 Intelligent Vehicles and ADAS (Advanced Driving Assistance Systems) Ph. Bonnifait Lab Heudiasyc CNRS, Université de Technologie de Compiègne FRANCE 1

2 Outline 1. Intelligent Vehicles 2. Pedestrian detection, recognition and tracking using Lidar 3. Map-Matching integrity monitoring 4. Data fusion of Geographical Information and rough GNSS measurements 5. Wheel ground vertical contact force 2

3 Outline Intelligent Vehicles Research interests itrans UTC interests Pedestrian detection, recognition and tracking using Lidar Map-Matching integrity monitoring Data fusion of Geographical Information and rough GNSS measurements Wheel ground vertical contact force 3

4 Intelligent Vehicles Definition (R. Bishop s proposal 2005) IV systems Sense the driving environment Provide information or vehicle control To assist the driver in optimal vehicle operation IV systems operate at tactical level of driving Throttle, brakes, steering IV systems are beyond active safety systems (ABS, ESP) 4

5 Intelligent Vehicles key issues An IV is a vehicle able to perform driving assistance tasks autonomous navigation in the presence of uncertainty and variability in its environment Artificial Perception and Contextual Information Analysis are key issues Managing uncertainty in fusion processes is crucial for reliable perception 5

6 Confidence indicators IV embedded systems need to Fuse redundant information Estimate unobserved parameters Monitor themselves Fault detection and isolation Diagnosis Integrity tests Confidence indicators are useful for The fusion processes (Input) The use of the provided information (since often no high enough reliability can be reached) (Output) 6

7 i-trans competitiveness cluster Intelligent transportation systems Scope Rail, automobiles, logistics, coastal and international shipping, and inland waterways 7

8 I-Trans activities linked to IV Comité de Programme 4 Sécurité et Acoustique des Equipements Embarqués Sécurité active : communication inter-véhicules et avec l infrastructure Sécurité intégrée : pilotage du passif par l actif Aide à l anticipation pour la sécurité dans le véhicule, Aide à l anticipation pour la sécurité des piétons. 8

9 I-Trans activities linked to IV (in English!) Active and Passive Safety Integrated Safety Systems Vehicle Environment Perception Cooperative Vehicle-Highway Systems Collision Avoidance (pedestrian detection) 9

10 Heudiasyc scientific interests ADAS - Advanced Driver Assistance Systems Techniques for Man-Machine cooperation assessment Perception State Observation of dynamic systems Multi-sensor fusion in a dynamic context Ego-localization using on-board sensors and GNSS associated with GIS information Dynamic behaviour (tire/road contact characterisation) 10

11 CARMEN GPS receiver (PolarX Septentrio) Front scene camera (Sony camera) CAN bus Wheel Speed Sensors Yaw rate gyro 4-layer Lidar (IBEO Alaska XT) 11

12 Outline Intelligent Vehicles Pedestrian detection, recognition and tracking using Lidar Objectives Method Results Map-Matching integrity monitoring Data fusion of Geographical Information and rough GNSS measurements Wheel ground vertical contact force 12

13 Perception objectives Obstacles detection and tracking in driving situation Pedestrians recognition Confidence indicators management Detection Recognition Tracking 13

14 Sensor consider here Object level fusion module 14

15 Object detection Four plane laser sensor Detecting ground Clustering 15

16 z For each detected object Object Detection Confidence z max P d = ( ω1234n ω123n123 + ω + ω N + ω N + ω N ) / N N 234 y (a) z min x N = Round arctan πα 2 L D Layer 4 Layer 3 Layer 2 Layer 1 1 ω ω ω234 0 (b) Pedestrian Recognition Confidence P r 1 ω23 ω ω3 0 1 L(cm) 0 (c) Width based Recognition function 16

17 Track s updating 17

18 Confidences management Method: belief functions To combine tracks To manage objects detection and recognition confidences Algorithm transform the detection and recognition probabilities into believe functions : basic believe assignment (bbas) Combine these dependent bbas with a cautious rule. Combine conjunctively with the associated track s confidence Calculate track s detection and recognition confidences 18

19 Confidences management Object Detection Object Recognition Track Detection Track Recognition s time 19

20 IEEE IV 2008 demo Projection on the image of the 4 scanning layers of a Lidar If the recognition confidence of the lidar-only-track > threshold Projection of a corresponding rectangle (2 meters high) Plot of lateral bars to represent the confidences 20

21 Extrinsic Calibration between a Multi-layer Lidar and a Camera 3D pose estimation of the calibration target from each sensor data. 3D scan data (4 layers) Images Z t l Pij Z c Y c X c Estimation of the relative sensor position 3D Robust Registration of different poses of the target Y t X t l C, c C X l Z l Calibration accuracy estimation based on registration residuals Y l 21

22 Lidar-only pedestrian detection/recognition on-board display (no gate on the confidence threshold) 22

23 Lidar-only pedestrian detection/recognition on-board display (confidence threshold=95%) 23

24 Architecture IBEO Alasca XT (12.5 Hz) Sony CCD Camera (15Hz) Data acquisition Obstacle detection Pedestrian recognition Tracking objects Confidence indicators computation Selected pedestrian list ROI projection into image Win32 Calibrated parameters Display 24

25 Outline Intelligent Vehicles Pedestrian detection, recognition and tracking using Lidar Map-Matching integrity monitoring Use of MM POMA/CVIS FP6 Integrated projet Method results Data fusion of Geographical Information and rough GNSS measurements Wheel ground vertical contact force 25

26 Map Matching - Definition map «map-matching» : determining the vehicle s position % a digital road database 26

27 What is the use of Navigable Maps? Curve speed warning/control Adaptive light control Speed limit assistant Path prediction Power train management, Fuel consumption optimization Vision enhancement Map aided ADAS ACC, LKA, LCA, collision driving, autonomous driving 27

28 Main functions of the position calculation process in POMA EGNOS Receiver Infrastructure Positioning Module Hybrid Solution GNSS-based Positioning Hybrid Positioning Map Matching Map Matched Solution GNSS Receiver DR Sensors Map Database Update Service 28

29 Modern Map-Matching Outputs MM outputs : up to 10 matched candidates Each candidate (Map-Matched hypothesis) Probability with respect to the others NIS Normalized Innovation Squared Very often it is the hypothesis with the maximum probability that is used: for navigation tasks or fleet management applications it is acceptable. But for many other applications, like ecall, Pay as you drive or Map-Aided ADAS, it is important to manage all the hypotheses. 29

30 Integrity and Localization Systems Integrity of a localization system: the measure of confidence that can be accorded to the exactitude of the positioning delivered by this system. Usual scheme apply successive checks to ensure that the input information is valid detect and eliminate aberrant measurements internal reliability estimate a positioning with a quantified inaccuracy. external reliability 30

31 xpl Protection Levels Maximum error due to an undetected fault Approximate Radius of Protection (ARP) 31

32 Integrity of a map-matching system Map-Matching Integrity definition (proposal) A multi-hypothesis map-matching process is reliable (or safe) if the ground truth matched location is within the hypotheses zones provided by the system. Real unknown matched position Candidate matched zones Real unknown matched position Candidate matched zones Safe Unsafe 32

33 How to characterize the localization system integrity in real-time? The Real Map-Matched position is Unknown! Our proposal Multi-Hypothesis Map-Matching (MHMM) Estimate the probability of each hypothesis with respect to the others Compute Normalized Residuals for each hypothesis Apply a decision rule (depending on the application) 33

34 Monitoring integrity using MHMM outputs Estimated position Candidate matched position Most likely candidate Estimated position Candidate matched position Most likely candidate Case 1 : confident MM Case 2 : ambiguous MM Estimated position Candidate matched position Most likely candidate Case 3: inconsistent candidates 34

35 Bayesian MHMM using Road Tracking 35

36 MOMKF Multiple Observation Models Kalman Filter GPS x z t t = f ( xt 1 = o( x t ) ) + + α β t t Proprioceptive sensors Multi-hypothesis Map-matching Hypotheses Estimation/ Selection Roads Selected Hypotheses Map observations are fused with the states 36

37 MOMKF Illustration Δ Δ : threshold for safe duplication 37

38 2 e technique: Multiple Model Particle Filter 38

39 39

40 40

41 Resampling 41

42 Managing the NIS of MHMM Criterion: heading + distance (for each hypothesis) 2 degrees of freedom Gaussian hypothesis NIS should follow a Chi Squared distribution Decision rule: compare each NIS with a Threshold Decision threshold depends on the probability of False Alarm Decision rule: accept hypothesis i if NIS(i)<chi2inv(2,P FA ) P FA = 10% Th = 4.6 P FA = 1% Th = 9.2 P FA = 0.1% Th = 13.8 P FA = 0.01% Th =

43 IEEE IV 2008 demo Map display -The yellow square and the arrow are the position and the heading of the car -The 4 red triangles correspond to the hypotheses -The brackets represent the confidence interval of the track Ambiguous Confident Historic of the most probable hypothesis Unsafe Reliable 3 other hypotheses Most probable hypothesis Safety level of the output : the lower the PFA (Probability of False Alarm), the more reliable the hypothesis 43

44 MM Replay (Real Data in Compiègne) 44

45 Architecture Septentrio GPS 10Hz CAN gateway Odometry (25Hz) & gyrometer (100Hz) Data acquisition Map server (TCP/IP) Win32 Positioning EKF Linux Display Map matching +Integrity Indicators Particle filter Two Map servers -NavTeQ -TeleAtlas 45

46 Outline Intelligent Vehicles Pedestrian detection, recognition and tracking using Lidar Map-Matching integrity monitoring Data fusion of Geographical Information and rough GNSS measurements GPS drawbacks Approach Results Wheel ground vertical contact force 46

47 GPS drawbacks in urban areas - Bad visibility - Satellite masked by high rise buildings - Bad satellite configuration - Urban canyons - Multipaths - Reflexion on Non Line Of Sight (NLOS) satellites - With less than 4 satellites, it is impossible to fix a point 47

48 On the use of rough GNSS measurements Often in a urban canyon, 1 or 2 satellites are still visible Idea: To use a tightly coupled approach for the data fusion process Pseudo-ranges ρ i c = R i + c dt Doppler measurements Phase measurements u ρ i c = h i ( x, y,z,dt u ), i =1, L, n 48

49 Tightly coupled GNSS/Map localization GNSS sensor Raw data Proprioceptive sensor 1 Proprioceptive sensor 2 Sensor data fusion and map-matching road database: map Position on the map Relevant attributes of the road segment 49

50 Tightly Coupling GPS and 2D map data terrestrial ellipsoid Z Map Plan constraint Segment O WGS84 Y X reference 50

51 Static Localisation Rough GPS data Map Static localisation -Variable elimination -Weighted Least Squares 51

52 Dynamic Localisation Initial localisation Dynamic localisation Gyro + odo Rough GPS data Map Pose tracking Kalman Filtering 52

53 Our European Maps! 53

54 How to handle GPS and Map troubles GPS: multi-paths and interferences Maps: Inaccurate (bias), obsolete, rough representation of the reality, Ambiguous at junctions Proposed strategy: To rely on dead reckoning and consider that integrity problems come from the GPS data and/or form the map Implementation: Mono-hypothesis road tracking Map used as a heading observation Integrity tests on pseudo-ranges and on the map observation (NIS Normalized Innovation Squared and Ch2 threshold) 54

55 Experimental results End L1 GPS receiver Start Odometer Gyro 2D map 55

56 Multiple models observation Kalman filter (real data) Only 7 states Kalman Filter 56

57 Advantages of tightly coupling Tightly fusion of GNSS and Map has many advantages Use of few satellite in LOS Possibility to apply efficient integrity tests Map-Matching is a sub-product of the method 57

58 Outline Intelligent Vehicles Pedestrian detection, recognition and tracking using Lidar Map-Matching integrity monitoring Data fusion of Geographical Information and rough GNSS measurements Wheel ground vertical contact force Problem State observation Results 58

59 Application of vehicle wheel-ground contact normal forces: Automatic detecting risk rollover situations 59

60 Problem Usual measurements: accelerations, roll rates, suspensions deflection, Missing important information Dynamic variables: roll angle, tire-road forces high costs sensors -> estimation 60

61 Objective Replace wheels transducers by virtual sensors (observers) + Tire-road forces + accelerometer, gyrometer, + k p k k k Modelling X f X U + 1, =, dt+ X X k+ 1= X k+ 1, p K Observer ( Yc Y k) 61

62 What is the use of the knowledge of tire forces? As a result of longitudinal and lateral accelerations, the load distribution in a vehicle changes during a journey. roll pitch Force Fz Normal tire-road forces: Improvement of safety systems (ABS, ESP) Influence steering behavior, vehicle stability and cornering stiffness Better calculation of the LTR (Load Transfer Ratio) rollover index parameter 62

63 LTR parameter definition LTR: LTR=(Fzr-Fzl)/(ΣFz) Convenient method for supervising the vehicle s dynamic roll behavior - LTR Lift-off of the right wheels No load transfer Lift-off of the left wheels 63

64 Estimation process Measurements Observer 1 (LKF) Roll plane vehicle model Lateral load transfer Observer 2 (EKF) Nonlinear wheel ground vertical contact force model Vertical forces, LTR 64

65 Some experimental results Front left vertical tire force Fz fl (N ) 6000 measured estimated Front right vertical tire force Fz fr (N ) 65

66 Outline Intelligent Vehicles Pedestrian detection, recognition and tracking using Lidar Map-Matching integrity monitoring Data fusion of Geographical Information and rough GNSS measurements Wheel ground vertical contact force 66

67 Conclusion Key technologies for fully automated vehicles Surround sensing Robust lane/road detection Drive-by-wire for electric actuation Car2Car communication Communication with traffic operation center Operation on dedicated lanes 67

68 Thank you for your attention! Philippe Bonnifait Professor at Lab Heudiasyc CNRS, Université de Technologie de Compiègne FRANCE 68