MATLAB 및 Simulink 를이용한운전자지원시스템개발

Transcription

1 MATLAB 및 Simulink 를이용한운전자지원시스템개발 김종헌차장 Senior Application Engineer MathWorks Korea 2015 The MathWorks, Inc. 1

2 Example : Sensor Fusion with Monocular Vision & Radar Configuration Monocular Vision installed on Wind Shield Radar installed in Front Grill High Level Sensor Fusion 2 Different Scenario Closest In-Path Vehicle Selection Stationary Vehicle Detection 2

3 Sensor Fusion Why Sensor Fusion? To complement weakness of individual sensor Radar More Coverage Camera Accurate Range Poor Measureable Directly Range Rate Estimated Poor Angle Resolution Relatively Good Not Available Classification Available 3

4 Issues? Communication Risk of Vehicle Test HOG KLT Tracking SGM TTC LQR/LQG AUTOSAR RCS Max Doppler Interference Huh!! Noise figure 4

5 How do we bridge the gap between engineers who speak different languages and reduce risk? Communication Risk of Vehicle Test HOG KLT Tracking SGM TTC LQR/LQG AUTOSAR RCS Max Doppler Interference Noise figure 5

6 Solution: Model based Multi-domain Modeling and System Level Simulation Here are the specs Here s the verification model Model serves as the executable specification and test environment Multiple engineering teams can collaborate more effectively 6

7 Example : Sensor Fusion with Monocular Vision & Radar Vision Lane Detection Vehicle Detection Extended Kalman Filter based Tracking Radar FMCW Radar Modeling Extended Kalman Filter based Tracking Sensor Fusion Data Association Data Fusion 7

8 Abstract Vision Lane Detection Vehicle Detection Etc. Radar FMCW Radar Modeling Tracking Extend Kalman Filter based Tracking Sensor Fusion Integration 8

9 Lane Detection ROI (Region Of Interest) ROI (Region Of Interest) Cropped Image Captured Image Edge Mapping to World Selecting Candidate Pts Curve Fitting 9

10 Lane Detection ROI (Region Of Interest) MATLAB Built-In Function CVST Function Image Processing Function ROI (Region Of Interest) Cropped Image Captured Image Easy Matrix Calculation Code Generation Easy to Debug Scripting Language Pixel level, 2D, 3D Visualization Edge Mapping to World Selecting Candidate Pts Curve Fitting 10

11 Lane Detection 14

12 Abstract Vision Lane Detection Vehicle Detection Etc. Radar FMCW Radar Modeling Tracking Extend Kalman Filter based Tracking Sensor Fusion Integration 15

13 Cascade Detector Applications in Automotive of Cascade Detector Vehicle detection Pedestrian detection Traffic Sign Recognition Difference between applications is Feature. 16

14 4 Easy Steps to Create a Car Detector Collect Car Image Data (Training Images) Label Objects of Interest Train Object Detector Test Detector Performance 17

15 Step 1: Collect Training Image Data Collect many images that represent object you want to detect Ensure that images represent different conditions Lighting Different variations of object size, color, model etc. 18

16 Step 2: Label Training Images Select and assign regions of interest Images from: 19

17 Step 3: Train Object Detector Single line of ML code 20

18 Step 4: Test Detector Performance 21

19 Comparison Data from Vehicle Dynamics S/W and Result of Cascade Detector 22

20 Further Information about Cascade Object Detector See documentation for detail explanation 23

21 Abstract Vision Lane Detection Vehicle Detection Etc. Radar FMCW Radar Modeling Tracking Extend Kalman Filter based Tracking Sensor Fusion Integration 24

22 Camera Calibration App. Original image Undistorted image Camera Calibration App. MATLAB commands for camera calibration are also available. 26

23 Camera Calibration App. Intrinsic Matrix Tangential Distortion K = f x 0 0 s f y 0 c x c y 1 c x, c y - camera center in pixels (f x, f y ) - focal length in pixels per world unit f x = f p x, f y = f p y s sensor axes skew coefficient s = f y tan α x tangential = x radial + 2p 1 xy + p 2 r 2 + 2x 2 y tangential = y radial + p 1 r 2 + 2y 2 + 2p 2 xy Radial Distortion x radial = 1 + k 1 r 2 + k 2 r 4 + k 3 r 6 x y radial = 1 + k 1 r 2 + k 2 r 4 + k 3 r 6 y x and y are in normalized image coordinates in world units relative to the optical center 27

24 Image Processing Image Enhancement Deblurring Noise Removal Image Segmentation Active Contours Watershed 28

25 Feature Detection, Extraction and Matching Edge Corner Template SURF MSER Image Matching Feature Point Tracking Video Stabilization 29

26 Stereo Vision (2014a) Stereo calibration Semi-global disparity matching Better results than block matching 3D scene reconstruction from disparity New demos available Estimate real distance to objects in video 3D scene reconstruction Pedestrian detection (2014b) 30

27 Stixel based Object Detection Available to detect objects which are not easy using other sensors Stationary object, overlapped object, kid. * Stereo and City, Franke Uwe, Daimler Research Center 31

28 Stereo Vision Demo Computer Vision System Toolbox Stereo camera calibration SGM based Stereo matching 3D reconstruction using camera calibration data Kanade-Lucas-Tomasi point tracking 2D/3D plotting Bird View Disparity Image Stixel Representation 3D Reconstruction KLT Point Tracking 32

29 Point Cloud Processing (2015a) pointcloud object Fast search using kdtree Tree is constructed implicitly when a search is called first time Downsampling Two method : gridaverage, random Visualization Rigid registration Iterative Closest Point Algorithm 33

30 OCR Optical Character Recognition (2014a) Highly requested in Frontlines Support for English, Japanese Users can download additional language support Shipping demo with text detection and OCR workflow 34

31 Abstract Vision Lane Detection Vehicle Detection Etc. Radar FMCW Radar Modeling Tracking Extend Kalman Filter based Tracking Sensor Fusion Integration 35

32 Modeling FMCW Radar for Automotive The received signal is a delayed copy of the transmitted signal Car+Radar Target Ultra-large signal bandwidth (>100MHz) Ultra high frequencies (>77GHz) 36

33 frequency beat frequency FMCW Radar: How Does it Work? delay RF Target = Delay LNA time PA DSP 37

34 What Behavior Can Be Modeled? Channel modeling (interference, noise) Modeling of antenna arrays (no of elements, position) LNA PA DSP Modeling of RF Impairments (noise, non-linearity, frequency dependency) Determination of waveforms (range, resolution) Algorithms for data analysis 38

35 Which MathWorks Tools Can Help? Communications System Toolbox Phased Array System Toolbox LNA PA DSP SimRF Phased Array System Toolbox Signal Processing Toolbox Instrument Control Toolbox Phased Array System Toolbox DSP System Toolbox 39

36 Phased Array System Toolbox Algorithms Simulation Tools Waveforms Pulse, LFM, FMCW, etc. Transmitter Monostatic and Bistatic Tx Antenna Arrays ULA, URA, etc. Waveform Generator Transmitter Transmit Array Environment, Targets, and Interference Signal Processing Beamforming, Matched Filtering, Detection, CFAR, STAP, etc. Receiver Receiver Monostatic and Bistatic Receive Array Rx Antenna Arrays ULA, URA, etc. Environment effects, impairments, interference 40

37 Phased Array System Toolbox Waveforms Pulse, LFM, FMCW, etc. Transmitter Monostatic and Bistatic Tx Antenna Arrays ULA, URA, etc. Algorithms Simulation Tools Design & Analyze Tools Waveform Generator Transmitter Transmit Array Environment, Targets, and Interference Signal Processing Beamforming, Matched Filtering, Detection, CFAR, STAP, etc. Receiver Receiver Monostatic and Bistatic Receive Array Rx Antenna Arrays ULA, URA, etc. Environment effects, impairments, interference 41

38 Radar Simulation 43

39 Abstract Vision Lane Detection Vehicle Detection Etc. Radar FMCW Radar Modeling Tracking Extend Kalman Filter based Tracking Sensor Fusion Integration 44

40 Moving From Detections to Tracking Detect moving objects Estimate motion of each object Assign detections to tracks using data association Innovate track data from detected objects 45

41 Kalman Filter From Wikipedia, Kalman Filter (Linear) Predict a priori state estimate a priori estimate covariance Update Innovation residual Innovation covariance Kalman gain a posteriori state estimate a posteriori estimate covariance 46

42 Kalman Filter Function in MATLAB (Computer Vision System Toolbox) Define : Kalman Filter (Linear) kalmanfilter Predict = configurekalmanfilter(motionmodel, InitialLocation, InitialEstimateError, MotionNoise, MeasurementNoise) a priori state estimate Prediction : a priori estimate covariance pred_location Update = kalmanfilter.predict() Correction : Innovation residual corrected_location Innovation = kalmanfilter.correct(observed_location) covariance Kalman gain a posteriori state estimate a posteriori estimate covariance 47

43 Kalman Filter Blocks in MATLAB (DSP System Toolbox) Kalman Filter (Linear) Predict a priori state estimate a priori estimate covariance Update Innovation residual Innovation covariance Kalman gain a posteriori state estimate a posteriori estimate covariance 48

44 Extended Kalman Filter Prediction Extended Kalman Filter Prediction Innovation Innovation Jacobian Matrix!!! 49

45 How to Make Jacobian Matrix From Wikipedia, 50

46 Solution MuPAD in Symbolic Math Toolbox Define X vector Define f vector Calculate f x 51

47 Solution MuPAD in Symbolic Math Toolbox 52

48 Data Association for Tracking Multiple Objects James Munkres's variant of the Hungarian assignment algorithm [assignments,unassignedtracks,unassigneddetections] = assigndetectionstotracks( costmatrix,costofnonassignment) Example of costmatrix costmatrix = zeros(ntracks,ndetect); for i = 1:nTracks costmatrix(i, :) = distance(predictedtracks(i), Detections); end Predicted location Detection 53

49 Abstract Vision Lane Detection Vehicle Detection Etc. Radar FMCW Radar Modeling Tracking Extend Kalman Filter based Tracking Sensor Fusion Integration 54

50 Integration in Simulink Input : Image & Target Info. Visualization Camera Module. Lane Detection. Vehicle Detection. Tracking Fusion Module. Data Association. Data Fusion, TTC. In-Path Vehicle Radar Module. Radar Modeling. Detection. Tracking 56

51 What s the problem which this model has in terms of Integration? Use a single file for multiple engineers Chaotic Keeping track of file dependencies How Can I Reduce Inefficiency when Integrating Models? 57

52 Partitioning a Model using Model Blocks Model Reference used to partition top-level model into components Signal Buses & Bus Objects used to define the interfaces between components One file-per-component fits well with Revision Control packages Enabler for parallel development of components Parent Model Child Model OLD MODEL 58

53 Variant Subsystems One model Variation Management 59

54 What s the problem which this model has in terms of Integration? Use a single file for multiple engineers Chaotic Keeping track of file dependencies How Can I Reduce Inefficiency when Integrating Models? 61

55 Simulink Projects Manage project-related files efficiently within Simulink Search, manage, and share project-related files Conduct peer review of changes using Simulink XML comparison tool Access source control functionality for team collaboration 62

56 Simulation Result Yaw Rate Based Lane Based * : Vision Target o : Radar Target x : Fusion Target 63

57 Simulation Result No Radar Target Stationary Vehicle 64

58 Simulation Result 65

59 Conclusion MATLAB/Simulink and etc. provide an environment to develop algorithm, modeling environment and integrate algorithms and various types of sensor models. For Vision or Lidar Sensor : Image Acquisition Toolbox Image Processing Toolbox Computer Vision System Toolbox For Radar Sensor Phased Array System Toolbox SimRF Signal Processing Toolbox Simulink is a good platform for team collaboration Simulink Project 66

60 More Information Computer Vision Computer Vision Workflow in MATLAB/Simulink Object Detection and Tracking Face Recognition with MATLAB Radar Design Modeling and Simulating Radar Systems Using MATLAB and Simulink Design and Verify RF Transceivers for Radar Systems html?s_tid=srchtitle Users Stories Traffic Recognition System from Continental Radar Sensor Alignment Algorithm from Delphi 67