MATLAB 및 Simulink 를이용한운전자지원시스템개발

MATLAB 및 Simulink 를이용한운전자지원시스템개발 김종헌차장 Senior Application Engineer MathWorks Korea 2015 The MathWorks, Inc. 1

Example : Sensor Fusion with Monocular Vision & Radar Configuration Monocular Vision installed on Wind Shield Radar installed in Front Grill High Level Sensor Fusion 2 Different Scenario Closest In-Path Vehicle Selection Stationary Vehicle Detection 2

Sensor Fusion Why Sensor Fusion? To complement weakness of individual sensor Radar More Coverage Camera Accurate Range Poor Measureable Directly Range Rate Estimated Poor Angle Resolution Relatively Good Not Available Classification Available 3

Issues? Communication Risk of Vehicle Test HOG KLT Tracking SGM TTC LQR/LQG AUTOSAR RCS Max Doppler Interference Huh!! Noise figure Power@50 4

How do we bridge the gap between engineers who speak different languages and reduce risk? Communication Risk of Vehicle Test HOG KLT Tracking SGM TTC LQR/LQG AUTOSAR RCS Max Doppler Interference Noise figure Power@50 5

Solution: Model based Multi-domain Modeling and System Level Simulation Here are the specs Here s the verification model Model serves as the executable specification and test environment Multiple engineering teams can collaborate more effectively 6

Example : Sensor Fusion with Monocular Vision & Radar Vision Lane Detection Vehicle Detection Extended Kalman Filter based Tracking Radar FMCW Radar Modeling Extended Kalman Filter based Tracking Sensor Fusion Data Association Data Fusion 7

Abstract Vision Lane Detection Vehicle Detection Etc. Radar FMCW Radar Modeling Tracking Extend Kalman Filter based Tracking Sensor Fusion Integration 8

Lane Detection ROI (Region Of Interest) ROI (Region Of Interest) Cropped Image Captured Image Edge Mapping to World Selecting Candidate Pts Curve Fitting 9

Lane Detection ROI (Region Of Interest) MATLAB Built-In Function CVST Function Image Processing Function ROI (Region Of Interest) Cropped Image Captured Image Easy Matrix Calculation Code Generation Easy to Debug Scripting Language Pixel level, 2D, 3D Visualization Edge Mapping to World Selecting Candidate Pts Curve Fitting 10

Lane Detection 14

Abstract Vision Lane Detection Vehicle Detection Etc. Radar FMCW Radar Modeling Tracking Extend Kalman Filter based Tracking Sensor Fusion Integration 15

Cascade Detector Applications in Automotive of Cascade Detector Vehicle detection Pedestrian detection Traffic Sign Recognition Difference between applications is Feature. 16

4 Easy Steps to Create a Car Detector Collect Car Image Data (Training Images) Label Objects of Interest Train Object Detector Test Detector Performance 17

Step 1: Collect Training Image Data Collect many images that represent object you want to detect Ensure that images represent different conditions Lighting Different variations of object size, color, model etc. 18

Step 2: Label Training Images Select and assign regions of interest Images from: www.emt.tugraz.at/~pinz/data/graz_02/ 19

Step 3: Train Object Detector Single line of ML code 20

Step 4: Test Detector Performance 21

Comparison Data from Vehicle Dynamics S/W and Result of Cascade Detector 22

Further Information about Cascade Object Detector See documentation for detail explanation http://www.mathworks.com/help/vision/ug/train-acascade-object-detector.html. 23

Abstract Vision Lane Detection Vehicle Detection Etc. Radar FMCW Radar Modeling Tracking Extend Kalman Filter based Tracking Sensor Fusion Integration 24

Camera Calibration App. Original image Undistorted image Camera Calibration App. MATLAB commands for camera calibration are also available. 26

Camera Calibration App. Intrinsic Matrix Tangential Distortion K = f x 0 0 s f y 0 c x c y 1 c x, c y - camera center in pixels (f x, f y ) - focal length in pixels per world unit f x = f p x, f y = f p y s sensor axes skew coefficient s = f y tan α x tangential = x radial + 2p 1 xy + p 2 r 2 + 2x 2 y tangential = y radial + p 1 r 2 + 2y 2 + 2p 2 xy Radial Distortion x radial = 1 + k 1 r 2 + k 2 r 4 + k 3 r 6 x y radial = 1 + k 1 r 2 + k 2 r 4 + k 3 r 6 y x and y are in normalized image coordinates in world units relative to the optical center 27

Image Processing Image Enhancement Deblurring Noise Removal Image Segmentation Active Contours Watershed 28

Feature Detection, Extraction and Matching Edge Corner Template SURF MSER Image Matching Feature Point Tracking Video Stabilization 29

Stereo Vision (2014a) Stereo calibration Semi-global disparity matching Better results than block matching 3D scene reconstruction from disparity New demos available Estimate real distance to objects in video 3D scene reconstruction Pedestrian detection (2014b) 30

Stixel based Object Detection Available to detect objects which are not easy using other sensors Stationary object, overlapped object, kid. * Stereo and City, Franke Uwe, Daimler Research Center 31

Stereo Vision Demo Computer Vision System Toolbox Stereo camera calibration SGM based Stereo matching 3D reconstruction using camera calibration data Kanade-Lucas-Tomasi point tracking 2D/3D plotting Bird View Disparity Image Stixel Representation 3D Reconstruction KLT Point Tracking 32

Point Cloud Processing (2015a) pointcloud object Fast search using kdtree Tree is constructed implicitly when a search is called first time Downsampling Two method : gridaverage, random Visualization Rigid registration Iterative Closest Point Algorithm 33

OCR Optical Character Recognition (2014a) Highly requested in Frontlines Support for English, Japanese Users can download additional language support Shipping demo with text detection and OCR workflow 34

Abstract Vision Lane Detection Vehicle Detection Etc. Radar FMCW Radar Modeling Tracking Extend Kalman Filter based Tracking Sensor Fusion Integration 35

Modeling FMCW Radar for Automotive The received signal is a delayed copy of the transmitted signal Car+Radar Target Ultra-large signal bandwidth (>100MHz) Ultra high frequencies (>77GHz) 36

frequency beat frequency FMCW Radar: How Does it Work? delay RF Target = Delay LNA time PA DSP 37

What Behavior Can Be Modeled? Channel modeling (interference, noise) Modeling of antenna arrays (no of elements, position) LNA PA DSP Modeling of RF Impairments (noise, non-linearity, frequency dependency) Determination of waveforms (range, resolution) Algorithms for data analysis 38

Which MathWorks Tools Can Help? Communications System Toolbox Phased Array System Toolbox LNA PA DSP SimRF Phased Array System Toolbox Signal Processing Toolbox Instrument Control Toolbox Phased Array System Toolbox DSP System Toolbox 39

Phased Array System Toolbox Algorithms Simulation Tools Waveforms Pulse, LFM, FMCW, etc. Transmitter Monostatic and Bistatic Tx Antenna Arrays ULA, URA, etc. Waveform Generator Transmitter Transmit Array Environment, Targets, and Interference Signal Processing Beamforming, Matched Filtering, Detection, CFAR, STAP, etc. Receiver Receiver Monostatic and Bistatic Receive Array Rx Antenna Arrays ULA, URA, etc. Environment effects, impairments, interference 40

Phased Array System Toolbox Waveforms Pulse, LFM, FMCW, etc. Transmitter Monostatic and Bistatic Tx Antenna Arrays ULA, URA, etc. Algorithms Simulation Tools Design & Analyze Tools Waveform Generator Transmitter Transmit Array Environment, Targets, and Interference Signal Processing Beamforming, Matched Filtering, Detection, CFAR, STAP, etc. Receiver Receiver Monostatic and Bistatic Receive Array Rx Antenna Arrays ULA, URA, etc. Environment effects, impairments, interference 41

Radar Simulation 43

Abstract Vision Lane Detection Vehicle Detection Etc. Radar FMCW Radar Modeling Tracking Extend Kalman Filter based Tracking Sensor Fusion Integration 44

Moving From Detections to Tracking Detect moving objects Estimate motion of each object Assign detections to tracks using data association Innovate track data from detected objects 45

Kalman Filter From Wikipedia, Kalman Filter (Linear) Predict a priori state estimate a priori estimate covariance Update Innovation residual Innovation covariance Kalman gain a posteriori state estimate a posteriori estimate covariance 46

Kalman Filter Function in MATLAB (Computer Vision System Toolbox) Define : Kalman Filter (Linear) kalmanfilter Predict = configurekalmanfilter(motionmodel, InitialLocation, InitialEstimateError, MotionNoise, MeasurementNoise) a priori state estimate Prediction : a priori estimate covariance pred_location Update = kalmanfilter.predict() Correction : Innovation residual corrected_location Innovation = kalmanfilter.correct(observed_location) covariance Kalman gain a posteriori state estimate a posteriori estimate covariance 47

Kalman Filter Blocks in MATLAB (DSP System Toolbox) Kalman Filter (Linear) Predict a priori state estimate a priori estimate covariance Update Innovation residual Innovation covariance Kalman gain a posteriori state estimate a posteriori estimate covariance 48

Extended Kalman Filter Prediction Extended Kalman Filter Prediction Innovation Innovation Jacobian Matrix!!! 49

How to Make Jacobian Matrix From Wikipedia, 50

Solution MuPAD in Symbolic Math Toolbox Define X vector Define f vector Calculate f x 51

Solution MuPAD in Symbolic Math Toolbox 52

Data Association for Tracking Multiple Objects James Munkres's variant of the Hungarian assignment algorithm [assignments,unassignedtracks,unassigneddetections] = assigndetectionstotracks( costmatrix,costofnonassignment) Example of costmatrix costmatrix = zeros(ntracks,ndetect); for i = 1:nTracks costmatrix(i, :) = distance(predictedtracks(i), Detections); end Predicted location Detection 53

Abstract Vision Lane Detection Vehicle Detection Etc. Radar FMCW Radar Modeling Tracking Extend Kalman Filter based Tracking Sensor Fusion Integration 54

Integration in Simulink Input : Image & Target Info. Visualization Camera Module. Lane Detection. Vehicle Detection. Tracking Fusion Module. Data Association. Data Fusion, TTC. In-Path Vehicle Radar Module. Radar Modeling. Detection. Tracking 56

What s the problem which this model has in terms of Integration? Use a single file for multiple engineers Chaotic Keeping track of file dependencies How Can I Reduce Inefficiency when Integrating Models? 57

Partitioning a Model using Model Blocks Model Reference used to partition top-level model into components Signal Buses & Bus Objects used to define the interfaces between components One file-per-component fits well with Revision Control packages Enabler for parallel development of components Parent Model Child Model OLD MODEL 58

Variant Subsystems One model Variation Management 59

What s the problem which this model has in terms of Integration? Use a single file for multiple engineers Chaotic Keeping track of file dependencies How Can I Reduce Inefficiency when Integrating Models? 61

Simulink Projects Manage project-related files efficiently within Simulink Search, manage, and share project-related files Conduct peer review of changes using Simulink XML comparison tool Access source control functionality for team collaboration 62

Simulation Result Yaw Rate Based Lane Based * : Vision Target o : Radar Target x : Fusion Target 63

Simulation Result No Radar Target Stationary Vehicle 64

Simulation Result 65

Conclusion MATLAB/Simulink and etc. provide an environment to develop algorithm, modeling environment and integrate algorithms and various types of sensor models. For Vision or Lidar Sensor : Image Acquisition Toolbox Image Processing Toolbox Computer Vision System Toolbox For Radar Sensor Phased Array System Toolbox SimRF Signal Processing Toolbox Simulink is a good platform for team collaboration Simulink Project 66

More Information Computer Vision Computer Vision Workflow in MATLAB/Simulink http://www.matlabexpo.com/kr/2015/sessions/t1-s2.html Object Detection and Tracking http://in.mathworks.com/videos/computer-vision-with-matlab-for-object-detection-andtracking-81866.html?s_tid=srchtitle Face Recognition with MATLAB http://in.mathworks.com/videos/face-recognition-with-matlab-98076.html?s_tid=srchtitle Radar Design Modeling and Simulating Radar Systems Using MATLAB and Simulink http://in.mathworks.com/videos/modeling-and-simulating-radar-systems-using-matlab-andsimulink-81966.html?s_tid=srchtitle Design and Verify RF Transceivers for Radar Systems http://in.mathworks.com/videos/design-and-verify-rf-transceivers-for-radar-systems- 81990.html?s_tid=srchtitle Users Stories Traffic Recognition System from Continental Radar Sensor Alignment Algorithm from Delphi 67