Computer Vision Slides curtesy of Professor Gregory Dudek

Size: px

Start display at page:

Download "Computer Vision Slides curtesy of Professor Gregory Dudek"

Janel Beasley
5 years ago
Views:

1 Computer Vision Slides curtesy of Professor Gregory Dudek Ioannis Rekleitis

2 Why vision? Passive (emits nothing). Discreet. Energy efficient. Intuitive. Powerful (works well for us, right?) Long and short range. Fast. CSCE 774: Robotic Systems 2

3 So, what s the problem? How hard is vision? Why do we think is do-able? Problems: Slow. Data-heavy. Impossible. Mixes up many factors. CSCE 774: Robotic Systems 3

4 The Vision Problem Input Vision Algorithm Output CSCE 774: Robotic Systems 4

5 The Vision Problem Input Vision Algorithm Output CSCE 774: Robotic Systems 5

The Vision Problem Input Vision Algorithm Output 207 200 194 194 203 130 105 095 107 153 192 196 190 186 175 242 194 222 254 255 124 074 082 072 076 208 206 202 194 185 254 170 204 255 248 122 153

6 The Vision Problem Input Vision Algorithm Output CSCE 774: Robotic Systems 6

7 The Vision Problem Input Vision Algorithm Output CSCE 774: Robotic Systems 7

8 The Vision Problem Input Vision Algorithm Output CSCE 774: Robotic Systems 8

The Vision Problem Input Vision Algorithm Output 020 067 073 058 055 076 069 050 074 064 065 066 066 059 023 047 109 107 118 107 115 110 120 120 124 120 128 124 132 131 047 125 130 130 122 121 117

9 The Vision Problem Input Vision Algorithm Output CSCE 774: Robotic Systems 9

10 doesn t need a full interpretation of available images This is Prof. X in his office offering me a can of spam. does need information about what to do... Run Away!! What does a robot need? reactive deliberative avoiding obstacles (or predators) pursuing objects localizing itself Mapping finding targets reasoning about the world environmental interactions CSCE 774: Robotic Systems 10

11 What does a robot need? What a camera does to the 3d world... Shigeo Fukuda squeezes away one dimension Material/ CSCE 774: Robotic Systems 11

12 What does a robot need? What a camera does to the 3d world... Shigeo Fukuda Material/ CSCE 774: Robotic Systems 12

13 Ill-posed In trying to extract 3d structure from 2d images, vision is an ill-posed problem. CSCE 774: Robotic Systems 13

14 The vision problem in general... In trying to extract 3d structure from 2d images, vision is an ill-posed problem. Basically, there are too many possible worlds that might (in theory) give rise to a particular image CSCE 774: Robotic Systems 14

15 Ill-posed In trying to extract 3d structure from 2d images, vision is an ill-posed problem. CSCE 774: Robotic Systems 15

16 Ill-posed In trying to extract 3d structure from 2d images, vision is an ill-posed problem. An image isn t enough to disambiguate the many possible 3d worlds that could have produced it. CSCE 774: Robotic Systems 16

17 Camera Geometry 3D2D transformation: perspective projection center of projection object focal length image plane CSCE 774: Robotic Systems 17

18 Coordinate Systems y u pixel coordinates v x object coordinates canonical axes at the C.O.P. u (col) z f Z optical axis y v (row) x principal point Add coordinate systems in order to describe feature points... CSCE 774: Robotic Systems 18

19 Coordinate Systems image can. coords: (x,y) y u pixel coordinates v (X,Y,Z) in canonical coords x object coordinates canonical axes z f CSCE 774: Robotic Systems 19

20 From 3d to 2d image can. coords: (x,y) y u pixel coordinates v (X,Y,Z) in canonical coords x object coordinates canonical axes z f x = f X Z y = f Y Z a nonlinear transformation goal: to recover information about (X,Y,Z) from (x,y) CSCE 774: Robotic Systems 20

21 Camera Calibration Camera Model [u v 1] Pixel coords World coords Intrinsic Parameters focal lengths in pixels skew coefficient focal point Extrinsic Parameters Rotation and Translation CSCE 774: Robotic Systems w w w c z y x T A R v u z T w w w z y x o y x v u A y y x x m f m f, u,v o 0 T R

22 Camera Calibration Existing packages in MATLAB, OpenCV, etc CSCE 774: Robotic Systems 22

23 A Vision solution If interpreting a single image is difficult... What about more?! multiple cameras multiple times CSCE 774: Robotic Systems 23

24 Robot vision sampler A brief overview of robotic vision processing... (1) Image streams simplified via generality simplified via specificity (2) Stereo vision (or beyond...) (3) Incorporating vision within robot control 3d reconstruction Visual servoing speaking of servoing... CSCE 774: Robotic Systems 24

25 Visual Servoing CSCE 774: Robotic Systems 25

26 Details Images are not actually continuous. The sampling (and hardware) issues lead to a few other minor problems. CSCE 774: Robotic Systems 26

27 CCD (Charge-Coupled Device) CSCE 774: Robotic Systems 27

28 Aliasing. To avoid: f sampling > 2F max Nyquist Rate CSCE 774: Robotic Systems 28

29 Aliasing: Moiré Patterns CSCE 774: Robotic Systems 29

30 Key problems Recognition: What is that thing in the picture? What are all the things in the image? Scene interpretation Describe the image? Scene reconstruction : What is the 3-dimensional layout of the scene? What are the physical parameters that gave rise to the image? What is a description of the scene? Notion of an inverse problem. CSCE 774: Robotic Systems 30

31 Correspondence Problem CSCE 774: Robotic Systems 31

32 Correspondence From I 1 From I 2? CSCE 774: Robotic Systems 32

33 Gaussian Blur CSCE 774: Robotic Systems 33

34 Gaussian Blur and Noise CSCE 774: Robotic Systems 34

35 Gaussian Blur and Noise CSCE 774: Robotic Systems 35

36 Gaussian Blur, Noise, Sobel CSCE 774: Robotic Systems 36

37 Fiduciary Markers/Fiducial Fourier Tag CSCE 774: Robotic Systems 37

38 Stereo Vision: Pinhole Camera image plane f 1 p O 1 focal points CSCE 774: Robotic Systems O 2 image plane f 2 38

39 Stereo Vision: Pinhole Camera image plane f 1 p p 1 O 1 focal points CSCE 774: Robotic Systems O 2 p 2 image plane f 2 39

40 Stereo Vision: Pinhole Camera image plane f 1 (part of) epipolar plane p O 1 p 1 p 2 epipolar line focal points O 2 image plane f 2 CSCE 774: Robotic Systems 40

41 Stereo Vision: Pinhole f O 1 baseline b x1 p x1 D p O 2 p x2 disparity: d=p x1 -p x2 CSCE 774: Robotic Systems x2 Depth: D=fb/d 41

42 Stereo Vision: Pinhole f a 1 q 1 p x1 D q 1 p a 2 x1 p x2 q 2 q 2 x2 CSCE 774: Robotic Systems 42

43 Large Baseline CSCE 774: Robotic Systems 43

44 Stereo: Disparity Map CSCE 774: Robotic Systems 44

45 Another Example (Hole Filling) Cloth Parameters and Motion Capture by David Pritchard B.A.Sc., University of Waterloo, 2001 CSCE 774: Robotic Systems 45

46 Depth Map in a City CSCE 774: Robotic Systems 46

47 Stereo Vision Large number of algorithms out there: rank 43 different algorithms. CSCE 774: Robotic Systems 47

48 Good Feature High Recall Good Precision Feature Detection Feature Matching Several Alternatives: Harris Corners (OpenCV) SURF (OpenCV) SIFT etc CSCE 774: Robotic Systems 48

49 Harris Corners CSCE 774: Robotic Systems 49

50 SURF CSCE 774: Robotic Systems 50

51 SIFT CSCE 774: Robotic Systems 51

52 Optical Flow Definition: the pattern of apparent motion of objects, surfaces, and edges in a visual scene caused by the relative motion between an observer (an eye or a camera) and the scene. CSCE 774: Robotic Systems 52

53 Optical Flow Field CSCE 774: Robotic Systems 53

54 Optical flow Information about image motion rather than the scene. This is a classic reconstruction problem. This next step might be to use the image motion to infer scene motion, robot motion or 3D layout. time sequence of images CSCE 774: Robotic Systems 54

55 Optical flow Information about scene motion rather than the scene. an image cube I(x,y,t) CSCE 774: Robotic Systems 55

56 Optical flow Information about scene motion rather than the scene. optical flow How? CSCE 774: Robotic Systems 56

57 Optical Flow By measuring the direction that intensities are moving... I(x,y,t) We can estimate things CSCE 774: Robotic Systems 57

58 Observations & Warnings How can we do this? Assume the scene itself is static. Find matching chunks in the images. An instance of correspondence. BUT World really isn t static. Lightning might change even in a static scene. CSCE 774: Robotic Systems 58

59 Optical Flow By measuring the direction that intensities are moving... I(x,y,t) I(x,y,0) I(0,0,0) I(2,-1,0) We can estimate things... di at (0,0,0) dx = I x I(x,y,1) I(0,0,1) CSCE 774: Robotic Systems 59

60 Optical Flow By measuring the direction that intensities are moving... I(x,y,t) I(x,y,0) I(0,0,0) I(2,-1,0) di at (0,0,0) I(x,y,1) I We can estimate things like = I x = x = dx I(0,0,1) I(1,0,0) - I(0,0,0) = CSCE 774: Robotic Systems 60

61 Optical Flow By measuring the direction that intensities are moving... I(x,y,t) I(x,y,0) I(0,0,0) I(2,-1,0) We can estimate things like di dx = I x I(x,y,1) = I y I(0,0,1) so... CSCE 774: Robotic Systems 61 di dy di dt = I t

62 Measuring Optical Flow Let I(x,y,t) be the sequence of images. (x,y,t) Simplest assumption (constant brightness constraint): I(x,y,t) = I(x + dx, y + dy, t + dt) CSCE 774: Robotic Systems 62

63 Measuring Optical Flow Let I(x,y,t) be the sequence of images. (x,y,t) Simplest assumption (constant brightness constraint): I(x,y,t) = I(x + dx, y + dy, t + dt) Reminder: f(x + dx) = f(x) + f (x) dx + f (x) dx 2 / CSCE 774: Robotic Systems 63

64 Measuring Optical Flow Let I(x,y,t) be the sequence of images. (x,y,t) Simplest assumption (constant brightness constraint): I(x,y,t) = I(x + dx, y + dy, t + dt) Reminder: f(x + dx) = f(x) + f (x) dx + f (x) dx 2 / I(x,y,t) = I(x,y,t) + I x dx + I y dy + I t dt + 2nd deriv. + higher CSCE 774: Robotic Systems 64

65 Measuring Optical Flow Let I(x,y,t) be the sequence of images. (x,y,t) Simplest assumption (constant brightness constraint): I(x,y,t) = I(x + dx, y + dy, t + dt) Reminder: f(x + dx) = f(x) + f (x) dx + f (x) dx 2 / I(x,y,t) = I(x,y,t) + I x dx + I y dy + I t dt + 2nd deriv. + higher 0 = I x dx + I y dy + I t dt ignore these terms CSCE 774: Robotic Systems 65

66 Measuring Optical Flow Let I(x,y,t) be the sequence of images. (x,y,t) Simplest assumption (constant brightness constraint): I(x,y,t) = I(x + dx, y + dy, t + dt) Reminder: f(x + dx) = f(x) + f (x) dx + f (x) dx 2 / I(x,y,t) = I(x,y,t) + I x dx + I y dy + I t dt + 2nd deriv. + higher 0 = I x dx + I y dy + I t dt ignore these terms -I t = I dx x dt + I y dy dt intensity-flow equation good and bad... CSCE 774: Robotic Systems 66

67 The aperture problem -I t = I dx x dt + I y dy dt The intensity-flow equation provides only one constraint on two variables ( x-motion and y-motion) It is only possible to find optical flow in one direction... CSCE 774: Robotic Systems 67

68 The aperture problem It is only possible to find optical flow in one direction... at any single point in the image! img1 img2 raw optical flow smoothed for ten iterations Smoothing can be done by incorporating neighboring points information. CSCE 774: Robotic Systems 68

69 Optical Flow Application Visual Odometry Wheel slip detection on future Mars Rovers CSCE 774: Robotic Systems 69

70 Image Downsampling CSCE 774: Robotic Systems 70

Introduction. Ioannis Rekleitis

Introduction. Ioannis Rekleitis Introduction Ioannis Rekleitis Why Image Processing? Who here has a camera? How many cameras do you have Point where computers fast/cheap Cameras become omnipresent Deep Learning CSCE 590: Introduction