Computational Optical Imaging - Optique Numerique

Size: px

Start display at page:

Download "Computational Optical Imaging - Optique Numerique"

Letitia Fleming
6 years ago
Views:

1 Computational Optical Imaging - Optique Numerique Autumn 2015 Ivo Ihrke

2 Organizational Issues Course schedule (tentative) 1. Intro / Recap: Img. Characteristics Monday HDR / Spectral / Polarization Monday Deblurring / Inverse Problems Monday Persp. Geometry and 3D Basics Monday Structure-from-Motion Tuesday Stereo Matching and Optical Flow Tuesday Active light 3D scanning Monday Volumetric 3D Tomography Mondayday Light Fields Tuesday Current topics Tuesday

3 Organizational Issues Written exam hours Personal notes allowed No computer / mobiles / books Active participation encouraged Send to ivo.ihrke@inria.fr to be added to course mailing list Slides will be available at

4 0. What this course is about Autumn 2015 Ivo Ihrke

computers complete digitization of imaging and

7 The computational aspect Game Changer Computation Within last decade Inexpensive, powerful, small computers complete digitization of imaging and display pipeline Acquisition, transmission, storage, analysis, display

8 Human -> Silicon Observer

9 Computational Imaging acquisition computation perception scene

10 Course Contents Fast Forward Digital Imaging 2-dimensional

11 Course Contents Fast Forward Digital Imaging 2-dimensional Computational Preprocessing Handle sensor imperfections Dynamic range, Noise Raw image development Color correction, white balancing Digital correction of imaging imperfections Geometric distortions Aberration correction Compression (and why not to use it)

12 2D Imaging Demosaicking Undistortion [van der Jeught 11] [RAW Explorer]

13 RAW processing [Heide et al. 2013] [Schuler et al. 2011]

14 Course Contents Fast Forward Digital Imaging 2-dimensional Computational Preprocessing Several 2D images Stereo Reconstruction

15 Stereo Reconstruction Disparity estimation Disparity = apparent parallax Inversely related to depth stereo image (try cross-eye) disparity map/ depth map [Tsukuba]

16 Stereo Left Image [Middlebury Stereo Data Sets]

17 Stereo Right Image [Middlebury Stereo Data Sets]

18 Stereo Disparity Map (bright is close) [Middlebury Stereo Data Sets]

19 Course Contents Fast Forward Digital Imaging 2-dimensional Computational Preprocessing Several 2D images Stereo Reconstruction Multi-View Stereo

20 [Agrawal Ivo Ihrke et / Autumn al 10] 2015

21 [Agrawal Ivo Ihrke et / Autumn al 10] 2015

22 [Agrawal Ivo Ihrke et / Autumn al 10] 2015

23 [Agrawal Ivo Ihrke et / Autumn al 10] 2015

24 [Agrawal Ivo Ihrke et / Autumn al 10] 2015

25 Course Contents Fast Forward Digital Imaging 2-dimensional Computational Preprocessing Several 2D images Stereo Reconstruction Multi-View Stereo 3D Scanning (active) Laser range scanning Structured light Kinect I

26 Active Systems - Components acquisition computation Active illumination scene

27 3D Scanning [Lanman & Taubin 09] [DAVID Laser Scanner] [Digital Michelangelo Project, Stanford Univ.]

28 3D Scanning - Kinect [Matthew Fisher] [Andres Reza]

29 Course Contents Fast Forward Digital Imaging 2-dimensional Computational Preprocessing Several 2D images Stereo Reconstruction Multi-View Stereo 3D Scanning (active) Time-of-Flight (active)

30 Time-of-Flight Sensing LIDAR Phase-Based Measurements [SICK AG] [Mesa Imaging AG] [PMD Technologies GmbH] [Microsoft Kinect 2.0] [usgs.gov] [Groupe Info Consult]

31 Application - Google Street View [Anguelov et al. 10]

32 Course Contents Fast Forward Digital Imaging 2-dimensional Computational Preprocessing Several 2D images Stereo Reconstruction Multi-View Stereo 3D Scanning (active) Time-of-Flight (active) Focal Stacks

33 Focal Stacks

34 Focal Stack Extended depth-of-field image

35 Focal Stack

36 Extended Depth of Field

37 What is this course about? Imaging 2-dimensional Computational Preprocessing Several 2D images 3D Surfaces 3-dimensional Tomography Fourier Slice Theorem Filtered Back Projection Algebraic Reconstruction Techniques

38 Tomography [Stierstorfer 2003] [DSPGuide] [NIH] [MadisonRadiologists]

39 Tomography Applications Surface Characterization Engineering measurements 3D Displays [Trifonov et al. 06] [Atcheson et al. 08] [Wetzstein et al. 11]

40 What is this course about? Imaging 2-dimensional Computational Preprocessing Several 2D images 3D Surfaces 3-dimensional Tomography Direct Volume Slicing Confocal Microscopy

41 Volume Slicing Photograph Digital rendering [Hullin et al. 08]

42 Confocal Microscopy [University of Illinois] [Schürmann,Ramachandra, Uni Münster]

43 What is this course about? Imaging 2-dimensional Computational Preprocessing Several 2D images 3D Surfaces 3-dimensional 3D Volumes Multi-dimensional Plenoptic function Light fields

44 The 7D Plenoptic Function l(q, f, l, t) Q: What is the set of all things that one can ever see? A: The Plenoptic Function [Adelson and Bergen 1991] (from plenus, complete or full, and optic)

45 The 7D Plenoptic Function l(q, f, l, t) Q: What is the set of all things that one can ever see? A: The Plenoptic Function [Adelson and Bergen 1991] (from plenus, complete or full, and optic)

46 The 7D Plenoptic Function l(q, f, l, t, p x, p y, p z ) P(q, f, l, t, p x, p y, p z ) defines the intensity of light: as a function of viewpoint as a function of time as a function of wavelength

47 The 7D Plenoptic Function l(q, f, l, t, p x, p y, p z ) P(q, f, l, t, p x, p y, p z ) defines the intensity of light: as a function of viewpoint as a function of time as a function of wavelength

48 Example of digital refocusing [Ng 2005]

49 Example of moving the observer [Ng 2005]

50 START OF THE LECTURE

51 Properties of Digital Images

52 Digital Images - Limitations Digital Sensor noise Dynamic Range Tone Curve Recording Medium Monochromatic -> Color processing & interpolation Compression Optical Distortions Aberrations

53 Noise

54 Noise Sources [Reibel2003] photon shot noise dark current noise read noise

55 Dark Current Noise Removal cooling the chip noise removal techniques to separate image data from noise e.g. median filtering uncooled cooled 25 s exposure time

56 Discretization in Space and Intensity

57 Analog/Digital Conversion

58 Digital Images Images are now numbers (corrupted by noise)

59 Dynamic Range

60 Dynamic Range dr = max output swing noise in the dark = Saturation level dark current Dark shot noise + readout noise noise in the dark is random noise sources that cannot be corrected with circuit tricks Photon shot noise and read noise

61 Dynamic Range of Standard Sensors 13.5 EVs or f-stops = contrast 11,000:1 = color Ivo Ihrke negative / Autumn 2015

62 Color Calibration Toolbox Verify response curve the example is for jpg on the Canon 5D mark II Make sure the samples are fit well Response curves (R,G,B) samples Inverse response curves

63 Color

64 source: Kodak KAF-5101ce data sheet Sensing color Eye has 3 types of color receptors Therefore we need 3 different spectral sensitivities

65 Ways to sense color Color filter array cover each sensor with an individual filter requires just one chip but loses some spatial resolution demosaicing requires tricky image processing G R B G primary

66 White Balance capture the spectral characteristics of the light source to assure correct color reproduction tungsten daylight flourescent flash

67 White Balance Camera built-in function derive scale from white point infrared red green blue ultra violet wavelength

68 White Balance Camera built-in function derive scale from white point infrared red green blue ultra violet wavelength

69 Imperfections in Imaging

70 Lens Aberrations Spherical aberration Coma Astigmatism Curvature of field Distortion

71 Sharpness Related Aberrations

$Chromatic Aberration Index of refraction varies with$ correct using a two-element achromatic doublet, with a

doublets only correct at two wavelengths Why don t humans see

72 Chromatic Aberration Index of refraction varies with wavelength For convex lens, blue focal length is shorter Can correct using a two-element achromatic doublet, with a different glass (different n ) for the second lens Achromatic doublets only correct at two wavelengths Why don t humans see chromatic aberration? (check )

73 Chromatic Aberrations Longitudinal chromatic aberration (change in focus with wavelength) image: Smith 2000

74 Chromatic Aberrations Lateral color (change in magnification with wavelength) image: Smith 2000

75 Spherical Aberration Focus varies with radius on pupil images: Forsyth&Ponce and Hecht 1987

76 Aberrations Coma off-axis will focus to different locations depending on lens region (magnification varies with ray height) images: Smith 2000 and Hecht 1987

77 Coma

78 Astigmatism The shape of the lens for an of center point might look distorted, e.g. elliptical different focus for tangential and sagittal rays image: Smith 2000 Hardy&Perrin

79 Astigmatism

80 Astigmatism red - unsharp

81 Field Curvature focus plane is actually curved Object Image

82 Field Curvature

83 Field Curvature different image distance

84 Bad Optics curvature of field, coma, chromatic aberration

85 Aberration Correction By deconvolution or correcting optics Correcting optics example Hubble space telescope uncorrected optics corrected optics PFS before correction Deconvolution (lecture 3 for details) uncorrected optics deconvolved with APEX [Carasso06]

86 Geometric Aberrations Distortion

87 Distortion Ratios of lengths are no longer preserved. Object Image

88 Geometric distortion Change in magnification with image position image: Smith 2000

89 Radial Distortion image: Kingslake

90 Radial distortion Barrel distortion Pincushion distortion

91 Radial distortion Straight lines are no longer straight Violation of our linear camera model Must be compensated for computer vision applications Radial distortion function can be approximated by a Taylor expansion in radial direction Maps radius to a new radius One or two coefficients are usually sufficient

92 Radial distortion Barrel distortion Pincushion distortion c c

93 Radial Distortion Correction - Example Camera output Corrected Checkerboard serves as target for estimation Corner positions define line segments Minimize curvature

94 Reminder: Image warping T(x,y) y y x f(x,y) x g(x,y ) Knowing the transformation (x,y )=T(x,y) and the source image f(x,y), how to compute the transformed image g(x,y )=f(t(x,t))? 94

95 Reminder: Inverse warping y T -1 (x,y) y x x f(x,y) g(x,y ) Find the position of each pixel g(x,y ) in The source image: (x,y) = T -1 (x,y ) Interpolate between the sampled neighbor positions (nearest-neighbor, bilinear ) 95

96 Contrast Issues

97 Radial Falloff Vignetting your lens is basically a long tube. Cos^4 falloff rule of thumb. At an angle, area of aperture reduced by cos(a) 1/r^2: Falls off as 1/cos(a)^2 (due to increased distance to lens) Light falls on film plane at an angle, another cos(a) reduction.

98 Vignetting - Example a white diffuse target White field at different f-stops f/2.8 f/2.8 actual photograph f/4 f/8 Compensation: divide image by pre-acquired white field

99 Bibliography Holst, G. CCD Arrays, Cameras, and Displays. SPIE Optical Engineering Press, Bellingham, Washington, Theuwissen, A. Solid-State Imaging with Charge- Coupled Devices. Kluwer Academic Publishers, Boston, Curless, CSE558 lecture notes (UW, Spring 01). El Gamal et al., EE392b lecture notes (Spring 01). Several Kodak Application Notes at pplicationnotes.jhtml Reibel et al., CCD or CMOS camera noise characterization, Eur. Phys. J. AP 21, 2003

100 ICC Profiles and HDR Image Generation profile connection spaces CIELAB (perceptual linear) linear CIEXYZ color space can be used to create an high dynamic range image in the profile connection space allows for a color calibrated workflow... input device (e.g. camera) input profile profile connection space output profile output device (e.g. printer)

Introduction to Image Processing and Computer Vision -- Noise, Dynamic Range and Color --

Introduction to Image Processing and Computer Vision -- Noise, Dynamic Range and Color -- Winter 2013 Ivo Ihrke Organizational Issues I received your email addresses Course announcements will be send via