When Does Computational Imaging Improve Performance?

Size: px

Start display at page:

Download "When Does Computational Imaging Improve Performance?"

Lesley Sparks
6 years ago
Views:

1 When Does Computational Imaging Improve Performance? Oliver Cossairt Assistant Professor Northwestern University Collaborators: Mohit Gupta, Changyin Zhou, Daniel Miau, Shree Nayar (Columbia University)

2 Iphone 4G Photograph Canon DLSR Photograph How to capture the best quality photograph?

3 Digital vs. Film Photography Film Camera Dynamic range fixed at time of exposure 1ms Exposure Time 4ms Exposure Time 8ms Exposure Time 16ms Exposure Time Digital Camera Dynamic range can be extended computationally High Dynamic Range Image

Digital Photography: Noise Film Camera Noise level fixed at capture time Limited by film grain size Milky Way, 6 min exposure (Jesse Levinson) Digital Camera Noise can be

4 Digital Photography: Noise Film Camera Noise level fixed at capture time Limited by film grain size Milky Way, 6 min exposure (Jesse Levinson) Digital Camera Noise can be averaged away SNR unlimited in principle Milky Way, 10x 6 min exposures averaged

5 Computational Imaging: Increased Functionality Take multiple pictures and computationally combine HDR Imaging Panoramic Stitching Light Field Capture [Wilburn et al. 04] Image-Based Lighting [Debevec et al. 00] Digital Holography [Greenbum et al. 12] Others Multiview Stereo Depth from Focus/Defocus Tomography Structured Light Deconvolution microscopy etc.

6 Focus Blur Film Camera Blur is fixed at time of capture M100 Galaxy captured by Hubble Telescope Digital Camera Images can be deblurred via deconvolution M100 Galaxy after blind deconvolution [Caraso, Opt. Eng 06]

7 Motion Blur Film Camera Short exposure avoids motion blur Image is noisy 1 millisecond exposure (noisy) Digital Camera Long exposure produces blurry image Blur can be removed via deconvolution 50 millisecond exposure (blurry) (deblurred)

8 Coded Blur and Multiplexing Camera Exposure 50 millisec Time [Raskar et al. 06]

9 Coded Blur and Multiplexing Camera Exposure Blur is shifted and summed copies 50 millisec Time [Raskar et al. 06]

10 Coded Blur and Multiplexing Camera Exposure Which copies to keep? 50 millisec Time [Raskar et al. 06]

11 Computational Imaging: Increased Performance Coded image capture for increased performance Coded Aperture Defocus Blur Motion Blur [Dowski, Cathey 96] [Mertz 65] [Gottesman 89] [Hausler 72] [Nagahara 08] [Levin et al. 07] [Zhou, Nayar 08] [Raskar 06] [Levin 08] [Cho 10] Multi/Hyper-Spectral Light Field Capture Reflectance [Sloane 79] [Hanley 99] [Baer 99] [Wetzstein et al., 12] [Lanman 08] [Veeraraghavan 07] [Liang 08] [Schechner 03] [Ratner 07] [Ratner 08]

12 Coded Imaging Performance Camera Exposure Camera Exposure 50 millisec Time 50 millisec Time Vs. Short Exposure Coded Exposure Deblurred Image When does computational imaging improve performance?

13 Measuring Computational Imaging Performance

14 Image Formation Model Scene Image Computational Camera Coded Image Coded Image Coding Matrix Noise Optical Coding Equation No diffraction Fully determined Assumption: A) Linear model of incoherent image formation

Affine Noise Model Noise Variance at k th Pixel: photon noise

electronics, ADC s, quantization Signal dependent /

noise modeled as Gaussian (ok for more than 10 photons) Photon

15 Affine Noise Model Noise Variance at k th Pixel: photon noise aperture, lighting, pixel size Noise PDF: read noise electronics, ADC s, quantization Signal dependent / independent noise Ignore Dark current, fixed pattern Photon noise modeled as Gaussian (ok for more than 10 photons) Photon noise spatially averaged Assumption: B) Affine noise model (photon noise is Gaussian)

16 Lighting Conditions Signal-level and photon noise depend on illumination Average Signal (e - ) Illumination (lux) Reflectivity Aperture Exposure Time (s) Quantum Efficiency Pixel Size (m) Ex) q=.5, R =.5, F/8, t = 6ms, p=6um Quarter moon Full moon Twilight Indoor lighting Cloudy day Sunny day Illumination I src (lux) Signal level J (e - ) Assumption: C) Naturally occurring light conditions for photography [Cossairt et al. TIP 12]

17 Measuring Performance For Gaussian noise, Mean-Squared-Error (MSE) can be computed analytically Ex) Coded Motion Deblurring Long Exposure Coded Exposure Observation: 1) Multiplexing performance depends on coding matrix

18 Multiplexing vs. Impulse Imaging Impulse imaging (identity sampling) Noise variance Coded imaging (multiplexed sampling) Noise variance Observation: Increased throughput 2) Multiplexing increases signal-dependent noise

19 Multiplexing vs. Impulse Imaging SNR Gain over impulse imaging: Hadamard Multiplexing: Noise Dependent Coding Dependent Decreases with C Increases with C [Sloane 79] No SNR gain for large signal Coded Aperture Astronomy Increasing scene points Fresnel zone plate [Mertz 65] Observation: Decreasing contrast 3) Performance depends on multiplexing and signal prior

the Maximum A Posteriori (MAP) estimate Data term Prior term MSE difficult to

20 Image Prior Models Assume we have a PDF for images, e.g. Power Spectra Prior Other priors Total Variation (TV) Wavelet/sparsity prior Learned priors (K-SVD) Compute the Maximum A Posteriori (MAP) estimate Data term Prior term MSE difficult to express analytically when Assumption: D) Signal prior models naturally occurring images

21 Image Priors and Noise Denoise Twilight (10 lux) PSNR = 5.5 db PSNR = 16.4 db Denoise Daylight 5 (10 lux) PSNR = 35 db PSNR = 35.9 db Observation: 4) Signal priors help more at low light levels

Observations: 1) Multiplexing performance depends on coding matrix 2) Multiplexing helps most in low light 3) Performance depends on both multiplexing and signal prior 4) Signal

22 Observations: 1) Multiplexing performance depends on coding matrix 2) Multiplexing helps most in low light 3) Performance depends on both multiplexing and signal prior 4) Signal priors help most in low light Assumptions A) Incoherent imaging B) Affine noise model C) Natural lighting conditions D) Natural image prior How to capture the best quality photograph?

23 Example: Motion Deblurring

24 Motion Deblurring vs. Impulse Imaging Optical efficiency (C) = total on time Camera Exposure 50 millisec Time 50 millisec Time Vs. Impulse Imaging (Short Exposure) Computational Imaging (Coded Exposure) What is the best possible coding performance we can get? [Ratner 07]

25 SNR Gain (G) Multiplexing Performance Bound S-Matrix 4 Average signal level 3 2 Read noise Optical Efficiency (C) [Ratner and Schechner 07]

depends only on lighting conditions! q=.5, R =.5, F/2.

26 When Does Motion Deblurring Improve Performance? Upper Bound on SNR Gain: Read Noise Average signal level Performance depends only on lighting conditions! q=.5, R =.5, F/2.1, p = 1um, Motion Invariant Levin et al. Flutter Shutter Raskar et al. Maximum object speed (pixels/sec) [Cossairt et al. TIP 12]

27 Flutter Shutter Simulation q=.5, R =.5, F/2.1, pixel size = 1um, read noise Impulse (4ms) Flutter Shutter (180ms) Deblurred Twilight (10 lux) PSNR = -7.2 db PSNR = -3.0 db Cloudy Day 3 (10 lux) PSNR = 12.4 db PSNR = 10.1 db

28 Example: Extended DOF Imaging

29 Depth of Field Small DOF Microscope Tachinid Fly

30 Depth of Field Large DOF Microscope Tachinid Fly

Depth of Field and Noise Image Lens F 1.4 2.8 5.6 8.

31 Depth of Field and Noise Image Lens F Small apertures have large depth of field and low SNR

32 Focal Sweep Sensor Lens (depth) Point Spread Function (PSF) [Hausler 72, Nagahara et al. 08]

Focal Sweep Sensor Lens 400 600 900 1200 1500 1700 2000 (depth) + + + + + + = t = 1 t = 2 t = 3 t = 4 t =

33 Focal Sweep Sensor Lens (depth) = t = 1 t = 2 t = 3 t = 4 t = 5 t = 6 t = 7 (400) (600) (900) (1200) (1500) (1700) (2000) Integrated PSF [Hausler 72, Nagahara et al. 08]

34 Quasi Depth Invariant PSF Focal Sweep PSF Traditional Camera PSF mm 750mm mm mm 2000mm mm mm 2000mm Extended depth of field with a single deconvolution

35 Extended Depth of Field Telescope 75 m 50 m Traditional Image 75 m 50 m Meade LX200 8 Telescope 2000mm FL Focal Sweep: Processed Captured

36 Focal Sweep Without Moving Parts Focal Sweep Image Lens Diffusion Coding Image (No Moving Parts) Lens 500 x 3 micron [Cossairt et al. Siggraph 10] Radial Diffuser

37 RMS Deblurring Error Diffusion Coding: Evaluation [Dowski and Cathey 95] [Hausler 72, Nagahara et al. 08] Cubic Phase Plate Focal Sweep Diffusion Coding.01 noise Depth (mm) Diffusion coding gives best performance without moving parts [Cossairt et al. Siggraph 10]

38 Diffusion Coding vs. Traditional Camera Traditional F/1.8 Diffusion Coding F/1.8 (Captured) Traditional F/18 (Normalized) Diffusion Coding F/1.8 (Deblurred)

39 Face Detection Traditional Camera (F/2.0) Diffusion Coding Camera (F/2.0)

40 Diffusion Coded Telescope: Optical Design Diffuser Annular Aperture Mirror 2 Mirror 1 Sensor 8 dia 80 Focal Length

41 Telephoto Focal Sweep with Deformable Optics Canon 800mm EFL Lens Sensor Deformable Lens [Miau et al. ICCP 13]

42 Telephoto Video Quality Comparison Conventional EDOF (Deformable Lens)

43 Focal Sweep Performance Impulse Camera Noise Variance: Mean-Squared Error: sensor lens A Focal Sweep Noise Variance: Mean-Squared Error: sensor lens C*A diffuser light increase [Cossairt et al. TIP 12]

44 When Does Defocus Deblurring Improve Performance? Focal sweep multiplexing gain can be expressed analytically Read Noise Average signal level Performance depends only on lighting conditions! q=.5, R =.5, t = 20ms, p = 5um, Maximum defocus at F/1 (pixels) [Cossairt et al. TIP 12]

45 Focal Sweep Simulation Pixel size = 5um Traditional Traditional Focal Sweep Read noise (F/2.0) (F/20.0) (F/2.0) Twilight (10 lux) PSNR = 5.5 db PSNR = 18.5 db Daylight 5 (10 lux) PSNR = 35 db PSNR = 38.5 db

46 Focal Sweep Simulation (with Prior) Pixel size = 5um Traditional Traditional Focal Sweep Read noise (F/2.0) (F/20.0) (F/2.0) Twilight (10 lux) PSNR = 16.4 db / 5.5 db PSNR = 22.8 db / 18.5 db Daylight 5 (10 lux) PSNR = 35.9 db / 35 db PSNR = 39.6 db / 38.5 db BM3D Algorithm: [Dabov et al. 06]

47 Simulated Focal Sweep Performance Focal Sweep Performance Impulse Imaging Focal Sweep performance bound is weak at low light levels [Cossairt et al., TIP 12]

48 Conclusions Results for Motion Deblurring, EDOF also applicable to many other computational cameras Computational imaging performance should always be measured relative to impulse imaging Computational imaging performance depends jointly on multiplexing, noise, and signal priors Important question: How much performance improvement from multiplexing above and beyond use of signal priors?

49 Visual Quality Metrics SSIM Metric UQI Metric VIF SSIM Metric Performance bound roughly holds for all metrics [Cossairt et al., TIP 12]

50 Computational Gigapixel Camera Computational Camera Design Prototype Camera Ball Ball Lens Lens Sensor Lens Array Array Ball Lens Sensor Computations Gigapixel Image Pan/Tilt Motor Also See: MOSAIC Program, Duke, UCSD, Distant Focus

51 Point Spread Function Resolution vs. Lens Scale Pixels PSF size increases linearly

52 Point Spread Function RMS Deblurring Error Resolution vs. Lens Scale [Cossairt et al. JOSA 11] Pixels PSF size increases linearly Scale (M) Deblurring Error is sub-linear

A Framework for Analysis of Computational Imaging Systems

A Framework for Analysis of Computational Imaging Systems Kaushik Mitra, Oliver Cossairt, Ashok Veeraghavan Rice University Northwestern University Computational imaging CI systems that adds new functionality