Dappled Photography: Mask Enhanced Cameras for Heterodyned Light Fields and Coded Aperture Refocusing Ashok Veeraraghavan, Ramesh Raskar, Ankit Mohan & Jack Tumblin Amit Agrawal, Mitsubishi Electric Research Labs (MERL), Cambridge, MA, USA Northwestern University, IL
Coded Exposure [Raskar, Agrawal, Tumblin SIGGRAPH 2006]
Coded Exposure (Flutter Shutter) Camera Raskar, Agrawal, Tumblin [Siggraph2006] Coding in Time: Shutter is opened and closed
Blurring == Convolution Sharp Photo Blurred Photo PSF == Sinc Function Traditional Camera: Shutter is OPEN: Box Filter ω
Coded Exposure Sharp Photo Blurred Photo PSF == Broadband Function Preserves High Spatial Frequencies Flutter Shutter: Shutter is OPEN and CLOSED
Traditional Coded Exposure Deblurred Image Deblurred Image Image of Static Object
How to handle focus blur?
Coded Exposure Coded Aperture Temporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring
In Focus Photo Point light source (LED)
Out of Focus Photo: Open Aperture
Lens and defocus Lens aperture Image of a point light source Lens Camera sensor Point spread function Focal plane Slide Credit: Levin et. al
Lens and defocus Lens aperture Image of a defocused point light source Object Lens Camera sensor Point spread function Focal plane Slide Credit: Levin et. al
Lens and defocus Lens aperture Image of a defocused point light source Object Lens Camera sensor Point spread function Focal plane Slide Credit: Levin et. al
Lens and defocus Lens aperture Image of a defocused point light source Object Lens Camera sensor Point spread function Focal plane Slide Credit: Levin et. al
Lens and defocus Lens aperture Image of a defocused point light source Object Lens Camera sensor Point spread function Focal plane Slide Credit: Levin et. al
Out of Focus Photo: Coded Aperture
Blurred Photos Open Aperture Coded Aperture, 7 * 7 Mask
Deblurred Photos Open Aperture Coded Aperture, 7 * 7 Mask
Captured Blurred Photo
Full Resolution Digital Refocusing
Blur Estimation & Segmentation Defocus blur dependent on depth Assumptions Layered Lambertian Scene Constant blur within each layer Deblur at different blur sizes k k = 1 Captured Blurred Photo k = 10
Define Cost Function k = 1 k = 1 k = 10 Deblurred Images k = 10 Cost Function Images Likelihood Error: (Blurred image - Sharp Image * PSF k ) 2 Gradient Error: Natural Image Statistics, Gradient Kurtosis
Blur Estimation & Segmentation == Labeling Graph cuts for labeling k = 1 K = 1 k = 10 Error Images K = 7
Captured Photo Reblur Deblur, k = 7 Fusion
Less is More Blocking Light == More Information Coded Exposure Coding in Time Coded Aperture Coding in Space
Flexible Depth of Field Photography Nagahara, Kuthirammal, Zhou, and Nayar ECCV 2008 Slide-deck credit: Nagahara et al.
Hardware Setup
Captured Image Aperture f/1.4, Exposure 0.36 sec
Deblurred EDOF image
Single traditional Image Aperture f/1.4, Exposure 0.36 sec
Single image with same EDOF Aperture f/8, Exposure 0.36 sec
Captured Image Aperture f/1.4, Exposure 0.36 sec
Deblurred EDOF image
Single traditional Image Aperture f/1.4, Exposure 0.36 sec
Single image with same EDOF Aperture f/8, Exposure 0.36 sec
Tunable focus ring
Discontinuous DOF
Discontinuous DOF Aperture f/11
Discontinuous DOF Aperture f/1.4
Tilted DOF
Image from normal camera Aperture f/1.4
Tilted DOF Aperture f/1.4
Non-planar DOF
Image from a normal camera Aperture f/1.4
Non-planar DOF Aperture f/1.4
Multi-Aperture Photography Paul Green MIT CSAIL Wenyang Sun MERL Wojciech Matusik MERL Frédo Durand MIT CSAIL
Motivation Depth of Field Control Portrait Landscape Large Aperture Shallow Depth of Field Small Aperture Large Field Depth of http://photographertips.net
Depth and Defocus Blur sensor lens plane of focus circle of confusion subject rays from point in focus converge to single pixel defocus blur depends on distance from plane of focus
Defocus Blur & Aperture sensor lens aperture plane of focus circle of confusion subject defocus blur depends on aperture size http://photographertips.net
Goals Aperture size is a critical parameter for photographers post-exposure depth of field control extrapolate shallow depth of field beyond physical aperture
Outline Multi-Aperture Camera New camera design Capture multiple aperture settings simultaneously Applications Depth of field control Depth of field extrapolation (Limited) refocusing
Related Work Computational Cameras Plenoptic Cameras Adelson and Wang 92 Ng et al 05 Georgiev et al 06 Split-Aperture Camera Aggarwal and Ahuja 04 Optical Splitting Trees McGuire et al 07 Coded Aperture Levin et al 07 Veeraraghavan et al 07 Wavefront Coding Dowski and Cathey 95 Depth from Defocus Pentland 87 Adelson and Wang 92 McGuire et al 07 Georgiev et al 06 Aggarwal and Ahuja 04 Levin et al 07 Veeraraghavan et al 07
Plenoptic Cameras Capture 4D LightField 2D Spatial (x,y) 2D Angular (u,v Aperture) Lens Aperture v Trade resolution for flexibility after capture Refocusing Depth of field control Improved Noise Characteristics Lenslet Array u Subject Sensor (x,y) Lens (u,v)
1D vs 2D Aperture Sampling Aperture v u 2D Grid Sampling http://photographertips.net
1D vs. 2D Aperture Sampling Aperture Aperture 45 Samples v 4 Samples u 2D Grid Sampling 1D Ring Sampling http://photographertips.net
Optical Design Principles 3D sampling 2D spatial 1D aperture size 1 image for each ring Aperture Sensor http://photographertips.net
Aperture Splitting Goal: Split aperture into 4 separate optical paths concentric tilted mirrors at aperture plane Tilted Mirrors
Aperture Splitting Mirrors Focusing lenses Sensor Incoming light Tilted Mirrors
Aperture Splitting Ideally at aperture plane, but not physically possible! Solution: Relay Optics to create virtual aperture plane Photographic Relay Lens system Aperture splitting optics X Aperture Plane New Aperture Plane
Optical Prototype mirrors lenses SLR Camera main lens relay optics tilted mirrors Mirror Close-up
Sample Data Raw data from our camera
Point Spread Function Occlusion inner ring 1 ring 2 outer combined Ideally would be rings Gaps are from occlusion
Outline Multi-Aperture Camera New camera design Capture multiple aperture settings simultaneously Applications Depth of field control Depth of field extrapolation Refocusing
DOF Navigation I0 I2 I I 1 3
DOF Extrapolation? Approximate defocus blur as convolution I n I K ( 0 n I 0 I 1 I 2 I 3 ) I E Depends on depth and aperture size What is at each pixel in?? E I E K ( n ) - Circular aperture blurring kernel
Blur size DOF Extrapolation Roadmap capture estimate blur fit model extrapolate blur Largest physical aperture I E I I I 3 2 I 1 0 Aperture Diameter
Blur size Defocus Gradient Defocus blur σ D σ Blur proportional to aperture diameter I 1 I I 3 2 I 0 Largest physical aperture I E d s sensor distance G ( d Defocus Gradient s (d s f ) d fd G f ) d fd o o d o focal length fd object distance s o o fd s D Aperture Diameter D G is slope of this line Defocus Gradient Map aperture diameter
Optimization solve for discrete defocus gradient values G at each pixel Data term 1 D ( G) I I K ( G ) Graph Cuts with spatial regularization term i i 0 N i Smallest Aperture Image Defocus Gradient Map
Depth of Field Extrapolation
Synthetic Refocusing Modify gradient labels and re-synthesize image gradient map refocused map extrapolated f/1.8 refocused synthetic f/1.8
Discussion Occlusion Could help depth discrimination (coded aperture) Difficult alignment process Mostly because prototype Refocusing limited by Depth of Field helped by depth-guided deconvolution Texture required for accurate defocus gradient map Not critical for depth of field and refocus
74 4D Frequency Analysis of Computational Cameras for Depth of Field Extension Anat Levin 1,2 Sam Hasinoff 1 Paul Green 1 Frédo Durand 1 Bill Freeman 1 1 MIT CSAIL 2 Weizmann Institute
Defocus blur in a standard lens 75 At focus depth, sharp Away from focus depth, blurred
Small aperture increased depth of field but noisy 76 Depth 1: sharp but noisy Depth 2: sharp but noisy
Extended depth of field cameras 77 input output odified optics Deconvolution Extended DOF cameras: remove blur computationally and design optics to make that easy
In this talk 78 How much can depth of field be extended? New lens extending depth of field
The lattice-focal lens 79 Our design: assembly of subsquares with different focal powers each element focuses on a different depth toy lattice-focal lens with 4 elements E s ( x, y ) 2 S 4 / 3 A 8/ 3 1/ 3 x, y
Hardware construction 80 Proof of concept 12 subsquares cut from plano-convex spherical lenses Attached to main lens extra focal power needed very low Modest DOF extension with only 12 subsquares
Depth estimation 81 Defocus kernels vary with depth defocus kernels at different depths Depth estimation as for the coded aperture camera [Levin et al. 07] input depth map
Standard lens reference 82
Lattice-focal lens
Standard lens reference 84
Lattice-focal lens
Standard lens reference 86
Results Lattice-focal lens
Application: Refocusing from single captured image 88
Application: Refocusing from single captured image 89
Application: Refocusing from single captured image 90
The lattice-focal lens limitations 91 Depth estimation needed for deblurring Only capture part of the 4D light field spectrum Subsquare size and focal power optimized for a given focusing range Higher spectrum than previous designs, but does not reach the upper bound