Dappled Photography: Mask Enhanced Cameras for Heterodyned Light Fields and Coded Aperture Refocusing

Size: px

Start display at page:

Download "Dappled Photography: Mask Enhanced Cameras for Heterodyned Light Fields and Coded Aperture Refocusing"

Cleopatra Hart
5 years ago
Views:

1 Dappled Photography: Mask Enhanced Cameras for Heterodyned Light Fields and Coded Aperture Refocusing Ashok Veeraraghavan, Ramesh Raskar, Ankit Mohan & Jack Tumblin Amit Agrawal, Mitsubishi Electric Research Labs (MERL), Cambridge, MA, USA Northwestern University, IL

2 Coded Exposure [Raskar, Agrawal, Tumblin SIGGRAPH 2006]

3 Coded Exposure (Flutter Shutter) Camera Raskar, Agrawal, Tumblin [Siggraph2006] Coding in Time: Shutter is opened and closed

4 Blurring == Convolution Sharp Photo Blurred Photo PSF == Sinc Function Traditional Camera: Shutter is OPEN: Box Filter ω

5 Coded Exposure Sharp Photo Blurred Photo PSF == Broadband Function Preserves High Spatial Frequencies Flutter Shutter: Shutter is OPEN and CLOSED

6 Traditional Coded Exposure Deblurred Image Deblurred Image Image of Static Object

7 How to handle focus blur?

8 Coded Exposure Coded Aperture Temporal 1-D broadband code: Motion Deblurring Spatial 2-D broadband mask: Focus Deblurring

9 In Focus Photo Point light source (LED)

10 Out of Focus Photo: Open Aperture

11 Lens and defocus Lens aperture Image of a point light source Lens Camera sensor Point spread function Focal plane Slide Credit: Levin et. al

12 Lens and defocus Lens aperture Image of a defocused point light source Object Lens Camera sensor Point spread function Focal plane Slide Credit: Levin et. al

13 Lens and defocus Lens aperture Image of a defocused point light source Object Lens Camera sensor Point spread function Focal plane Slide Credit: Levin et. al

14 Lens and defocus Lens aperture Image of a defocused point light source Object Lens Camera sensor Point spread function Focal plane Slide Credit: Levin et. al

15 Lens and defocus Lens aperture Image of a defocused point light source Object Lens Camera sensor Point spread function Focal plane Slide Credit: Levin et. al

16 Out of Focus Photo: Coded Aperture

17 Blurred Photos Open Aperture Coded Aperture, 7 * 7 Mask

18 Deblurred Photos Open Aperture Coded Aperture, 7 * 7 Mask

19 Captured Blurred Photo

20 Full Resolution Digital Refocusing

21 Blur Estimation & Segmentation Defocus blur dependent on depth Assumptions Layered Lambertian Scene Constant blur within each layer Deblur at different blur sizes k k = 1 Captured Blurred Photo k = 10

22 Define Cost Function k = 1 k = 1 k = 10 Deblurred Images k = 10 Cost Function Images Likelihood Error: (Blurred image - Sharp Image * PSF k ) 2 Gradient Error: Natural Image Statistics, Gradient Kurtosis

23 Blur Estimation & Segmentation == Labeling Graph cuts for labeling k = 1 K = 1 k = 10 Error Images K = 7

24 Captured Photo Reblur Deblur, k = 7 Fusion

26 Less is More Blocking Light == More Information Coded Exposure Coding in Time Coded Aperture Coding in Space

27 Flexible Depth of Field Photography Nagahara, Kuthirammal, Zhou, and Nayar ECCV 2008 Slide-deck credit: Nagahara et al.

29 Hardware Setup

30 Captured Image Aperture f/1.4, Exposure 0.36 sec

31 Deblurred EDOF image

32 Single traditional Image Aperture f/1.4, Exposure 0.36 sec

33 Single image with same EDOF Aperture f/8, Exposure 0.36 sec

34 Captured Image Aperture f/1.4, Exposure 0.36 sec

35 Deblurred EDOF image

36 Single traditional Image Aperture f/1.4, Exposure 0.36 sec

37 Single image with same EDOF Aperture f/8, Exposure 0.36 sec

38 Tunable focus ring

39 Discontinuous DOF

40 Discontinuous DOF Aperture f/11

41 Discontinuous DOF Aperture f/1.4

42 Tilted DOF

43 Image from normal camera Aperture f/1.4

44 Tilted DOF Aperture f/1.4

45 Non-planar DOF

46 Image from a normal camera Aperture f/1.4

47 Non-planar DOF Aperture f/1.4

48 Multi-Aperture Photography Paul Green MIT CSAIL Wenyang Sun MERL Wojciech Matusik MERL Frédo Durand MIT CSAIL

49 Motivation Depth of Field Control Portrait Landscape Large Aperture Shallow Depth of Field Small Aperture Large Field Depth of

50 Depth and Defocus Blur sensor lens plane of focus circle of confusion subject rays from point in focus converge to single pixel defocus blur depends on distance from plane of focus

51 Defocus Blur & Aperture sensor lens aperture plane of focus circle of confusion subject defocus blur depends on aperture size

52 Goals Aperture size is a critical parameter for photographers post-exposure depth of field control extrapolate shallow depth of field beyond physical aperture

53 Outline Multi-Aperture Camera New camera design Capture multiple aperture settings simultaneously Applications Depth of field control Depth of field extrapolation (Limited) refocusing

et al 05 Georgiev et al 06 Split-Aperture

al 07 Wavefront Coding Dowski and Cathey

and Wang 92 McGuire et al 07 Georgiev et

54 Related Work Computational Cameras Plenoptic Cameras Adelson and Wang 92 Ng et al 05 Georgiev et al 06 Split-Aperture Camera Aggarwal and Ahuja 04 Optical Splitting Trees McGuire et al 07 Coded Aperture Levin et al 07 Veeraraghavan et al 07 Wavefront Coding Dowski and Cathey 95 Depth from Defocus Pentland 87 Adelson and Wang 92 McGuire et al 07 Georgiev et al 06 Aggarwal and Ahuja 04 Levin et al 07 Veeraraghavan et al 07

55 Plenoptic Cameras Capture 4D LightField 2D Spatial (x,y) 2D Angular (u,v Aperture) Lens Aperture v Trade resolution for flexibility after capture Refocusing Depth of field control Improved Noise Characteristics Lenslet Array u Subject Sensor (x,y) Lens (u,v)

56 1D vs 2D Aperture Sampling Aperture v u 2D Grid Sampling

57 1D vs. 2D Aperture Sampling Aperture Aperture 45 Samples v 4 Samples u 2D Grid Sampling 1D Ring Sampling

58 Optical Design Principles 3D sampling 2D spatial 1D aperture size 1 image for each ring Aperture Sensor

59 Aperture Splitting Goal: Split aperture into 4 separate optical paths concentric tilted mirrors at aperture plane Tilted Mirrors

60 Aperture Splitting Mirrors Focusing lenses Sensor Incoming light Tilted Mirrors

61 Aperture Splitting Ideally at aperture plane, but not physically possible! Solution: Relay Optics to create virtual aperture plane Photographic Relay Lens system Aperture splitting optics X Aperture Plane New Aperture Plane

62 Optical Prototype mirrors lenses SLR Camera main lens relay optics tilted mirrors Mirror Close-up

63 Sample Data Raw data from our camera

64 Point Spread Function Occlusion inner ring 1 ring 2 outer combined Ideally would be rings Gaps are from occlusion

65 Outline Multi-Aperture Camera New camera design Capture multiple aperture settings simultaneously Applications Depth of field control Depth of field extrapolation Refocusing

66 DOF Navigation I0 I2 I I 1 3

67 DOF Extrapolation? Approximate defocus blur as convolution I n I K ( 0 n I 0 I 1 I 2 I 3 ) I E Depends on depth and aperture size What is at each pixel in?? E I E K ( n ) - Circular aperture blurring kernel

68 Blur size DOF Extrapolation Roadmap capture estimate blur fit model extrapolate blur Largest physical aperture I E I I I 3 2 I 1 0 Aperture Diameter

69 Blur size Defocus Gradient Defocus blur σ D σ Blur proportional to aperture diameter I 1 I I 3 2 I 0 Largest physical aperture I E d s sensor distance G ( d Defocus Gradient s (d s f ) d fd G f ) d fd o o d o focal length fd object distance s o o fd s D Aperture Diameter D G is slope of this line Defocus Gradient Map aperture diameter

70 Optimization solve for discrete defocus gradient values G at each pixel Data term 1 D ( G) I I K ( G ) Graph Cuts with spatial regularization term i i 0 N i Smallest Aperture Image Defocus Gradient Map

71 Depth of Field Extrapolation

72 Synthetic Refocusing Modify gradient labels and re-synthesize image gradient map refocused map extrapolated f/1.8 refocused synthetic f/1.8

Discussion Occlusion Could help depth discrimination (coded aperture) Difficult alignment process Mostly because prototype Refocusing limited by

73 Discussion Occlusion Could help depth discrimination (coded aperture) Difficult alignment process Mostly because prototype Refocusing limited by Depth of Field helped by depth-guided deconvolution Texture required for accurate defocus gradient map Not critical for depth of field and refocus

74 74 4D Frequency Analysis of Computational Cameras for Depth of Field Extension Anat Levin 1,2 Sam Hasinoff 1 Paul Green 1 Frédo Durand 1 Bill Freeman 1 1 MIT CSAIL 2 Weizmann Institute

75 Defocus blur in a standard lens 75 At focus depth, sharp Away from focus depth, blurred

76 Small aperture increased depth of field but noisy 76 Depth 1: sharp but noisy Depth 2: sharp but noisy

77 Extended depth of field cameras 77 input output odified optics Deconvolution Extended DOF cameras: remove blur computationally and design optics to make that easy

78 In this talk 78 How much can depth of field be extended? New lens extending depth of field

79 The lattice-focal lens 79 Our design: assembly of subsquares with different focal powers each element focuses on a different depth toy lattice-focal lens with 4 elements E s ( x, y ) 2 S 4 / 3 A 8/ 3 1/ 3 x, y

80 Hardware construction 80 Proof of concept 12 subsquares cut from plano-convex spherical lenses Attached to main lens extra focal power needed very low Modest DOF extension with only 12 subsquares

81 Depth estimation 81 Defocus kernels vary with depth defocus kernels at different depths Depth estimation as for the coded aperture camera [Levin et al. 07] input depth map

82 Standard lens reference 82

83 Lattice-focal lens

84 Standard lens reference 84

85 Lattice-focal lens

86 Standard lens reference 86

87 Results Lattice-focal lens

88 Application: Refocusing from single captured image 88

89 Application: Refocusing from single captured image 89

90 Application: Refocusing from single captured image 90

91 The lattice-focal lens limitations 91 Depth estimation needed for deblurring Only capture part of the 4D light field spectrum Subsquare size and focal power optimized for a given focusing range Higher spectrum than previous designs, but does not reach the upper bound

Computational Cameras. Rahul Raguram COMP

Computational Cameras. Rahul Raguram COMP Computational Cameras Rahul Raguram COMP 790-090 What is a computational camera? Camera optics Camera sensor 3D scene Traditional camera Final image Modified optics Camera sensor Image Compute 3D scene