Computational Cameras. Rahul Raguram COMP

Size: px

Start display at page:

Download "Computational Cameras. Rahul Raguram COMP"

Marlene Barker
6 years ago
Views:

1 Computational Cameras Rahul Raguram COMP

2 What is a computational camera? Camera optics Camera sensor 3D scene Traditional camera Final image Modified optics Camera sensor Image Compute 3D scene Computational camera Coded image Additional information

3 Computational cameras - examples * Catadioptric cameras * Source: S. K. Nayar, 2006

4 Computational cameras - examples * * Catadioptric cameras HDR imaging with assorted pixels * Source: S. K. Nayar, 2006

5 Computational cameras - examples * * Catadioptric cameras HDR imaging with assorted pixels * # Multiview radial cameras Time-of-flight cameras * Source: S. K. Nayar, 2006 # Source: L. Guan and M. Pollefeys, 2008

The aperture Glossographia Anglicana Nova, 1707

$diaphragm) Expressed as a fraction of focal length$ 0 with a 50mm lens: aperture is 25mm f/2.

6 The aperture Glossographia Anglicana Nova, 1707 Diameter of the lens opening (controlled by diaphragm) Expressed as a fraction of focal length (f-number) f/2.0 with a 50mm lens: aperture is 25mm f/2.0 with a 100mm lens: aperture is 50mm Typical f-numbers: f/1.4, f/2, f/2.8, f/4, f/5.6, f/8 see a pattern?

7 Varying the aperture Small aperture large depth of field

8 Varying the aperture Large aperture small depth of field Bokeh (derived from Japanese boke ぼけ, a noun form of bokeru ぼける, "become blurred or fuzzy")

9 Multi-Aperture Photography Paul Green MIT CSAIL Wenyang Sun MERL Wojciech Matusik MERL Frédo Durand MIT CSAIL Slides by Green et al.

10 Motivation Depth of Field Control Shallow Depth of Field Portrait Landscape Large Aperture Large Depth of Field Small Aperture

11 Depth and Defocus Blur sensor lens plane of focus circle of confusion subject rays from point in focus converge to single pixel defocus blur depends on distance from plane of focus

12 Defocus Blur & Aperture circle of confusion sensor lens aperture plane of focus subject defocus blur depends on aperture size

13 Goals Aperture size is a critical parameter for photographers post-exposure depth of field control extrapolate shallow depth of field beyond physical aperture

14 Outline Multi-Aperture Camera New camera design Capture multiple aperture settings simultaneously Applications Depth of field control Depth of field extrapolation (Limited) refocusing

et al 05 Georgiev et al 06 Split-Aperture

al 07 Wavefront Coding Dowski and Cathey

and Wang 92 McGuire et al 07 Georgiev et

15 Related Work Computational Cameras Plenoptic Cameras Adelson and Wang 92 Ng et al 05 Georgiev et al 06 Split-Aperture Camera Aggarwal and Ahuja 04 Optical Splitting Trees McGuire et al 07 Coded Aperture Levin et al 07 Veeraraghavan et al 07 Wavefront Coding Dowski and Cathey 95 Depth from Defocus Pentland 87 Adelson and Wang 92 McGuire et al 07 Georgiev et al 06 Aggarwal and Ahuja 04 Levin et al 07 Veeraraghavan et al 07

16 Plenoptic Cameras Capture 4D LightField 2D Spatial (x,y) 2D Angular (u,v Aperture) Lens Aperture v Trade resolution for flexibility after capture Refocusing Depth of field control Improved Noise Characteristics Lenslet Array u Subject Sensor (x,y) Lens (u,v)

17 1D vs 2D Aperture Sampling Aperture v u 2D Grid Sampling

18 1D vs. 2D Aperture Sampling Aperture Aperture v 45 Samples 4 Samples u 2D Grid Sampling 1D Ring Sampling

19 Goals post-exposure depth of field control extrapolate shallow depth of field (limited) refocusing 1d sampling no beamsplitters single sensor removable

20 Optical Design Principles 3D sampling 2D spatial 1D aperture size 1 image for each ring Aperture Sensor

21 Aperture Splitting Mirrors Focusing lenses Sensor Incoming light Tilted Mirrors

22 Aperture Splitting Ideally at aperture plane, but not physically possible! Solution: Relay Optics to create virtual aperture plane Photographic Relay Lens system Aperture splitting optics Aperture Plane New Aperture Plane

23 Optical Prototype mirrors lenses SLR Camera main lens relay optics tilted mirrors Mirror Close-up

24 Sample Data Raw data from our camera

25 Point Spread Function Occlusion inner ring 1 ring 2 outer combined Ideally would be rings Gaps are from occlusion

26 Outline Multi-Aperture Camera New camera design Capture multiple aperture settings simultaneously Applications Depth of field control Depth of field extrapolation Refocusing

27 DOF Navigation I0 I2 I I 1 3

28 DOF Extrapolation? Approximate defocus blur as convolution I n = I K( σ 0 n I 0 I 1 I 2 I 3 ) Depends on depth and aperture size? I E K( σ n ) - Circular aperture blurring kernel

29 DOF Extrapolation Roadmap capture estimate blur fit model extrapolate blur Blur size Largest physical aperture I I I 3 2 I 1 0 I E Aperture Diameter

30 Summary Multi-aperture camera 1D sampling of aperture Removable Post-Exposure depth of field control Depth of field extrapolation

31 Image and Depth from a Conventional Camera with a Coded Aperture Anat Levin, Rob Fergus, Frédo Durand, William Freeman MIT CSAIL Slides by Levin et al.

32 Single input image: Output #1: Depth map

33 Output #1: Depth map Single input image: Output #2: All-focused image

34 Output #1: Depth map Single input image: Output #2: All-focused image

37 Lens and defocus Lens aperture Image of a point light source Lens Camera sensor Point spread function Focal plane

38 Lens and defocus Lens aperture Image of a defocused point light source Object Lens Camera sensor Point spread function Focal plane

39 Lens and defocus Lens aperture Image of a defocused point light source Object Lens Camera sensor Point spread function Focal plane

40 Lens and defocus Lens aperture Image of a defocused point light source Object Lens Camera sensor Point spread function Focal plane

41 Lens and defocus Lens aperture Image of a defocused point light source Object Lens Camera sensor Point spread function Focal plane

42 Depth and defocus Out of focus Depth from defocus: Infer depth by analyzing local scale of defocus blur In focus

43 Challenges Hard to discriminate a smooth scene from defocus blur? Out of focus Hard to undo defocus blur Input Ringing with conventional deblurring algorithm

44 Key contributions Exploit prior on natural images - Improve deconvolution - Improve depth discrimination Natural Unnatural Coded aperture (mask inside lens) - make defocus patterns different from natural images and easier to discriminate

45 Defocus as local convolution Input defocused image Calibrated blur kernels at different depths

46 Defocus as local convolution Input defocused image yy = f x k Local sub-window k Calibrated blur kernels at depth k Sharp sub-window Depth k=1: y = f k x Depth k=2: y = f k x Depth k=3: y = f k x

47 Overview Try deconvolving local input windows with different scaled filters: =? Larger scale =? Correct scale =? Smaller scale Somehow: select best scale.

48 Hard to deconvolve even when kernel is known Challenges Input Ringing with the traditional Richardson-Lucy deconvolution algorithm Hard to identify correct scale: =? Larger scale =? Correct scale? = Smaller scale

49 Deconvolution is ill posed f x = y? =

50 Deconvolution is ill posed f x = y Solution 1:? = Solution 2:? =

51 Idea 1: Natural images prior What makes images special? Natural Unnatural Image gradient Natural images have sparse gradients put a penalty on gradients

52 Deconvolution with prior x = argmin f x y 2 + λ i ρ( x ) i Convolution error Derivatives prior 2 _ +? Equal convolution error Low 2? _ + High

edu/graphics/codedaperture/ Input ρ ( x) = x spread gradients

53 Comparing deconvolution algorithms (Non blind) deconvolution code available online: Input ρ ( x) = x spread gradients 2 ρ ( x) = x 0.8 localizes gradients Richardson-Lucy Gaussian prior Sparse prior

Comparing deconvolution algorithms (Non blind) deconvolution code available online: http://groups.csail.mit.

54 Comparing deconvolution algorithms (Non blind) deconvolution code available online: Input ρ ( x) = x spread gradients 2 ρ ( x) = x 0.8 localizes gradients Richardson-Lucy Gaussian prior Sparse prior

55 Recall: Overview Try deconvolving local input windows with different scaled filters: = Larger scale? = Correct scale? = Smaller scale? Somehow: select best scale. Challenge: smaller scale not so different than correct

56 Idea 2: Coded Aperture Mask (code) in aperture plane - make defocus patterns different from natural images and easier to discriminate Conventional aperture Our coded aperture

57 Solution: lens with occluder Object Lens Camera sensor Point spread function Focal plane

58 Solution: lens with occluder Aperture pattern Image of a defocused point light source Object Lens with coded aperture Camera sensor Point spread function Focal plane

59 Solution: lens with occluder Aperture pattern Image of a defocused point light source Object Lens with coded aperture Camera sensor Point spread function Focal plane

60 Solution: lens with occluder Aperture pattern Image of a defocused point light source Object Lens with coded aperture Camera sensor Point spread function Focal plane

61 Solution: lens with occluder Aperture pattern Image of a defocused point light source Object Lens with coded aperture Camera sensor Point spread function Focal plane

62 Solution: lens with occluder Aperture pattern Image of a defocused point light source Object Lens with coded aperture Camera sensor Point spread function Focal plane

63 Why coded? Coded aperture- reduce uncertainty in scale identification Conventional Coded Larger scale Correct scale Smaller scale

64 Depth results

Regularizing depth estimation Try deblurring with

window + regularization 200 235 245 255 265 275

65 Regularizing depth estimation Try deblurring with 10 different aperture scales x = argmin f x _ y 2 Convolution error + λ 2 + i ρ( x ) Derivatives prior i Keep minimal error scale in each local window + regularization Input Local depth estimation Regularized depth

66 Regularizing depth estimation Local depth estimation Input Regularized depth 305

67 Sometimes, manual intervention Input Local depth estimation Regularized depth 305 After user corrections 305

68 All focused results

69 Input

70 All-focused (deconvolved)

71 Close-up Original image All-focus image

72 Input

73 All-focused (deconvolved)

74 Close-up Original image All-focus image Naïve sharpening

75 Comparison- conventional aperture result Ringing due to wrong scale estimation

76 Comparison- coded aperture result

77 Application: Digital refocusing from a single image

78 Application: Digital refocusing from a single image

79 Application: Digital refocusing from a single image

80 Application: Digital refocusing from a single image

81 Application: Digital refocusing from a single image

82 Application: Digital refocusing from a single image

83 Application: Digital refocusing from a single image

84 Coded aperture: pros and cons Image AND depth at a single shot No loss of image resolution Simple modification to lens Depth is coarse unable to get depth at untextured areas, might need manual corrections But depth is a pure bonus Loss some light But deconvolution increases depth of field

85 Deconvolution code available

86 50mm f/1.8: $79.95 Cardboard: $1 Tape: $1 Depth acquisition: priceless

Dappled Photography: Mask Enhanced Cameras for Heterodyned Light Fields and Coded Aperture Refocusing

Dappled Photography: Mask Enhanced Cameras for Heterodyned Light Fields and Coded Aperture Refocusing Ashok Veeraraghavan, Ramesh Raskar, Ankit Mohan & Jack Tumblin Amit Agrawal, Mitsubishi Electric Research