Introduction to Light Fields

Size: px

Start display at page:

Download "Introduction to Light Fields"

Derick Stewart
6 years ago
Views:

1 MIT Media Lab Introduction to Light Fields Camera Culture Ramesh Raskar MIT Media Lab

2 Introduction to Light Fields Ray Concepts for 4D and 5D Functions Propagation of Light Fields Interaction with Occluders Fourier Domain Analysis and Relationship to Fourier Optics Coded Photography: Modern Methods to Capture Light Field Wigner and Ambiguity Function for Light Field in Wave Optics New Results in Augmenting Light Fields

3 Light Fields Goal: Representing propagation, interaction and image formation of light using purely position and angle parameters Radiance per ray Ray parameterization: position direction Position : s, x, r Direction : u, θ, s Reference plane Courtesy of Se Baek Oh. Used with permission.

$diffraction & interference$ rotational PSF Courtesy of Se

4 Limitations of Traditional Lightfields rigorous but cumbersome wave optics based Traditional Light Field Wigner Distribution Function hologram s beam shaping ray optics based simple and powerful limited in diffraction & interference rotational PSF Courtesy of Se Baek Oh. Used with permission. Se Baek 3D Optical Systems Group CVPR 2009

5 Example: New Representations Augmented Lightfields rigorous but cumbersome wave optics based Wigner Distribution Function WDF Augmented LF Traditional Light Field Traditional Light Field ray optics based simple and powerful limited in diffraction & interference Se Baek 3D Optical Courtesy of Se Baek Oh. Used with permission. Interference & Diffraction Interaction w/ optical elements Non-paraxial propagation Systems Group CVPR 2009

6 The Plenoptic Function Figure removed due to copyright restrictions. Q: What is the set of all things that we can ever see? A: The Plenoptic Function (Adelson & Bergen) Let s s start with a stationary person and try to parameterize everything that he can see

7 Grayscale snapshot Figure removed due to copyright restrictions. P(θ,φ) is intensity of light Seen from a single view point At a single time Averaged over the wavelengths of the visible spectrum (can also do P(x,y), but spherical coordinate are nicer)

8 Color snapshot Figure removed due to copyright restrictions. P(θ,φ,λ) is intensity of light Seen from a single view point At a single time As a function of wavelength

9 A movie Figure removed due to copyright restrictions. P(θ,φ,λ,t) is intensity of light Seen from a single view point Over time As a function of wavelength

10 Holographic movie Figure removed due to copyright restrictions. is intensity of light Seen from ANY viewpoint Over time As a function of wavelength P(θ,φ,λ,t,V X,V Y,V Z )

11 The Plenoptic Function Figure removed due to copyright restrictions. P(θ,φ,λ,t,V X,V Y,V Z ) Can reconstruct every possible view, at every moment, from every position, at every wavelength Contains every photograph, every movie, everything that anyone has ever seen.

12 Sampling Plenoptic Function (top view)

13 Ray Let s s not worry about time and color: 5D 3D position 2D direction P(θ,φ,V X,V Y,V Z ) Courtesy of Rick Szeliski and Michael Cohen. Used with permission. Slide by Rick Szeliski and Michael Cohen

14 Ray No Occluding Objects 4D 2D position 2D direction P(θ,φ,V X,V Y,V Z ) The space of all lines in 3-D 3 D space is 4D. Courtesy of Rick Szeliski and Michael Cohen. Used with permission. Slide by Rick Szeliski and Michael Cohen

15 Lumigraph/Lightfield - Organization 2D position 2D direction θ s Courtesy of Rick Szeliski and Michael Cohen. Used with permission. Slide by Rick Szeliski and Michael Cohen

16 2D position 2D position s u 2 plane parameterization Courtesy of Rick Szeliski and Michael Cohen. Used with permission. Slide by Rick Szeliski and Michael Cohen

17 2D position 2D position t s,t s,t u,v v 2 plane parameterization u,v s u Courtesy of Rick Szeliski and Michael Cohen. Used with permission. Slide by Rick Szeliski and Michael Cohen

18 Light Field = Array of (virtual) Cameras Sub-aperture Virtual Camera = Sub-aperture View Σ Based on original slide by Marc Levoy. Used with permission Marc Levoy

19 Conventional versus plenoptic camera Scene Pixel = (s,t) Virtual Camera = (u,v) uv-plane Pixel = (s,t) st-plane Based on original slide by Marc Levoy. Used with permission Marc Levoy

20 Light Field = Array of (virtual) Cameras Σ Based on original slide by Marc Levoy. Used with permission Marc Levoy

21 Light Field = Array of (virtual) Cameras Sub-aperture Virtual Camera = Sub-aperture View Σ Courtesy of Marc Levoy. Used with permission Marc Levoy

22 Light Field = Array of (virtual) Cameras Sub-aperture Virtual Camera = Sub-aperture View Σ Based on original slide by Marc Levoy. Used with permission Marc Levoy

23 Light Field Inside a Camera Courtesy of Ren Ng. Used with permission.

24 Light Field Inside a Camera Lenslet-based Light Field camera [Adelson and Wang, 1992, Ng et al ] Courtesy of Ren Ng. Used with permission.

array 125μ square-sided microlenses Courtesy of Ren Ng.

25 Stanford Plenoptic Camera [Ng et al 2005] Contax medium format camera Kodak 16-megapixel sensor Adaptive Optics microlens array 125μ square-sided microlenses Courtesy of Ren Ng. Used with permission pixels lenses = pixels per lens

26 Digital Refocusing [Ng et al 2005] Courtesy of Ren Ng. Used with permission.

27 Adaptive Optics A deformable mirror can be used to correct wavefront errors in an astronomical telescope _optics_correct.png

28 Shack Hartmann wavefront sensor (commonly used in Adaptive optics).

Measuring shape of wavefront = Lightfield Capture http://www.

html Courtesy of David Williams Lab @ the Center for Visual

The spots formed on the CCD chip for the eye will be

29 Measuring shape of wavefront = Lightfield Capture Courtesy of David Williams the Center for Visual Science, University of Rochester. Used with permission. The spots formed on the CCD chip for the eye will be displaced because the wavefront will hit each lenslet at an angle rather than straight on.

" Proceedings of CVPR 2004. Courtesy of IEEE.

30 Example using 45 cameras [Vaish CVPR 2004] Vaish, V., et al. "Using Plane + Parallax for Calibrating Dense Camera Arrays." Proceedings of CVPR Courtesy of IEEE. Used with permission IEEE. Courtesy of Marc Levoy. Used with permission Marc Levoy

31 Synthetic aperture videography Image removed due to copyright restrictions.

32 Vaish, V., et al. "Using Plane + Parallax for Calibrating Dense Camera Arrays." Proceedings of CVPR Courtesy of IEEE. Used with permission IEEE.

33 x 1 θ i θ j x 2 θ j θ Visualizing Lightfield (i)position-angle space (ii)phase-space (iii)space- Spatial Frequency (iv)spectrogram θi θ l(x,θ) θj x 2 x 1 x x l(x,θ)

34 x 1 = x 1 + θ i *z x 1 θ i θ j x 2 θ j Shear of Light Field θ θi θ l(x,θ) θj x 2 x 1 x x x 1 x' 1 l(x,θ)

35 θ l(x,θ) x

36 10 0 θ l(x,θ) 10 0 θ θ l(x,θ) x x

37 10 0 θ l(x,θ) 10 0 θ θ l(x,θ) x x

38 Light Field = Array of (virtual) Cameras Sub-aperture Virtual Camera = Sub-aperture View Σ Courtesy of Marc Levoy. Used with permission Marc Levoy

39 Three ways to capture LF inside a camera Shadows using pin-hole array Refraction using lenslet array Heterodyning using masks

42 MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal, Wilson & Veeraraghavan

43 MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal, Wilson & Veeraraghavan

44 Lens Glare Reduction [Raskar, Agrawal, Wilson, Veeraraghavan SIGGRAPH 2008] Glare/Flare due to camera lenses reduces contrast

45 MERL, MIT Media Lab Glare Aware Photography: 4D Ray Sampling for Reducing Glare Raskar, Agrawal, Wilson & Veeraraghavan Reducing Glare Conventional Photo After removing outliers Glare Reduced Image Raskar, R., et al. Glare Aware Photography: 4D Ray Sampling for Reducing Glare Effects of Camera Lenses. Proceedings of SIGGRAPH 2008.

46 Light Field Inside a Camera Lenslet-based Light Field camera [Adelson and Wang, 1992, Ng et al ] Courtesy of Ren Ng. Used with permission.

47 Prototype camera Contax medium format camera Kodak 16-megapixel sensor Adaptive Optics microlens array 125μ square-sided microlenses Courtesy of Ren Ng. Used with permission pixels lenses = pixels per lens

48 Courtesy of Ren Ng. Used with permission.

49 Zooming into the raw photo Courtesy of Ren Ng. Used with permission Marc Levoy

50 Digital Refocusing [Ng et al 2005] Courtesy of Ren Ng. Used with permission. Can we achieve this with a Mask alone?

51 Mask based Light Field Camera Mask Sensor [Veeraraghavan, Raskar, Agrawal, Tumblin, Mohan, Siggraph 2007 ]

52 How to Capture 4D Light Field with 2D Sensor? What should be the pattern of the mask?

53 Lens Copies the Lightfield of Conjugate Plane Object Main Lens 1D Sensor θ -plane x-plane x 0 x 0 θ 0 x θ θ 0 x

54 Object Main Lens 1D Sensor θ -plane x-plane θ l(x,θ) x Line Integral Captured Photo

55 Object Main Lens 1D Sensor θ -plane x-plane θ l(x,θ) x Line Integral Captured Photo

56 Fourier Slice Theorem θ l(x,θ) 2-D FFT f θ L(f x,f θ ) x f x Line Integral Central Slice 1-D FFT Captured Photo FFT of Captured Photo

57 Light Propagation (Defocus Blur) θ l(x,θ) 2-D FFT f θ L(f x,f θ ) x f x Line Integral Central Slice 1-D FFT Captured Photo FFT of Captured Photo

58 In Focus Photo LED

59 Out of Focus Photo: Open Aperture

60 Coded Aperture Camera The aperture of a 100 mm lens is modified Insert a coded mask with chosen binary pattern Rest of the camera is unmodified

61 Out of Focus Photo: Coded Aperture

62 Modeling and Synthesis of Aperture Effects in Cameras Douglas Lanman, Ramesh Raskar, and Gabriel Taubin Computational Aesthetics June,

63 Slides removed due to copyright restrictions. See this paper and associated presentation at

64 Cosine Mask Used Mask Tile 1/f 0

65 Captured 2D Photo Encoding due to Mask

Fields and Coded Aperture Refocusing. Proceedings of SIGGRAPH 2007.

66 Veraraghavan, Raskar, Agrawal, Mohan, Tumblin. Dappled Photography: Mask Enhanced Cameras for Heterodyned Light Fields and Coded Aperture Refocusing. Proceedings of SIGGRAPH D FFT Traditional Camera Photo Magnitude of 2D FFT 2D FFT Heterodyne Camera Photo Magnitude of 2D FFT

67 Extra sensor bandwidth cannot capture extra angular dimension of the light field f θ f θ0 Extra sensor bandwidth f x0 f x Sensor Slice Fourier Light Field Space (Wigner Transform)

68 Sensor Slice captures entire Light Field f θ f θ0 f x0 f x Modulation Function Modulated Light Field

69 Where to place the Mask? Mask Sensor f θ f x Mask Modulation Function

Computing 4D Light Field 2D Sensor Photo, 1800*1800 2D Fourier

Field 4D IFFT Rearrange 2D tiles into 4D planes 200*200*9*9

Dappled Photography: Mask Enhanced Cameras for Heterodyned

70 Computing 4D Light Field 2D Sensor Photo, 1800*1800 2D Fourier Transform, 1800*1800 2D FFT 9*9=81 spectral copies 4D Light Field 4D IFFT Rearrange 2D tiles into 4D planes 200*200*9*9 200*200*9*9 Veraraghavan, Raskar, Agrawal, Mohan, Tumblin. Dappled Photography: Mask Enhanced Cameras for Heterodyned Light Fields and Coded Aperture Refocusing. Proceedings of SIGGRAPH 2007.

71 x 1 = x 1 + θ i *z x 1 θ i θ j x 2 θ j Shear of Light Field θ θi θ l(x,θ) θj x 2 x 1 x x x 1 x' 1 l(x,θ)

72 Light Propagation (Defocus Blur) θ l(x,θ) 2-D FFT f θ L(f x,f θ ) x f x Line Integral Central Slice 1-D FFT Captured Photo FFT of Captured Photo

73 MERL Mask-Enhanced Cameras: Heterodyned Light Fields & Coded Aperture Veeraraghavan, Raskar, Agrawal, Mohan & Tumblin Sensor Sensor Microlens array Mask Plenoptic Camera Heterodyne Camera Samples individual rays Samples coded combination of rays Predefined spectrum for lenses Chromatic abberration Supports any wavelength High alignment precision Reconfigurable f/#, Easier alignment Peripheral pixels wasted pixels No wastage High resolution image for parts of scene in focus Negligible Light Loss 50 % Light Loss due to mask

representations Observable LF Traditional light field Augmented LF WDF

74 Space of LF representations Time-frequency representations Phase space representations Quasi light field Other LF representations Other LF representations Observable LF Traditional light field Augmented LF WDF Rihaczek Distribution Function incoherent Courtesy of Se Baek Oh. Used with permission. coherent

Quasi light fields the utility of light fields, the versatility of Maxwell Other LF representatio ns Rihaczek Distribution Function Other LF representatio ns Observable LF Traditiona

75 Quasi light fields the utility of light fields, the versatility of Maxwell Other LF representatio ns Rihaczek Distribution Function Other LF representatio ns Observable LF Traditiona l light field incoherent Augmented LF coherent WDF We form coherent images by formulating, capturing, and integrating quasi light fields. Courtesy of Se Baek Oh. Used with permission.

76 (i) Observable Light Field move aperture across plane look at directional spread continuous form of plenoptic camera Courtesy of Se Baek Oh. Used with permission. scene aperture position s direction u

$(diffractive) optical element negative radiance LF propagation Interaction at the optical elements Courtesy of Se Baek Oh.$

77 (ii) Augmented Light Field with LF Transformer light field transformer WDF Augmented LF Light Field LF LF LF LF LF propagation (diffractive) optical element negative radiance LF propagation Interaction at the optical elements Courtesy of Se Baek Oh. Used with permission. 7

78 Virtual light projector with real valued (possibly negative radiance) along a ray real projector first null (OPD = λ/2) virtual light projector real projector Courtesy of Se Baek Oh. Used with permission. 7

79 (ii) ALF with LF Transformer Courtesy of Se Baek Oh. Used with permission. 8

localization cross terms (iii) Rihaczek localization complex 3 m

80 Tradeoff between cross-interference terms and localization u y (i) Spectrogram non-negative localization (ii) Wigner localization cross terms (iii) Rihaczek localization complex 3 m u 0 m 0 m y 3 m y 0 m 3 m 0 m y 3 m Courtesy of Se Baek Oh. Used with permission.

81 Property of the Representation Constant along rays Non-negativity negativity Coherence Wavelength Interference Cross term Traditional LF always constant always positive only incoherent zero no Observable LF nearly constant always positive any coherence state any yes Augmented LF only in the paraxial region positive and negative any any yes WDF only in the paraxial region positive and negative any any yes Rihaczek DF no; linear drift complex any any reduced Courtesy of Se Baek Oh. Used with permission.

82 Benefits & Limitations of the Representation Ability to propagate Modeling wave optics Simplicity of computation Adaptability to current pipe line Near Field Far Field Traditional LF x-shear no very simple high no yes Observable LF not x-shearx yes modest low yes yes Augmented LF x-shear yes modest high no yes WDF x-shear yes modest low yes yes Rihaczek DF x-shear yes better than WDF, not as simple as LF low no yes Courtesy of Se Baek Oh. Used with permission.

83 Motivation What is the difference between a hologram and a lenticular screen? How they capture phase of a wavefront for telescope applications? What is wavefront coding lens for extended depth of field imaging?

84 Acknowledgements Dartmouth Marcus Testorf, MIT Ankit Mohan, Ahmed Kirmani, Jaewon Kim George Barbastathis Stanford Marc Levoy, Ren Ng, Andrew Adams Adobe Todor Georgiev, MERL Ashok Veeraraghavan, Amit Agrawal

85 MIT Media Lab Light Fields Camera Culture Ramesh Raskar MIT Media Lab CameraCulture. info/

86 MIT OpenCourseWare MAS.531 Computational Camera and Photography Fall 2009 For information about citing these materials or our Terms of Use, visit:

Coded Computational Imaging: Light Fields and Applications

Coded Computational Imaging: Light Fields and Applications Ankit Mohan MIT Media Lab Coded Computational Imaging Agrawal, Veeraraghavan, Narasimhan & Mohan Schedule Introduction Assorted Pixels Coding