Transfer Efficiency and Depth Invariance in Computational Cameras

Transfer Efficiency and Depth Invariance in Computational Cameras Jongmin Baek Stanford University IEEE International Conference on Computational Photography 2010 Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 1 / 30

The Problem: Defocus Blur f/1.4 Jongmin Baek (Stanford University) f/11.0 Transfer Efficiency and Depth Invariance March 29, 2010 2 / 30

The Cause: Image Formation Image Object d Sensor Lens Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 3 / 30

The Cause: Image Formation Image Object d Sensor Lens Response at the sensor plane depends on how all paths interfere. Under paraxial wave optics, imaging process is a convolution. + η = I in PSF Noise I out Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 3 / 30

The Cause: Image Formation Image Object d Sensor Lens Response at the sensor plane depends on how all paths interfere. In the frequency domain, it is a multiplication. + η = frequency frequency frequency Noise F(I in ) OTF F(I out ) Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 3 / 30

The Cause: Image Formation Image Object d ǫ Sensor Lens Response at the sensor plane depends on how all paths interfere. Under paraxial wave optics, imaging process is a convolution. + η = I in PSF Noise I out Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 3 / 30

A Solution: Computational Imaging Engineer the optics to preserve scene information at all depths. Reconstruct the scene later. Scene Optics Scene Noise Digital Processing Scene Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 4 / 30

A Solution: Aperture Modulation The aperture controls how the paths interfere. Engineer the aperture to control defocus. Top: Levin et al. and Veeraraghavan et al., SIGGRAPH 2007; Bottom: Nagahara et al., ECCV 2008; Levin et al., SIGGRAPH 2009. Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 5 / 30

Solution #1: Amplitude Modulation Image Object d ǫ Sensor Lens e.g. coded aperture (Levin et al., 2007) lattice focal lens (Levin et al., 2009) Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 6 / 30

Solution #2: Phase Modulation Image Object d ǫ Sensor Lens e.g. wavefront coding (Dowski et al., 1995), lattice focal lens (Levin et al., 2009) Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 7 / 30

Solution #3: Time Modulation Image Object d ǫ Sensor Lens e.g. focus sweep (Nagahara et al., 2008) lattice focal lens (Levin et al., 2009) Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 8 / 30

The Metaproblem Many spatiotemporally modulated apertures exist for extending the depth of field. How do we analyze them? Can we make general statements about computational cameras based on such apertures? Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 9 / 30

Outline Space of Optical Transfer Functions Space of Optical Transfer Functions Optimizing Computational Cameras Evaluating Computational Cameras Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 10 / 30

Space of Optical Transfer Functions Optical Transfer Function: The Tool of the Trade Every spatiotemporally modulated aperture corresponds to one. The OTF is generally thought of as a 2D function in the frequency domain, that also varies with defocus. Focused Slightly defocused More defocused Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 11 / 30

Optical Transfer Function (Cont d) ψ := the defocus parameter in optics: ψ 1 f 1 d i 1 d o. Space of Optical Transfer Functions ψ f x f y Figure: The OTF of a square aperture as a 3D function. Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 12 / 30

OTF Slices Space of Optical Transfer Functions The OTF is a 3D function. Typically, we slice at a particular depth ψ. ψ f x f y Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 13 / 30

OTF Slices Space of Optical Transfer Functions The OTF is a 3D function. Typically, we slice at a particular depth ψ. Instead, fix (f x, f y ). ψ f x f y Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 13 / 30

OTF Slices Space of Optical Transfer Functions The OTF is a 3D function. Typically, we slice at a particular depth ψ. Instead, fix (f x, f y ). Now a function of single variable ψ. ψ f x f y Figure: The OTF slice at (0.3, 0.3) Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 13 / 30

What OTF slices are realisable? Space of Optical Transfer Functions Not all complex 1D functions are realisable. Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 14 / 30

What OTF slices are realisable? Space of Optical Transfer Functions Not all complex 1D functions are realisable. Why do we want to know? We want to make a general statement about all spatiotemporally modulated apertures. Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 14 / 30

What OTF slices are realisable? Space of Optical Transfer Functions Not all complex 1D functions are realisable. Why do we want to know? We want to make a general statement about all spatiotemporally modulated apertures. Previously known: OTF(ψ) Diffraction Limit. The L 2 -norm of OTF slices is bounded (Bagheri 2009; Levin 2009). Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 14 / 30

Space of OTF Slices Space of Optical Transfer Functions f x = 0.1 Observation: OTF slices are very structured. f y = 0.1 f y = 0.3 f y = 0.5 f y = 0.7 f y = 0.9 f x = 0.3 f x = 0.5 f x = 0.7 f x = 0.9 Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 15 / 30

Space of OTF Slices Space of Optical Transfer Functions Observation: OTF slices are very structured. We show all OTF slices are complex 1D functions bounded by a trapezoid in magnitude, and vice versa. OTF(ψ)e 2πiγψ dγ π f x f y max {0, min(k 1, K 2 ) 2πγ K 1 K 2 } See proofs in the paper. Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 15 / 30

Space of OTF Slices Space of Optical Transfer Functions Observation: OTF slices are very structured. We show all OTF slices are complex 1D functions bounded by a trapezoid in magnitude, and vice versa. OTF(ψ)e 2πiγψ dγ π f x f y max {0, min(k 1, K 2 ) 2πγ K 1 K 2 } See proofs in the paper. Caveat: set of all apertures set of all OTF slices Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 15 / 30

Space of Optical Transfer Functions Utilizing the Dual Structure: Application Suppose we want a box -shaped OTF slice: Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 16 / 30

Space of Optical Transfer Functions Utilizing the Dual Structure: Application Suppose we want a box -shaped OTF slice: The dual OTF slice is a sinc and must fit under the trapezoid. Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 16 / 30

Space of Optical Transfer Functions Utilizing the Dual Structure: Application Suppose we want a box -shaped OTF slice: The dual OTF slice is a sinc and must fit under the trapezoid. Tells us how tall and wide the box can be. Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 16 / 30

Space of Optical Transfer Functions Utilizing the Dual Structure: Application Suppose we want a box -shaped OTF slice: The dual OTF slice is a sinc and must fit under the trapezoid. Tells us how tall and wide the box can be. Note that a lot of energy under the trapezoid is wasted. Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 16 / 30

Outline Optimizing Computational Cameras Space of Optical Transfer Functions Optimizing Computational Cameras Evaluating Computational Cameras Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 17 / 30

What is a good OTF/MTF? Optimizing Computational Cameras Scene Optics Scene Noise Digital Processing Scene Sources of reconstruction error: 1 OTF/MTF is low (ill-conditioned inversion.) 2 OTF/MTF is depth-dependent (inversion with wrong kernel.) Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 18 / 30

What is a good OTF/MTF? Optimizing Computational Cameras Sources of reconstruction error: 1 OTF/MTF is low (ill-conditioned inversion.) 2 OTF/MTF is depth-dependent (inversion with wrong kernel.) Design Cubic phase plate Focus sweep Lattice focal lens Goal Depth-invariant MTF Depth-invariant OTF High transfer efficiency Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 19 / 30

What is a good OTF/MTF? Optimizing Computational Cameras Sources of reconstruction error: 1 OTF/MTF is low (ill-conditioned inversion.) 2 OTF/MTF is depth-dependent (inversion with wrong kernel.) Design Cubic phase plate Focus sweep Lattice focal lens Goal Depth-invariant MTF Depth-invariant OTF High transfer efficiency Want either depth invariance or transfer efficiency Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 19 / 30

What is a good OTF/MTF? Optimizing Computational Cameras Sources of reconstruction error: 1 OTF/MTF is low (ill-conditioned inversion.) 2 OTF/MTF is depth-dependent (inversion with wrong kernel.) Design Cubic phase plate Focus sweep Lattice focal lens Goal Depth-invariant MTF Depth-invariant OTF High transfer efficiency Want either depth invariance or transfer efficiency Why not aim for both? Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 19 / 30

Optimizing Computational Cameras What is a good OTF/MTF? (Cont d) Depth Variance :-( :-( :-) 2D scatter plot of the scores: :-( depth depth depth Transfer Efficiency depth Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 20 / 30

Optimizing Computational Cameras What is a good OTF/MTF? (Cont d) Depth Variance :-( :-( :-) Minimize depth variance: :-( depth depth Closed aperture depth Transfer Efficiency depth Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 20 / 30

Optimizing Computational Cameras What is a good OTF/MTF? (Cont d) Depth Variance :-( :-( :-) Maximize transfer efficiency: :-( depth depth Regular aperture Closed aperture depth Transfer Efficiency depth Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 20 / 30

Best of Both Worlds? Optimizing Computational Cameras We want transfer efficiency and depth invariance. Trivial to get one out of two. Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 21 / 30

Best of Both Worlds? Optimizing Computational Cameras We want transfer efficiency and depth invariance. Trivial to get one out of two. Can we get both simultaneously? Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 21 / 30

Best of Both Worlds? Optimizing Computational Cameras We want transfer efficiency and depth invariance. Trivial to get one out of two. Can we get both simultaneously? No. The two metrics are highly negatively correlated: Can t obtain one without sacrificing the other. See the paper for the scary math. Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 21 / 30

Outline Evaluating Computational Cameras Space of Optical Transfer Functions Optimizing Computational Cameras Evaluating Computational Cameras Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 22 / 30

Depth Variance Evaluating computational Cameras Evaluating Computational Cameras Transfer Efficiency green is strictly better than red. Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 23 / 30

Depth Variance Evaluating computational Cameras Evaluating Computational Cameras Transfer Efficiency how do we compare yellow and green? Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 23 / 30

Evaluating Computational Cameras Evaluating computational Cameras (Cont d) Observation: Most designs do not represent a single fixed camera. Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 24 / 30

Evaluating Computational Cameras Evaluating computational Cameras (Cont d) Observation: Most designs do not represent a single fixed camera. Each represents a parametrized family of cameras. We should compare families of cameras. Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 24 / 30

OTF Depth Variance OTF Tradeoff Evaluating Computational Cameras Transfer Efficiency Transfer Efficiency Square aperture Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 25 / 30

OTF Depth Variance OTF Tradeoff Evaluating Computational Cameras Transfer Efficiency Transfer Efficiency Square aperture, parametrized by f-number Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 25 / 30

OTF Depth Variance OTF Tradeoff Evaluating Computational Cameras Transfer Efficiency Transfer Efficiency Square aperture, parametrized by f-number Cubic phase plate (Dowski 1995), parametrized by thickness Focus sweep (Nagahara 2008), parametrized by sweep distance Lattice focal lens (Levin 2009), parameterized by lenslets Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 25 / 30

MTF Tradeoff Evaluating Computational Cameras Square aperture, parametrized by f-number Cubic phase plate (Dowski 1995), parametrized by thickness Focus sweep (Nagahara 2008), parametrized by sweep distance Lattice focal lens (Levin 2009), parameterized by lenslets Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 26 / 30

MTF Depth Variance MTF Tradeoff Evaluating Computational Cameras Transfer Efficiency Transfer Efficiency Square aperture, parametrized by f-number Cubic phase plate (Dowski 1995), parametrized by thickness Focus sweep (Nagahara 2008), parametrized by sweep distance Lattice focal lens (Levin 2009), parameterized by lenslets Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 26 / 30

Observations Evaluating Computational Cameras 1 All the following families outperform stopping down. Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 27 / 30

Observations Evaluating Computational Cameras 1 All the following families outperform stopping down. 2 Focus sweep exhibits the best tradeoff in obtaining OTF invariance. Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 27 / 30

Observations Evaluating Computational Cameras 1 All the following families outperform stopping down. 2 Focus sweep exhibits the best tradeoff in obtaining OTF invariance. 3 Cubic phase plate exhibits the best tradeoff in obtaining MTF invariance. Only on certain frequencies, however. Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 27 / 30

Observations Evaluating Computational Cameras 1 All the following families outperform stopping down. 2 Focus sweep exhibits the best tradeoff in obtaining OTF invariance. 3 Cubic phase plate exhibits the best tradeoff in obtaining MTF invariance. Only on certain frequencies, however. 4 Lattice focal lens falls in bewteen a regular lens and CPP. Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 27 / 30

Observations Evaluating Computational Cameras 1 All the following families outperform stopping down. 2 Focus sweep exhibits the best tradeoff in obtaining OTF invariance. 3 Cubic phase plate exhibits the best tradeoff in obtaining MTF invariance. Only on certain frequencies, however. 4 Lattice focal lens falls in bewteen a regular lens and CPP. See the paper for discussions. Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 27 / 30

Beyond Existing Designs Evaluating Computational Cameras Can we do better than focus sweep and cubic phase plate? Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 28 / 30

Beyond Existing Designs Evaluating Computational Cameras Can we do better than focus sweep and cubic phase plate? Not by much. Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 28 / 30

Beyond Existing Designs Evaluating Computational Cameras Can we do better than focus sweep and cubic phase plate? Not by much. We identified the space of all realisable OTF slices: Mathematically calculate an upper bound over this space. We show that... focus sweep is near optimal in trading off transfer efficiency for OTF invariance. for a fixed spatial frequency (f x, f y ), there exists a cubic phase plate near optimal in trading off transfer efficiency for MTF invariance. Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 28 / 30

Take-aways Conclusions 1 OTF slices of an imaging system obey a particular structure. Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 29 / 30

Take-aways Conclusions 1 OTF slices of an imaging system obey a particular structure. 2 There is an inherent tradeoff between transfer efficiency and depth invariance. Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 29 / 30

Take-aways Conclusions 1 OTF slices of an imaging system obey a particular structure. 2 There is an inherent tradeoff between transfer efficiency and depth invariance. 3 Some existing designs are already pretty good. Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 29 / 30

Future work Conclusions Aggregate bounds over all spatial frequencies Information-theoretic treatment of OTFs Jongmin Baek (Stanford University) Transfer Efficiency and Depth Invariance March 29, 2010 30 / 30