To Denoise or Deblur: Parameter Optimization for Imaging Systems Kaushik Mitra, Oliver Cossairt and Ashok Veeraraghavan 1 ECE, Rice University 2 EECS, Northwestern University 3/3/2014 1
Capture moving object Short exposure Medium exposure Large exposure Increasing exposure time Noise decreases but motion blur increases
Recovered image Captured image Denoise or Deblur? Short exposure Large exposure Denoise Deblur
Effect of illumination Low illumination Denoise Deblur Strong illumination Denoise Deblur Find optimal exposure as a function of illumination
Depth of field Small aperture Medium aperture Large aperture Increasing aperture Noise decreases but defocus blur increases Find optimal aperture as a function of illumination 3/3/2014 5
A framework for analysis of computational imaging (CI) systems K. Mitra, O. Cossairt and A. Veeraraghavan, A framework for analysis of computational imaging systems and its Practical Implications, under review at IEEE PAMI, copy available in arxiv.
Computational imaging (CI) model Scene Image Computational camera Multiplexed image Multiplexed image Multiplexing matrix (Read+photon) Noise Image prior P(x)
Prior Work: Analysis of CI systems 1. Analysis under read noise without prior 2. Analysis under affine noise without prior Harwitt et al. 1979 Prior Analysis Ignores Ratner et al. 2007, Wuttig 2007, Hasinoff et al. 2008, Ihrke et al. 2010, Cossairt et al. 2011 the Impact of Image Priors Completely 3. Relates performance to practical considerations such as illumination, sensor characteristics, etc. Cossairt et al. 2012
Large exposure Short exposure Effect of image priors Captured image Recovery without Image prior Recovery with image prior
Our analysis framework Scene Image Computational camera Multiplexed image Multiplexed image Multiplexing matrix (Read+photon) Noise Image prior P(x) Our analysis takes into account: Image prior, P(x) Multiplexing matrix, H Noise characteristics
Our analysis framework: GMM as signal prior Advantages of GMM 1.Universal approximation property 3. State-of-the-art results Image processing Yu et al. 2010 LF processing Sorenson et al., 1971 2. Analytically tractable A special case is Gaussian prior, whose MMSE can be computed analytically Mitra et al. 2012
Our analysis framework: Linear system Multiplexed image Multiplexing matrix Noise Motion blur Defocus blur Single pixel camera [Raskar 06] [Levin 08] [Cho 10] Light Field Capture [Hausler 72] [Nagahara 08] [Dowski, Cathey 96] [Levin et al. 07] [Zhou, Nayar 08] Reflectance [Wakin et al., 2006] High speed video [Lanman 08] [Veeraraghavan 07] [Liang 08] [Schechner 03] [Ratner 07] [Ratner 08] [Hitomi et al. 2011][Veera et al., 2011]
Our analysis framework: Affine noise model Noise Variance at i th Pixel: photon noise aperture, lighting, pixel size read noise electronics, ADC s, quantization J i : i th pixel intensity Photon noise signal dependent Read noise signal independent Noise PDF: Photon noise modeled as Gaussian (good approx. if #photons > 10) Slide courtesy Oliver Cossairt
MMSE as a performance metric Mean Squared Error (MSE) of an estimator is defined as: MMSE estimator: Defined as the estimator that achieves the minimum MSE Performance measure: SNR gain w.r.t impulse imaging : Impulse imaging system: Captures image directly -> H is identity Motion capture -> small exposure Depth of field -> small aperture
Practical system performance depends on 1. Illumination condition 2. Scene reflectivity 3. Camera parameters F/#, Exposure time t, quantum efficiency q, pixel size p Average signal-level is given by: Average Signal (e - ) Illumination (lux) Reflectivity Aperture Exposure Time (s) Quantum Efficiency Pixel Size (m) H depends only on camera parameters
Optimal exposure for capturing motion
Capture moving object Short exposure Medium exposure Large exposure Increasing exposure time Noise decreases but motion blur increases Find optimal exposure as a function of illumination
Modeling using CI framework CI framework : Captured image Multiplexing matrix Image prior P(x) (Read+photon) Noise Short exposure Medium exposure Large exposure : Identity matrix Impulse imaging system : Toeplitz matrix with medium PSF : Toeplitz matrix with large PSF Image prior P(x) is common, GMM Photon noise depends on light throughput (illumination and exposure)
Decoded image Captured image SNR gain (in db) Optimal PSF, Low illumination 1 lux Analytic SNR gain Vs. Blur size Exposure: t impulse =6 ms t opt = 138 ms Impulse image PSF length Other camera parameters Aperture: F/11 Pixel size: 5 µm Scene reflec: 0.5 Quantum eff: 0.5 PSF length 1 PSF length 5 PSF length 11 PSF length 17 PSF length 23 SNR = 5.9 db SNR = 9.1 db SNR = 12.2 db SNR = 13.5 db SNR=13.9 db
Decoded image Captured image SNR gain (in db) Optimal PSF, medium illumination 10 lux Analytic SNR gain Vs. Blur PSF t impulse =6 ms t opt = 30 ms Impulse image PSF size PSF length 1 PSF length 5 PSF length 11 PSF length 17 PSF length 23 SNR = 12.3 db SNR = 18.2 db SNR = 18.4 db SNR = 17.7 db SNR = 17.2 db
Decoded image Captured image SNR gain (in db) Optimal PSF, strong illumination 1000 lux Analytic SNR gain Vs. Blur PSF t impulse =6 ms t opt = 6 ms Impulse image PSF size PSF length 1 PSF length 5 PSF length 11 PSF length 17 PSF length 23 SNR = 28.4 db SNR = 25.9 db SNR = 23.6 db SNR = 22.0 db SNR = 20.1 db
Optimal PSF size Optimal PSF vs. illumination Optimal PSF at different light levels Illuminance (lux) For illumination > 150 lux, impulse setting is optimal Look up table for optimal exposure 3/3/2014 22
Optimal aperture for depth of field
Depth of field Increasing aperture Noise decreases but defocus blur increases Small aperture Medium aperture Large aperture Find optimal aperture as a function of illumination 3/3/2014 24
Modeling using CI framework CI framework : Captured image Multiplexing matrix Image prior P(x) (Read+photon) Noise Small aperture Medium aperture Large aperture : Identity matrix : Block Toeplitz matrix Impulse imaging system with medium PSF : Block Toeplitz matrix With large PSF Common image prior P(x), GMM Photon noise depends on light throughput Aperture and illumination 3/3/2014 25
Decoded image Captured image SNR gain (in db) Optimal PSF, low illumination 1 lux Analytic SNR gain Vs. Blur size Aperture: Impulse: f/11 Optimal: f/1.2 Impulse image PSF size Other camera parameters Exposure: 6 ms Pixel size: 5 µm Scene reflec: 0.5 Quantum eff: 0.5 PSF size 1 1 PSF size 3 3 PSF size 5 5 PSF size 7 7 PSF size 9 9 SNR=10.3 db SNR=12.0 db SNR=15.4 db SNR=17.0 db SNR=17.3 db
Decoded image Captured image SNR gain (in db) Optimal PSF, medium illumination 10 lux Analytic SNR gain Vs. Blur PSF Aperture: Impulse: f/11 Optimal: f/2.2 Impulse image PSF size PSF size 1 1 PSF size 3 3 PSF size 5 5 PSF size 7 7 PSF size 9 9 SNR=12.3 db SNR=19.5 db SNR=20 db SNR=19.2 db SNR=18.5 db
Decoded image Captured image SNR gain (in db) Optimal PSF, strong illumination 1000 lux Analytic SNR gain Vs. Blur PSF Aperture: Impulse: f/11 Optimal: f/11 Impulse image PSF size PSF size 1 1 PSF size 3 3 PSF size 5 5 PSF size 7 7 PSF size 9 9 SNR=26.6 db SNR=26.1 db SNR=23.3 db SNR=19.4 db SNR=17.4 db
Optimal PSF size Optimal PSF vs. illumination Optimal PSF at different light levels Illuminance (lux) For illumination > 400 lux, impulse setting is optimal Look up table for optimal aperture 3/3/2014 29
Optimal PSF size Conclusion Analytic framework for CI systems Project webpage: http://users.eecs.northwestern.edu/~ollie/frameworkci/ Optimal camera parameter estimation Illuminance (lux)