To Denoise or Deblur: Parameter Optimization for Imaging Systems

Size: px

Start display at page:

Download "To Denoise or Deblur: Parameter Optimization for Imaging Systems"

Gervais Williamson
6 years ago
Views:

1 To Denoise or Deblur: Parameter Optimization for Imaging Systems Kaushik Mitra, Oliver Cossairt and Ashok Veeraraghavan 1 ECE, Rice University 2 EECS, Northwestern University 3/3/2014 1

2 Capture moving object Short exposure Medium exposure Large exposure Increasing exposure time Noise decreases but motion blur increases

3 Recovered image Captured image Denoise or Deblur? Short exposure Large exposure Denoise Deblur

4 Effect of illumination Low illumination Denoise Deblur Strong illumination Denoise Deblur Find optimal exposure as a function of illumination

5 Depth of field Small aperture Medium aperture Large aperture Increasing aperture Noise decreases but defocus blur increases Find optimal aperture as a function of illumination 3/3/2014 5

6 A framework for analysis of computational imaging (CI) systems K. Mitra, O. Cossairt and A. Veeraraghavan, A framework for analysis of computational imaging systems and its Practical Implications, under review at IEEE PAMI, copy available in arxiv.

7 Computational imaging (CI) model Scene Image Computational camera Multiplexed image Multiplexed image Multiplexing matrix (Read+photon) Noise Image prior P(x)

2007, Wuttig 2007, Hasinoff et al. 2008, Ihrke et al.

8 Prior Work: Analysis of CI systems 1. Analysis under read noise without prior 2. Analysis under affine noise without prior Harwitt et al Prior Analysis Ignores Ratner et al. 2007, Wuttig 2007, Hasinoff et al. 2008, Ihrke et al. 2010, Cossairt et al the Impact of Image Priors Completely 3. Relates performance to practical considerations such as illumination, sensor characteristics, etc. Cossairt et al. 2012

9 Large exposure Short exposure Effect of image priors Captured image Recovery without Image prior Recovery with image prior

10 Our analysis framework Scene Image Computational camera Multiplexed image Multiplexed image Multiplexing matrix (Read+photon) Noise Image prior P(x) Our analysis takes into account: Image prior, P(x) Multiplexing matrix, H Noise characteristics

State-of-the-art results Image processing Yu et al.

11 Our analysis framework: GMM as signal prior Advantages of GMM 1.Universal approximation property 3. State-of-the-art results Image processing Yu et al LF processing Sorenson et al., Analytically tractable A special case is Gaussian prior, whose MMSE can be computed analytically Mitra et al. 2012

12 Our analysis framework: Linear system Multiplexed image Multiplexing matrix Noise Motion blur Defocus blur Single pixel camera [Raskar 06] [Levin 08] [Cho 10] Light Field Capture [Hausler 72] [Nagahara 08] [Dowski, Cathey 96] [Levin et al. 07] [Zhou, Nayar 08] Reflectance [Wakin et al., 2006] High speed video [Lanman 08] [Veeraraghavan 07] [Liang 08] [Schechner 03] [Ratner 07] [Ratner 08] [Hitomi et al. 2011][Veera et al., 2011]

Our analysis framework: Affine noise model Noise Variance

read noise electronics, ADC s, quantization J i : i th

signal independent Noise PDF: Photon noise modeled as

13 Our analysis framework: Affine noise model Noise Variance at i th Pixel: photon noise aperture, lighting, pixel size read noise electronics, ADC s, quantization J i : i th pixel intensity Photon noise signal dependent Read noise signal independent Noise PDF: Photon noise modeled as Gaussian (good approx. if #photons > 10) Slide courtesy Oliver Cossairt

MMSE as a performance metric Mean Squared Error (MSE) of an estimator is defined as: MMSE estimator: Defined as the estimator that achieves the minimum MSE Performance

14 MMSE as a performance metric Mean Squared Error (MSE) of an estimator is defined as: MMSE estimator: Defined as the estimator that achieves the minimum MSE Performance measure: SNR gain w.r.t impulse imaging : Impulse imaging system: Captures image directly -> H is identity Motion capture -> small exposure Depth of field -> small aperture

Practical system performance depends on 1.

Camera parameters F/#, Exposure time t, quantum efficiency

Signal (e - ) Illumination (lux) Reflectivity Aperture

15 Practical system performance depends on 1. Illumination condition 2. Scene reflectivity 3. Camera parameters F/#, Exposure time t, quantum efficiency q, pixel size p Average signal-level is given by: Average Signal (e - ) Illumination (lux) Reflectivity Aperture Exposure Time (s) Quantum Efficiency Pixel Size (m) H depends only on camera parameters

16 Optimal exposure for capturing motion

17 Capture moving object Short exposure Medium exposure Large exposure Increasing exposure time Noise decreases but motion blur increases Find optimal exposure as a function of illumination

18 Modeling using CI framework CI framework : Captured image Multiplexing matrix Image prior P(x) (Read+photon) Noise Short exposure Medium exposure Large exposure : Identity matrix Impulse imaging system : Toeplitz matrix with medium PSF : Toeplitz matrix with large PSF Image prior P(x) is common, GMM Photon noise depends on light throughput (illumination and exposure)

Decoded image Captured image SNR gain (in db) Optimal PSF, Low

Blur size Exposure: t impulse =6 ms t opt = 138 ms Impulse image

19 Decoded image Captured image SNR gain (in db) Optimal PSF, Low illumination 1 lux Analytic SNR gain Vs. Blur size Exposure: t impulse =6 ms t opt = 138 ms Impulse image PSF length Other camera parameters Aperture: F/11 Pixel size: 5 µm Scene reflec: 0.5 Quantum eff: 0.5 PSF length 1 PSF length 5 PSF length 11 PSF length 17 PSF length 23 SNR = 5.9 db SNR = 9.1 db SNR = 12.2 db SNR = 13.5 db SNR=13.9 db

20 Decoded image Captured image SNR gain (in db) Optimal PSF, medium illumination 10 lux Analytic SNR gain Vs. Blur PSF t impulse =6 ms t opt = 30 ms Impulse image PSF size PSF length 1 PSF length 5 PSF length 11 PSF length 17 PSF length 23 SNR = 12.3 db SNR = 18.2 db SNR = 18.4 db SNR = 17.7 db SNR = 17.2 db

21 Decoded image Captured image SNR gain (in db) Optimal PSF, strong illumination 1000 lux Analytic SNR gain Vs. Blur PSF t impulse =6 ms t opt = 6 ms Impulse image PSF size PSF length 1 PSF length 5 PSF length 11 PSF length 17 PSF length 23 SNR = 28.4 db SNR = 25.9 db SNR = 23.6 db SNR = 22.0 db SNR = 20.1 db

22 Optimal PSF size Optimal PSF vs. illumination Optimal PSF at different light levels Illuminance (lux) For illumination > 150 lux, impulse setting is optimal Look up table for optimal exposure 3/3/

23 Optimal aperture for depth of field

24 Depth of field Increasing aperture Noise decreases but defocus blur increases Small aperture Medium aperture Large aperture Find optimal aperture as a function of illumination 3/3/

25 Modeling using CI framework CI framework : Captured image Multiplexing matrix Image prior P(x) (Read+photon) Noise Small aperture Medium aperture Large aperture : Identity matrix : Block Toeplitz matrix Impulse imaging system with medium PSF : Block Toeplitz matrix With large PSF Common image prior P(x), GMM Photon noise depends on light throughput Aperture and illumination 3/3/

Decoded image Captured image SNR gain (in db)

2 Impulse image PSF size Other camera parameters

26 Decoded image Captured image SNR gain (in db) Optimal PSF, low illumination 1 lux Analytic SNR gain Vs. Blur size Aperture: Impulse: f/11 Optimal: f/1.2 Impulse image PSF size Other camera parameters Exposure: 6 ms Pixel size: 5 µm Scene reflec: 0.5 Quantum eff: 0.5 PSF size 1 1 PSF size 3 3 PSF size 5 5 PSF size 7 7 PSF size 9 9 SNR=10.3 db SNR=12.0 db SNR=15.4 db SNR=17.0 db SNR=17.3 db

27 Decoded image Captured image SNR gain (in db) Optimal PSF, medium illumination 10 lux Analytic SNR gain Vs. Blur PSF Aperture: Impulse: f/11 Optimal: f/2.2 Impulse image PSF size PSF size 1 1 PSF size 3 3 PSF size 5 5 PSF size 7 7 PSF size 9 9 SNR=12.3 db SNR=19.5 db SNR=20 db SNR=19.2 db SNR=18.5 db

28 Decoded image Captured image SNR gain (in db) Optimal PSF, strong illumination 1000 lux Analytic SNR gain Vs. Blur PSF Aperture: Impulse: f/11 Optimal: f/11 Impulse image PSF size PSF size 1 1 PSF size 3 3 PSF size 5 5 PSF size 7 7 PSF size 9 9 SNR=26.6 db SNR=26.1 db SNR=23.3 db SNR=19.4 db SNR=17.4 db

29 Optimal PSF size Optimal PSF vs. illumination Optimal PSF at different light levels Illuminance (lux) For illumination > 400 lux, impulse setting is optimal Look up table for optimal aperture 3/3/

Optimal PSF size Conclusion Analytic framework for CI systems Project webpage: http://users.

30 Optimal PSF size Conclusion Analytic framework for CI systems Project webpage: Optimal camera parameter estimation Illuminance (lux)

A Framework for Analysis of Computational Imaging Systems

A Framework for Analysis of Computational Imaging Systems Kaushik Mitra, Oliver Cossairt, Ashok Veeraghavan Rice University Northwestern University Computational imaging CI systems that adds new functionality