Admin Deblurring & Deconvolution Different types of blur

Size: px

Start display at page:

Download "Admin Deblurring & Deconvolution Different types of blur"

Chester Nash
6 years ago
Views:

1 Admin Assignment 3 due Deblurring & Deconvolution Lecture 10 Last lecture Move to Friday? Projects Come and see me Different types of blur Camera shake User moving hands Scene motion Objects in the scene moving Defocus blur [NEXT WEEK] Depth of field effects 1

2 Overview Let s take a photo Removing Camera Shake Non-blind Blind Removing Motion Blur Non-blind Blind Blurry result Focus on software approaches Slow-motion replay Slow-motion replay Motion of camera Image formation model: Convolution Blind vs Non-blind = Non-blind Blurry image Input to algorithm Model is approximation Assume static scene Sharp image Desired output Convolution operator Blur kernel Blind 2

right Bot. left Bot. right Person 3 Person 4 What if scene not static?

3 Camera Shake is it a convolution? 8 different people, handholding camera, using 1 second exposure Dots from each corner Person 1 Person 2 Top left Top right Bot. left Bot. right Person 3 Person 4 What if scene not static? Partition the image into regions Overview Removing Camera Shake Non-blind Blind Removing Motion Blur Non-blind Blind Deconvolution is ill posed Deconvolution is ill posed f x = y f x = y? = Solution 1:? = Solution 2:? = Slide from Anat Levin Slide from Anat Levin 3

Convolution- frequency domain representation Idea 1: Natural images prior spectrum spectrum

gradient Spatial convolution frequency multiplication Slide from Anat Levin Natural images have

prior f x y 2 + λ i ρ( x ) Convolution error Derivatives prior i Comparing deconvolution

edu/graphics/codedaperture/ Slide from Anat Levin?

8 ρ ( x) = x spread gradients localizes gradients?

4 Convolution- frequency domain representation Idea 1: Natural images prior spectrum spectrum Sharp Image 0 Frequency 0 Frequency = spectrum 1 st observed image 0 Frequency Image What makes images special? Natural Unnatural 1-D Example Output spectrum has zeros where filter spectrum has zeros gradient Spatial convolution frequency multiplication Slide from Anat Levin Natural images have sparse gradients put a penalty on gradients Slide from Anat Levin x = argmin Deconvolution with prior f x y 2 + λ i ρ( x ) Convolution error Derivatives prior i Comparing deconvolution algorithms (Non blind) deconvolution code available online: Slide from Anat Levin? _ Equal convolution error 2 + Low Input ρ ( x) = x ρ ( x) = x spread gradients localizes gradients? _ 2 + High Richardson-Lucy Gaussian prior Sparse prior Comparing deconvolution algorithms (Non blind) deconvolution code available online: Slide from Anat Levin Input ρ ( x) = x ρ ( x) = x spread gradients localizes gradients Richardson-Lucy Gaussian prior Sparse prior 4

Application: Hubble Space Telescope Launched with flawed mirror Initially used

Non-Blind Deconvolution Matlab Demo http://groups.csail.mit.

html Overview Removing Camera Shake Non-blind Blind Removing Motion Blur

T. Freeman Removing Camera Shake from a Single Photograph Original Our algorithm

5 Application: Hubble Space Telescope Launched with flawed mirror Initially used deconvolution to correct images before corrective optics installed Image of star Non-Blind Deconvolution Matlab Demo daperture/deconvolutioncode.html Overview Removing Camera Shake Non-blind Blind Removing Motion Blur Non-blind Blind Overview Joint work with B. Singh, A. Hertzmann, S.T. Roweis & W.T. Freeman Removing Camera Shake from a Single Photograph Original Our algorithm Rob Fergus, Barun Singh, Aaron Hertzmann, Sam T. Roweis and William T. Freeman Massachusetts Institute of Technology and University of Toronto 5

Close-up Image formation process Original Naïve sharpening Our algorithm = Blurry image Sharp image Blur kernel Input to algorithm Model is approximation Assume static scene Desired output

, Restoration of Photographs Blurred by Image Motion, Bell System Tech., Vol. 46, No. 10, pp. 2353-2362, 3 2362 1967.

shake Assumed forms of blur kernels Image constraints are frequency-domain power-laws Existing work on image deblurring Why is this hard?

6 Close-up Image formation process Original Naïve sharpening Our algorithm = Blurry image Sharp image Blur kernel Input to algorithm Model is approximation Assume static scene Desired output Convolution operator Existing work on image deblurring Old problem: Trott, T., The Effect of Motion of Resolution, Photogrammetric Engineering, Vol. 26, pp , Slepian, D., Restoration of Photographs Blurred by Image Motion, Bell System Tech., Vol. 46, No. 10, pp , Existing work on image deblurring Software algorithms for natural images Many require multiple images Mainly Fourier and/or Wavelet based Strong assumptions about blur not true for camera shake Assumed forms of blur kernels Image constraints are frequency-domain power-laws Existing work on image deblurring Why is this hard? Hardware approaches Image stabilizers Dual cameras Coded shutter Simple analogy: 11 is the product of two numbers. What are they? Ben-Ezra & Nayar CVPR 2004 Raskar et al. SIGGRAPH 2006 Our approach can be combined with these hardware methods No unique solution: 11 = 1 x = 2 x = 3 x etc.. Need more information!!!! 6

Multiple possible solutions Sharp image = Natural image statistics Characteristic distribution with heavy tails Histogram of image gradients Blurry

pixels Log # pixels Use parametric model of sharp image statistics Uses of natural image statistics Denoising [Portilla et al.

, ICCV 2003] Reflections [Levin and Weiss, ECCV 2004] Video matting [Apostoloff & Fitzgibbon, CVPR 2005] Three sources of information 1.

7 Multiple possible solutions Sharp image = Natural image statistics Characteristic distribution with heavy tails Histogram of image gradients Blurry image = Log # pixels = Blurry images have different statistics Parametric distribution Histogram of image gradients Histogram of image gradients Log # pixels Log # pixels Use parametric model of sharp image statistics Uses of natural image statistics Denoising [Portilla et al. 2003, Roth and Black, CVPR 2005] Superresolution [Tappen et al., ICCV 2003] Intrinsic images [Weiss, ICCV 2001] Inpainting i [Levin et al., ICCV 2003] Reflections [Levin and Weiss, ECCV 2004] Video matting [Apostoloff & Fitzgibbon, CVPR 2005] Three sources of information 1. Reconstruction constraint: Estimated sharp image Estimated blur kernel = Input blurry image 2. Image prior: 3. Blur prior: Corruption process assumed known Distribution of gradients Positive & Sparse 7

8 Three sources of information Three sources of information y = observed image b = blur kernel x = sharp image y = observed image b = blur kernel x = sharp image p( b; xjy) Posterior Three sources of information 1. Likelihood p( yjb; x ) y = observed image b = blur x = sharp image y = observed image b = blur kernel x = sharp image p( b; xjy) = k p(yjb;x) p(x) p(b) Posterior 1. Likelihood (Reconstruction constraint) 2. Image prior 3. Blur prior Reconstruction constraint: p( yjb; x) = Q i N(y i jx i - b;¾ 2 ) i - pixel index / Q i e (x i -b y i ) 2 2¾ 2 2. Image prior p( x) y = observed image b = blur x = sharp image 3. Blur prior p( b) y = observed image b = blur x = sharp image p( x) = Q P Cc= i 1 ¼ c N(f(x i )j0;s 2 c ) p( b) = Q j P Dd= 1 ¼ d E( b j j d) Mixture of Gaussians fit to empirical distribution of image gradients i - pixel index c - mixture component index f - derivative filter Log # pixels Mixture of Exponentials Positive & sparse No connectivity constraint j - blur kernel element d - mixture component index p(b) Most elements near zero A few can be large b 8

The obvious thing to do p( b; xjy) = k p(yjb;x) p(x) p(b) Posterior 1. Likelihood (Reconstruction constraint) 2.

Blur prior Variational Bayesian approach Keeps track of uncertainty in estimates of image and blur by using a

is Maximum a-posteriori (MAP) No success!

Independent Component Analysis Miskin and Mackay, 2000 Binary images Priors on intensities Overview of algorithm 1.

Kernel estimation - Multi-scale approach 3.

9 The obvious thing to do p( b; xjy) = k p(yjb;x) p(x) p(b) Posterior 1. Likelihood (Reconstruction constraint) 2. Image prior 3. Blur prior Variational Bayesian approach Keeps track of uncertainty in estimates of image and blur by using a distribution instead of a single estimate Combine 3 terms into an objective function Run conjugate gradient descent This is Maximum a-posteriori (MAP) No success! Score Optimization surface for a single variable Maximum a-posteriori (MAP) Variational Bayes Pixel intensity Variational Independent Component Analysis Miskin and Mackay, 2000 Binary images Priors on intensities Overview of algorithm 1. Pre-processing Input image Small, synthetic blurs Not applicable to natural images 2. Kernel estimation - Multi-scale approach 3. Image reconstruction - Standard non-blind deconvolution routine Digital image formation process RAW values Gamma correction Remapped values Preprocessing Input image Blur process applied here Convert to grayscale Remove gamma correction User selects patch from image Bayesian inference too slow to run on whole image Infer kernel from this patch P. Debevec & J. Malik, Recovering High Dynamic Range Radiance Maps from Photographs, SIGGRAPH 97 9

Initialization Input image Inferring the kernel: multiscale method Input image Convert to grayscale Remove gamma correction Convert to grayscale Remove gamma correction User selects patch from image

10 Initialization Input image Inferring the kernel: multiscale method Input image Convert to grayscale Remove gamma correction Convert to grayscale Remove gamma correction User selects patch from image Loop over scales User selects patch from image Initialize 3x3 blur kernel Upsample estimates Variational Bayes Initialize 3x3 blur kernel Use multi-scale approach to avoid local minima: Blurry patch Initial image estimate Initial blur kernel Image Reconstruction Input image Full resolution blur estimate Convert to grayscale Loop over scales Upsample estimates Remove gamma correction Variational Bayes User selects patch from image Initialize 3x3 blur kernel Synthetic experiments Non-blind deconvolution (Richardson-Lucy) Deblurred image Synthetic example Synthetic blurry image Sharp image Artificial blur trajectory 10

Image before Inference initial scale Image after Image before Inference

after Image before Inference scale 3 Image after Image before Inference

after Image before Inference scale 5 Image after Image before Inference

11 Image before Inference initial scale Image after Image before Inference scale 2 Image after Kernel before Kernel after Kernel before Kernel after Image before Inference scale 3 Image after Image before Inference scale 4 Image after Kernel before Kernel after Kernel before Kernel after Image before Inference scale 5 Image after Image before Inference scale 6 Image after Kernel before Kernel after Kernel before Kernel after 11

12 Image before Inference Final scale Image after Comparison of kernels True kernel Estimated kernel Kernel before Kernel after Blurry image Matlab s deconvblind Blurry image Our output 12

True sharp image What we do and don t model DO Gamma

Jpeg artifacts Scene motion Color channel correlations

from their own photo collections Type of camera unknown

13 True sharp image What we do and don t model DO Gamma correction Tone response curve (if known) DON T Saturation Jpeg artifacts Scene motion Color channel correlations Results on real images Real experiments Submitted by people from their own photo collections Type of camera unknown Output does contain artifacts Increased noise Ringing Compare with existing methods 13

14 Close-up Original photograph Original Output Our output Matlab s deconvblind Close-up Original photograph Original Our output Matlab s deconvblind 14

15 Our output Photoshop sharpen more Original image Close-up Original photograph Close-up of image Close-up of our output Our output Original image 15

16 Our output Close-up Original image Our output What about a sharp image? Original photograph Our output Original photograph Our output 16

17 Close-up Original image Our output Original photograph Blurry image patch Original photograph Our output Our output Close-up of bird Original Unsharp mask Our output 17

Original photograph Our output Image artifacts & estimated

image patches, NOT the image patterns shown Summary Method for

can handle complicated blur kernels Uses natural image

Overview Removing Camera Shake Non-blind Blind Removing Motion

18 Original photograph Our output Image artifacts & estimated kernels s Code available online Image patterns Note: blur kernels were inferred from large image patches, NOT the image patterns shown Summary Method for removing camera shake from real photographs First method that can handle complicated blur kernels Uses natural image statistics Non-blind deconvolution currently simplistic Overview Removing Camera Shake Non-blind Blind Removing Motion Blur Non-blind Blind Things we have yet to model: Correlations in colors, scales, kernel continuity JPEG noise, saturation, object motion 18

19 Input Photo Deblurred Result Traditional Camera Our Camera Shutter is OPEN Flutter Shutter 19

20 Shutter is OPEN and CLOSED Comparison of Blurred Images Lab Setup Implementation Completely Portable Sync Function Blurring == Convolution Preserves High Frequencies!!! Traditional Camera: Box Filter Flutter Shutter: Coded Filter 20

21 Comparison Inverse Filter stable Inverse Filter Unstable Short Exposure Long Exposure Coded Exposure Overview Matlab Lucy Our result Ground Truth Removing Camera Shake Non-blind Blind Removing Motion Blur Non-blind Blind Use statistics to determine blur size Assumes direction of blur known 21

22 Input image Deblur whole image at once Local Evidence Proposed boundary Result image Input image (for comparison) 22

p( b; xjy) = k p(yjb; x) p( x) p( b) Let y = 2 σ 2 = 0.

surface: ar gmax b;x p( x; bjy) 0.16 0.16 0.14 0.14 0.12 0.12 Bayes p(b y) 0.1 0.

23 p( b; xjy) = k p(yjb; x) p( x) p( b) Let y = 2 σ 2 = N(yjbx ; ¾ 2 ) p( b; xjy) = k p(yjb; x) p( x) p( b) p( b; xjy) = k p(yjb; x) p( x) p( b) Gaussian distribution: N ( xj0; 2) Marginal distribution p(b y) p( bjy) = R p( b; xjy) dx = k R p( yjb; x) p( x) dx MAP solution Highest point on surface: ar gmax b;x p( x; bjy) Bayes p(b y) Bayes p(b y) b b 23

24 MAP solution Highest point on surface: ar gmax b;x p( x; bjy) Variational Bayes True Bayesian approach not tractable Approximate posterior with simple distribution Fitting posterior with a Gaussian Approximating distribution q( x; b) is Gaussian Minimize KL(q(x;b)jj p( x; bjy)) KL-Distance vs Gaussian 11 width 10 9 KL(q p) Gaussian width Fitting posterior with a Gaussian Approximating distribution q( x; b) is Gaussian Minimize KL(q(x;b)jj p( x; bjy)) Variational Approximation of Marginal Variational p(b y) True marginal MAP b 24

25 Try sampling from the 1 model Let true b = 2 Repeat: Sample x ~ N(0,2) Sample n ~ N(0,σ 2 ) p(b y) y = xb + n b Compute p MAP (b y), p Bayes (b y) & p Variational (b y) Multiply with existing density estimates (assume iid) Setup of Variational Approach Work in gradient domain: x- b= y! r x - b= r y Approximate posterior p( r x; bjr y) with q( r x; b) Assume q(r x;b) = q(r x)q(b) q( r x) is Gaussian on each pixel q( b) Cost function is rectified Gaussian on each blur kernel element K L(q(r x)q(b) jj p( r x; bjr y)) 25

Recent Advances in Image Deblurring. Seungyong Lee (Collaboration w/ Sunghyun Cho)

Recent Advances in Image Deblurring Seungyong Lee (Collaboration w/ Sunghyun Cho) Disclaimer Many images and figures in this course note have been copied from the papers and presentation materials of previous