Deblurring. Basics, Problem definition and variants

Size: px

Start display at page:

Download "Deblurring. Basics, Problem definition and variants"

Wesley Gilbert
6 years ago
Views:

1 Deblurring Basics, Problem definition and variants

2 Kinds of blur Hand-shake Defocus Credit: Kenneth Josephson Motion

5 Credit: Kenneth Josephson

6 Kinds of blur Spatially invariant vs. Spatially varying

7 Kinds of blur 1D vs 2D blur kernels

10 Problem definition Lets assume a spatially invariant blurring model

11 LTI modeling Blurring h(x)

12 LTI modeling Blurring h(x) Can we get back the sharp image?

13 LTI modeling Blurring h(x) Could we consider getting back the sharp image?

14 LTI modeling Blurring h(x) Could we consider getting back the sharp image? Challenges 1. What if has zeros? 2. Do we know h(t)? 3. (peril of DFT) Periodicity!

15 Image Priors Suppose h(t) is known, today image deblurring is considered close-to-solved Regularizer Data term Commonly used regularizers Wavelet sparsity Gradients of the image are sparse Poisson gradients Mixture of Gaussians model for image gradients

16 BlurBurst Multi-image deblurring technique

17 Telephoto imaging Low vision photography

18 Large blur Blur due to camera shake can be very large Telephoto (zoom) imaging Narrow field of view implies small handshake are amplified Low-light imaging Large exposure times leads to large blurs Hand-held high dynamic range (HDR) imaging

19 Deblurring on large blur (Shan et al, 2008)

20 Deblurring on large blur (Shan et al, 2008)

21 Why is large blur hard? Extreme loss of high-frequency information

22 Use multiple images!!! Most cameras have a burst-mode to capture images in rapid succession Two key ideas More images implies great resiliency to noise Camera shake is never the same across images! Different frequency components are attenuated in different images

23 Use multiple images!!!

24 Use multiple images!!! Most cameras have a burst-mode to capture images in rapid succession Two key ideas More images implies great resiliency to noise Camera shake is never the same across images! Different frequency components are attenuated in different images Challenges Registration!!!!

Why deblurring is easier with multiple image?

perpendicular directions of blur kernels.

suppression (c) Two images Blur PSF 25 pixels

25 Why deblurring is easier with multiple image? (a) Single image Blur PSF 5 pixels (b) Single image Blur PSF 25 pixels Compensating high-frequency information each other along perpendicular directions of blur kernels. Improving SNR just by virtue of noise suppression (c) Two images Blur PSF 25 pixels (d) Six image Blur PSF 25 pixels (a) (b) (c) (d) Orig. Comparison of select patches

26 Algorithm Blur Latent Registration Kernel Image Regularization: L1 TV norm norm Feature extraction and matching: - Extracting SIFT features and match them to a pre-selected reference image. Homography estimation: - The feature correspondences are used to fit a homography transformation using RANSAC. Noise Model: Solved by by SPG-L1 M-Fista

27 Deblurring on large blur (Shan et al, 2008)

29 (Shan et al, 2008)

31 Input image

32 Shan et al.

33 BlurBurst (us)

34 Tripod image

35 Real Data Results Low Light Scene Exposure time: 1.6sec 49x49 blur kernel size 6 input images at 512x512 Shan s Single Deblur (08) Cho s Single Deblur (09) Input Images Estimated Blur Kernels Šroubek s Multi Deblur (12) Ours

Shan s Single Deblur (08) Cho s Single Deblur (09) Input

36 Real Data Results Low Light Scene Exposure time: 0.25sec 39x39 blur kernel size 8 input images at 512x512 Shan s Single Deblur (08) Cho s Single Deblur (09) Input Images Estimated Blur Kernels Šroubek s Multi Deblur (12) Ours

37 Real Data Results Telephoto & Low Light Focal Length: 300mm Exposure time: 0.2sec 49x49 blur kernel size 6 input images at 512x512 Shan s Single Deblur (08) Cho s Single Deblur (09) Input Images Estimated Blur Kernels Šroubek s Multi Deblur (12) Ours

38 Coded Exposure Photography: Motion deblurring using fluttered shutter Raskar, Agrawal, and Tumblin SIGGRAPH 2006 Slide-deck credit: Raskar, Agrawal, Tumblin

39 Traditional Camera Shutter is OPEN

40 Our Camera Flutter Shutter

41 Shutter is OPEN and CLOSED

42 Comparison of Blurred Images

43 Implementation Completely Portable

44 Lab Setup

45 Sync Function Blurring == Convolution Traditional Camera: Box Filter

46 Preserves High Frequencies!!! Flutter Shutter: Coded Filter

47 Comparison

48 Inverse Filter Unstable Inverse Filter stable

49 Motion Blur as Convolution

50 Short Exposure Long Exposure Coded Exposure Our result Matlab Lucy Ground Truth

51 Are all codes good? Our Code All ones Alternate Random

52 Need to consider zero padded codes!!!

53 License Plate Retrieval

54 License Plate Retrieval

57 Input Image

58 Rectified Crop Deblurred Result

59 Varying Exposure Video Exposure Time Fourier Transform Slide courtesy Agrawal

60 Varying Exposure Video No common nulls Exposure Time Fourier Transform Exposure Time Slide courtesy Agrawal

61 Varying Exposure Video No common nulls Exposure Time Fourier Transform Exposure Time Exposure Time Slide courtesy Agrawal

62 Varying Exposure Video = PSF Null-Filling Fourier Transform Joint Frequency Spectrum Preserves All Frequencies Slide courtesy Agrawal

63 Varying Exposure Video

64 Blurred Photos Deblurred Result Slide courtesy Agrawal

Key Idea: PSF Null-Filling Individual non-invertible PSF s combined into jointly-invertible PSF Information lost in any single photo is captured in some other

65 Key Idea: PSF Null-Filling Individual non-invertible PSF s combined into jointly-invertible PSF Information lost in any single photo is captured in some other photo For motion deblurring Achieve PSF null-filling by varying the exposure time of successive photos Varying Exposure Photo 1 Photo 2 Photo 3 Slide courtesy Agrawal

66 Single Image Deblurring (SID) Multiple Image Deblurring (MID) Slide courtesy Agrawal

67 Motion Invariant Photography Levin, Sand, Cho, Durand, Freeman SIGGRAPH 2008 Slide-deck credit: Levin et al.

68 Motion Blur Most scene is static Can moving linearly from left to right

69 Overcoming motion blur? Reduce shutter speed But reduces amount of light When shutter speed is as fast as we physically can and there is still blur Computational solution: Deconvolution

70 Why is motion deblurring hard? Need to know blur kernel (motion velocity)? Input blurry image Correct kernel Output from correct kernel? Input blurry image Wrong kernel Output from wrong kernel

71 Why is motion deblurring hard? Need to know blur kernel (motion velocity) Need to segment image Entire image deblurred with kernel corresponding to the cans velocity

72 Why is motion deblurring hard? Need to know blur kernel (motion velocity) Need to segment image Existing solutions: - Multiple input images/ additional hardware: Bascle et al 1996; Rav-Acha and Peleg 2005; Zheng 2005; Bar et al 2007; Ben-Ezra and Nayar 2004; Yuan et al Image statistics- single image with restricted assumptions: Fergus et al 2006; Levin 2006; Shan et al 2008;

73 Why is motion deblurring hard? Need to know blur kernel (motion velocity) Need to segment image Information loss (reduced signal to noise ratio) blurred input deblurred static input

74 Why is motion deblurring hard? Need to know blur kernel (motion velocity) Need to segment image Information loss - Existing approach: Flutter Shutter, Raskar et al 2006 Close & open shutter during exposure, achieves broad-band kernel. But does not address kernel estimation and segmentation

75 Counter intuitive solution: To reduce motion blur, increase it! - move camera as picture is taken Makes blur invariant to motion- can be removed with spatially uniform deconvolution - kernel is known (no need to estimate motion) - kernel identical over the image (no need to segment) Makes blur easy to invert

Dowski 94 Vary object/detector distance during

76 Inspiration: depth invariant defocus Wave front coding - manipulate optical element Cathey and Dowski 94 Vary object/detector distance during integration - Hausler 72 - Nagahara, Kuthirummal, Zhou, Nayar 08

77 Motion invariant blur- disclaimers: Assumes 1D motion (e.g. horizontal) Degrades quality for static objects

78 Controlling motion blur

79 Controlling motion blur Can we control motion blur?

80 Controlling motion blur

81 Controlling motion blur

82 Controlling motion blur

83 Motion invariant blur Controlling motion blur

84 Parabolic sweep Time t Sensor position x(t)=a t 2 Start by moving very fast to the right Continuously slow down until stop Continuously accelerate to the left Intuition: For any velocity, there is one instant where we track perfectly. Sensor position x

85 Motion invariant blur

86 Motion invariant blur

87 Motion invariant blur

88 Static camera Unknown and variable blur kernels Our parabolic input Blur kernel is invariant to velocity Our output after deblurring NON-BLIND deconvolution

89 Deblurring and information loss Assume: we could perfectly identify blur kernel Which camera has motion blur that is easy to invert? - Static? Flutter Shutter? Parabolic? Prove: parabolic motion achieves near optimal information preservation blurred input deblurred static input

90 Comparing camera reconstruction Blurred input Static Flutter Shutter Parabolic Deblurred output Note: synthetic rendering, exact PSF is known

91 Hardware construction Ideally move sensor (requires same hardware as existing stabilization systems) In prototype implementation: rotate camera variable radius cam Lever Rotating platform

92 Linear rail Static camera input- Unknown and variable blur Our parabolic input- Blur is invariant to velocity

93 Linear rail Static camera input- Unknown and variable blur Our output after deblurring- NON-BLIND deconvolution

94 Human motion- no perfect linearity Input from a static camera Deblurred output from our camera

95 Violating 1D motion assumption- forward motion Input from a static camera Deblurred output from our camera

96 Violating 1D motion assumption- stand-up motion Input from a static camera Deblurred output from our camera

97 Violating 1D motion assumption- rotation Input from a static camera Deblurred output from our camera

Coded Computational Photography!

Coded Computational Photography! EE367/CS448I: Computational Imaging and Display! stanford.edu/class/ee367! Lecture 9! Gordon Wetzstein! Stanford University! Coded Computational Photography - Overview!!