Toward Non-stationary Blind Image Deblurring: Models and Techniques

Size: px

Start display at page:

Download "Toward Non-stationary Blind Image Deblurring: Models and Techniques"

Maryann Walsh
5 years ago
Views:

1 Toward Non-stationary Blind Image Deblurring: Models and Techniques Ji, Hui Department of Mathematics National University of Singapore NUS, 30-May-2017

2 Outline of the talk Non-stationary Image blurring Motion blurring Out-of-focus blurring Brief Introduction to blind deconvolution (stationary image blurring) A two-stage approach for recovering images with nonstationary motion blurring A fast method for estimating de-focus map of images with non-stationary out-of-focus blurring

3 Image blurring Degradation of sharpness and contrast of the image, causing loss of image details (high frequency information)

4 Image blurring Degradation of sharpness and contrast of the image, causing loss of image details (high frequency information) Motion blurring Out-of-focus blurring

5 Motion blurring Blurring caused by the relative motion between camera or object during shutter time t Larger motion; more blurring Object point image sensor t+δt lens Motion blurring

6 Out-of-focus (defocus) blurring Blurring caused by objects away from focal plane More away from focal plane; more blurring Defocus plane Focal plane Lens Image sensor c d d f f 0

7 Motion blur : motion path on image plane Pinhole camera x X f y Y Z 1 Z 3D rigid camera motion: t x x y y t t= t, z z 2D Motion field in image 2 xt () f tx tzx xy x ( x 1) y y z rt () y() t ty tzy 2 Z ( y 1) x xy y x z Spatially invariant motion blur == constant motion field Scene depth Z is close to constant Camera motion: t ( t, t, 0); 0 x y

8 Motion blurring: Stationary VS Non-stationary Constant scene depth In-plane camera translation Slanted scene depth In-plane camera translation Rotational camera motion Dynamic scene with moving object

9 De-focus blurring: usually nonstationary Image usually contains several depth layer Different layer has different blurring De-focus blurring amount Ordinal scene depth

10 Convolution model for stationary image blurring = + : Convolution (non-invertible) g p mn, gk1, k2] p[ m k1, n k2 [ ] Blind deblurring: f Blurred image known g Sharp image unknown p Kernel (PSF) unknown Noise unknown

11 Regularization for blind image deconvolution f g p Infinite number of solutions: how to avoid the trivial solution: f f l 1 -norm relating regularization (either TV or wavelet) f g p ( g) ( p) s t p g, p min.. 1 ( g) Wg ; 2 2 ( h) Wh 1 h 2 { p: p[ j] 1, p[ j] 0} jj 2 [1] J. Cai, H. Ji, C. Liu and Z. Shen, Blind motion deblurring from a single image using sparse approximation, CVPR 09

12 Regularization for blind image deconvolution f g p Infinite number of solutions: how to avoid the trivial solution: f f l 1 -norm relating regularization (either TV or wavelet) f g p ( g) ( p) s t p g, p min.. 1 ( g) Wg ; 2 2 ( h) Wh 1 h 2 { p: p[ j] 1, p[ j] 0} jj 2 [1] J. Cai, H. Ji, C. Liu and Z. Shen, Blind motion deblurring from a single image using sparse approximation, CVPR 09 2 Remark: h 2 is for avoiding convergence to δ, as 1 2 n [1,,1] arg min h s.t. h 2

13 Demonstration Real blurred image Our result

14 Demonstration Real blurred image Our result

15 Non-stationary image blurring Motion blurring Out-of-focus blurring

16 Stationary VS Non-stationary (in 1D case) Matrix form of Convolution: f Kg, K nn Stationary: all rows of K are same, up to a shift Nonstationary: each row of K might be different Motion blurring Each row is of free-form, but with compact support Out-of-focus blurring Each row is a Gaussian, but with different standard deviation

17 A piece-wise stationary model based framework [2] Input blurred image Piece-wise uniform motion-blur approx. Estimate one kernel for each region Identify and remove erroneous kernels Interp. for blurring matrix & deblurring PCA-based Interp. for blurring matrix Non-blind Image deblurring robust to matrix error The output [2] H. Ji and K. Wang, A two-stage approach to remove spatially-varying motion blur from a single photograph, CVPR 12

18 Sensitivity of deconvolution to blur kernel error Clear image Image blurred by horizontal constant kernel of size 10 pixels

19 Sensitivity of deconvolution to blur kernel error Clear image Image blurred by horizontal constant kernel of size 10 pixels Image de-blurred by l 1 -norm based regularization, and an erroneous kernel (horizontal constant of size 12 pixels

20 Robust non-blind image deconvolution [3] An EIV (Error-In-Variable) model for de-convolution f p g n ( p ) g n : kernel err or; n: image noise p p Problem : given f and p, estimate g [3] H. Ji and K. Wang, Robust image de-convolution with an inaccurate blur kernel. IEEE Trans. Image Proc

21 Robust non-blind image deconvolution [3] An EIV (Error-In-Variable) model for de-convolution f p g n ( p ) g n : kernel err or; n: image noise p p Problem : given f and Reformulation: Two unknowns: p, estimate g f = (p -d p )* g + n = p*g - q + n { g : clear image q p g : image distortion by kernel error [3] H. Ji and K. Wang, Robust image de-convolution with an inaccurate blur kernel. IEEE Trans. Image Proc

22 Two sparsity-relating regularization Sparsity of q = d p * g in pixel space g p * g p* g d p * g

23 Two sparsity-relating regularization q = d p * g Sparsity of in pixel space g p * g p* g d p * g Second: Artifacts in solution caused by kernel error is sparse in DCT domain Result using Erroneous kernel The resulting error along edges

24 Convex minimization model Model for robust image deconvolution Clear image Artifacts System error W: framelet transform, D: DCT transform

25 Demo. Blurry image Stationary blind deconvolution Gupta et al. ECCV 10 (nonstationary Our nonstationary method

26 Demo. Blurry image Stationary blind deconvolution Gupta et al. ECCV 10 (nonstationary) Our nonstationary method

27 Demo. Blurry image Stationary blind deconvolution Whyte et al. CVPR 10 (nonstationary) Our nonstationary method

28 Demo. Blurry image Stationary blind deconvolution Whyte et al. CVPR 10 (nonstationary) Our nonstationary method

29 Out-of-focus (defocus) blurring Defocus plane Focal plane Lens Image sensor c Circle of Confusion d c d f f 0 d d f f 2 0 d n ( d f ) s f 0 for each pixel r, 1 0 blur kernel r r ( 0) exp( ) ( r0 ) pr ~ (defocus amount) c( r ) ( r ) (Gaussian s.t.d. ) 0 0

30 Defocus amount estimation from a single image [4] Darker color = less defocus amount = less blurring = closer distance [4] G. Xu, Y. Quan and H. Ji, Defocus amount estimation via maximum rank of patches, 2017

Defocus amount estimation from a single image [4] Darker color = less defocus amount = less blurring = closer distance Defocus amount ordinal scene depth

31 Defocus amount estimation from a single image [4] Darker color = less defocus amount = less blurring = closer distance Defocus amount ordinal scene depth foreground/background segmentation Image matting; image refocusing [4] G. Xu, Y. Quan and H. Ji, Defocus amount estimation via maximum rank of patches, 2017

32 Rank of patches and Separable blur kernel Proposition Consider three matrices U,I,G associated by 2D convolution: I=U G. Suppose U is positive (negative) definite and G gg. Then, Rank(I)= gˆ, whe e gˆ is DFT of 0 r g. Constructing positive (negative) patches at edges points Sampling K image patches with different orientations. One of these different oriented patches is positive definite.

Constructing positive (negative) patches at edges points Sampling K image patches with different orientations.

33 Rank of patches and Separable blur kernel Proposition Consider three matrices U,I,G associated by 2D convolution: I=U G. Suppose U is positive (negative) definite and G gg. Then, Rank(I)= gˆ, whe e gˆ is DFT of 0 r g. Constructing positive (negative) patches at edges points Sampling K image patches with different orientations. One of these different oriented patches is positive definite. Defocus amount and maximum rank of oriented patches 1 1 (or c max Rank (P ) 0kK k ) ~ ln(1 ) n

34 Completion of defocus map Input image Defocus estimation at edge points

35 Completion of defocus map Input image Defocus estimation at edge points Defocus map completion by matting Laplacian method Keep the values in complete map are close to the ones given at edge points Keeping the discontinuities of defocus map consistent with image edges.

36 Demonstration Input image defocus map at edges

37 Demonstration Input image defocus map at edges Complete defocus map

38 Demonstration Input image defocus map at edges Complete defocus map Foreground segmentation

39 More Input image Bae et al. Tang et al. ours

40 Evaluation on fore/background segmentation Test defocus dataset from CUHK: 704 images Manually segmented in-focus foreground and out-of-focus background Precision and recall curves of foreground/background segmentation using the defocus maps generated by different methods

41 Occlusion-aware image composition Source image Target

42 Occlusion-aware image composition Source image Image composition 1 Target

43 Occlusion-aware image composition Source image Image composition 1 Target Image composition 2

44 List of co-authors Blind deconvolution for removing motion blur Jianfeng Cai, Chaoqiang Liu and Zuowei Shen Non-stationery blind motion deblurring Wang Kang Non-stationary out-of-focus blurring estimation and applications Xu Guodong and Yuhui Quan

45 Thank You

Spline wavelet based blind image recovery

Spline wavelet based blind image recovery Ji, Hui ( 纪辉 ) National University of Singapore Workshop on Spline Approximation and its Applications on Carl de Boor's 80 th Birthday, NUS, 06-Nov-2017 Spline