Spline wavelet based blind image recovery

Size: px

Start display at page:

Download "Spline wavelet based blind image recovery"

Sharleen Crawford
5 years ago
Views:

1 Spline wavelet based blind image recovery Ji, Hui ( 纪辉 ) National University of Singapore Workshop on Spline Approximation and its Applications on Carl de Boor's 80 th Birthday, NUS, 06-Nov-2017

2 Spline function [ref] B-splines (plot by MATLAB curvefit toolbox) Linear B-spline Refinable functions k 1 1 k 1 Bk() t Bk(2 t j) k 2 j 0 j Cubic B-spline [ref] Carl de Boor, A Practical Guide to Splines, Springer, 2001.

3 Spline wavelet tight frames [ref] MRA-based spline wavelet tight frames 1 { )} 2,, : ( 2 j j k k j, j, k Linear B-spline Linear spline framelet Wavelet filter bank a0 1 [1,2,1]; a [ 1,0,1]; a2 [ 1,2, 1] [ref] I. Daubechies, B.Han, A. Ron, and Z. Shen, Framelets:MRA-based constructions of wavelet frames, Applied and Computational Harmonic Analysis, 14 (2003), 1-46

4 Discrete tensor Gabor tight frames w/. Optimal orientation selectivity [1] Gabor system: { g ( m) g(( m ak) mod N) e,0 m N} 2 i bm k, k K, L Gabor filter bank { g } L Real part Imaginary part A good choice of window : square root of B-spline B ( ) 1, for B-splines on knots {0,,, } k k an a ak [1] Hui Ji, Zuowei Shen,Yufei Zhao, Directional Frames for Image Recovery: Multiscale Discrete Gabor Frames, Journal of Fourier Analysis and Applications, 2017

5 Discrete wavelet decomposition and reconstruction Cascade algorithm a 0 ( ) 2 w O,n 2 a 0 ( ) w O,n-1 a 1 ( ) 2 w 1,n 2 a 1 ( ) w O,n-1 a L ( ) 2 w L,n 2 a L ( ) Tight frame property Analysis operator: W Synthesis operator : W T W T W I; WW T I

ck k most values of c k are zero or close to zero l 1 -norm relating

6 Sparsity prompting regularization in wavelet domain An ill-posed inverse problem that estimating f from g f Ag n Suppose f is compressible under wavelet system g ck k most values of c k are zero or close to zero l 1 -norm relating regularization 1 2 g : argmin g f Ag Wg 2 k f : images { k }: wavelet system {c k}: coefficients 2 1

7 Single particle analysis in Electron Microscopy Computing 3D structure of macromolecules from TEM images TEM images A large-scale linear problem with low very SNR ratio data Ax b

structures of Dengue virus CWAIP, NUS [2] M. Li, Z. Fan, H. Ji and Z.

8 Spline wavelet based method 3D reconstruction in EM [2] Sparsity-based forward-backward projection scheme u ( k 1 : ( I A A) W T Wuk) A g ~500 TEM images of Dengue virus NUS CBIS 3D structures of Dengue virus CWAIP, NUS [2] M. Li, Z. Fan, H. Ji and Z. Shen, Wavelet frame based algorithm for 3D reconstruction in electron microscop, SIAM Journal on Scientific Computing, 2 012

9 Image blurring Degradation of sharpness and contrast of the image, causing loss of image details (high frequency information) Motion blurring Out-of-focus blurring

10 Blind image deblurring Blind image de-convolution: recovering the clear image from a blurred one without given information on how it is blurred

11 Blind image deblurring Blind image de-convolution: recovering the clear image from a blurred one without given information on how it is blurred

12 Blind image deblurring Blind image de-convolution: recovering the clear image from a blurred one without given information on how it is blurred

13 Motion blurring Blurring caused by the relative motion between camera or object during shutter time Larger motion; more blurring Object point t image sensor t+δt lens

Type of motion blurring Blurring effect caused by 2D image

between camera and scene Constant scene depth Image-plane

camera translation Stationary: blurring is same everywhere.

14 Type of motion blurring Blurring effect caused by 2D image motion determined by Scene depth configuration 3D motion between camera and scene Constant scene depth Image-plane translation Varying scene depth Camera roation Image-plane camera translation Stationary: blurring is same everywhere. Nonstationary: blurring is different at different pixels

15 Blind motion deblurring Blind image de-convolution: recovering the clear image from a blurred one without knowing how it is blurred

16 Blind motion deblurring Blind image de-convolution: recovering the clear image from a blurred one without knowing how it is blurred

17 Blind motion deblurring Blind image de-convolution: recovering the clear image from a blurred one without knowing how it is blurred

18 Stationary blind motion deblurring Convolution: shift-invariant blurring f p g = + f g p A ill-posed bi-liner inverse problem Estimating (p, g) form f Many mathematical feasible solutions e.g. f p g f

19 Regularization for blind image deconvolution [5] Optimization model 1 2 min f p g ( g) ( p) s. t. p g, p l 1 -norm relating regularization in wavelet transform 1( g) Wg ; ( p) Wp h j { p : p[ j] 1, p[ j] 0} Alternating iteration based numerical solver 2 F 2 Remark: h 2 is for avoiding convergence to δ arg min h s.t. h 2 n [5] Jianfeng Cai, Hui Ji, Chaoqiang Liu and Zuowei. Shen, Framelet based blind image deblurring from a single image, IEEE TIP 2012

20 Demonstration Input Fergus et al. Shan et al. Tzikas Cai et al. Ours

21 Demonstration Input Fergus et al. Shan et al. Tzikas Cai et al. Ours

22 Demonstration Real blurred image Our result

23 Demonstration Real blurred image Our result

Non-stationary image blurring Problem formulation f Kg, K n n K: a block-wise band matrix Stationary: all rows of K are same, up to a shift Nonstationary:

24 Non-stationary image blurring Problem formulation f Kg, K n n K: a block-wise band matrix Stationary: all rows of K are same, up to a shift Nonstationary: each row of K might be different Two key questions How to efficiently approximate measurement matrix K How to estimating clear image f with a non-perfect K

A piece-wise stationary model based framework [2] Input blurred image Piece-wise uniform motion-blur approx. Estimate one kernel for each region Removing erroneous estimation PCA-based Interp.

25 A piece-wise stationary model based framework [2] Input blurred image Piece-wise uniform motion-blur approx. Estimate one kernel for each region Removing erroneous estimation PCA-based Interp. for blurring matrix Interpolation matrix PCA-based Interp. for blurring matrix [2] Hui Ji and Kang Wang, A two-stage approach to remove spatially-varying motion blur from a single photograph, CVPR 12

26 Sensitivity of deconvolution to blur kernel error Clear image Image blurred by horizontal constant kernel of size 10 pixels

27 Sensitivity of deconvolution to blur kernel error Clear image Image blurred by horizontal constant kernel of size 10 pixels Image de-blurred by l 1 -norm based regularization, and an erroneous kernel (horizontal constant of size 12 pixels

28 Convex minimization model Key idea Error induced by kernel error δ K g is sparse Artifacts (ringing artifacts) are sparse in DCT domain Model for robust image deconvolution 1 T 1 ( c, h, u) f K( W c D u) h ( I W W ) c T Clear image Artifacts Error induced by kernel W: framelet transform, D: DCT transform [3] Hui Ji and Kang Wang, Robust image de-convolution with an inaccurate blur kernel. IEEE Trans. Image Proc

29 Demo. Blurry image Stationary blind deconvolution Whyte et al. CVPR 10 (nonstationary) Our nonstationary method

30 Demo. Blurry image Stationary blind deconvolution Whyte et al. CVPR 10 (nonstationary) Our nonstationary method

31 Out-of-focus (defocus) blurring Blurring caused by objects away from focal plane More away from focal plane; more blurring Defocus plane Focal plane Lens Image sensor c d d f f 0 Circle of Confusion c d d f f 2 0 d n ( d f ) s f 0

32 De-focus blurring: usually nonstationary Image usually contains several depth layer Different layer has different blurring De-focus blurring amount f Kg Ordinal scene depth 2 1 r r0 2 Each row of K is a Gaussian kernel pr ( ) exp( ) ( r0 )

Defocus map estimation A two-stage approach Defocus amount estimation at edge points Completing defocus map by propagation Matting Laplacian method for map completion [8] Completion is done by

33 Defocus map estimation A two-stage approach Defocus amount estimation at edge points Completing defocus map by propagation Matting Laplacian method for map completion [8] Completion is done by keeping the defocus amount close to the given ones at edge points, and keeping the discontinuities consistent with that of image edges. [8] Anat, Alex Rav-Acha, and Dani Lischinski. Spectral matting, IEEE PAMI 2008

0 Two observations The rank of a positive/negative de

34 Estimating defocus blur by rank of local patches [9] Proposion 1. convolution: I=U Consider three matrices U,I,G related by 2D G. Suppose U is positive (negative) definite and G gg. Then, Rank(I)= gˆ, where gˆ is DFT of g. 0 Two observations The rank of a positive/negative definite patch after defocus blurring determines s.t.d. of a Gaussian kernel Rotation can convert a rank-deficit patch to a full-rank patch [9] Gudong Xu, Yuhui Quan, and Hui Ji, Estimating defocus blur through rank of local patches, ICCV 17

35 Rank-based estimator Sampling symmetric patches in gradient domain along different orientations: Q0[ i, j] I[ i0 p i, j0 p j]: horizontal Q1[ i, j] I[ i0 p j, j0 p i]: vertical Q2[ i, j] I[ i0 i j, j0 p i]: diagonal Q3[ i, j] I[ i0 p i, j0 i j]: anti-diagonal T P Q Q, k 1,2,3,4 k k k Defocus amount estimator ln(1 max rank( P ) / n), n P 0 k 3 k 0

36 Demonstration Input image defocus map at edges

37 Demonstration Input image defocus map at edges Complete defocus map

38 Foreground/background segmentation Input image defocus map at edges Complete defocus map Foreground segmentation

39 More Input image Bae et al. Tang et al. ours

40 Evaluation on fore/background segmentation Test defocus dataset from CUHK: 704 images Manually segmented in-focus foreground and out-of-focus background Precision and recall curves of foreground/background segmentation using the defocus maps generated by different methods

41 Blind defocus deblurring and Recocus Blind defocus de-convolution: recover a all-in-focus image from an image with both in-focus and defocus region

42 Blind defocus deblurring and Recocus Blind defocus de-convolution: recover a all-in-focus image from an image with both in-focus and defocus region

43 Blind defocus deblurring and Recocus Blind defocus de-convolution: recover a all-in-focus image from an image with both in-focus and defocus region

44 Blind defocus deblurring and Recocus Blind defocus de-convolution: recover a all-in-focus image from an image with both in-focus and defocus region Applications Surveillance Photography Robotics

45 From an image with defocus to an all-in-focus image Problem formulation f: input image with multiple de-focus regions α i : binary mask of i-th defocus region, i.e. 1 for related pixels and 0 otherwise u i : i-th in-focus region k i : the defocus blur kernel of i-th de-focus region η: noise Unknowns: everything in right sides Output: f L f i ( ki ui ) i 0 L i i 0 u i {, k, u, } i i i i

46 Removing defocus blurring from image [10] Alternating scheme between blind deconvolution and segmentation { k, u } { } i i i [10] Guodong Xu, Chaoqiang Liu and Hui Ji, Removing partial out-of-focus blur from images. Preprint, 2017

47 Refining α i, given {k i t, u i t } Observation A region is deblurred by an inaccurate kernel will lead to noticeable ringing artifacts Basic idea: deblurring image with the estimated kernel The pixels with ringing artifacts in estimated defocus region should be in in-focus region The pixels with ringing artifacts in the estimated in-focus region should be in de-focus region Key question? How to detect the pixels with ringing artifacts?

48 Residual function for detecting ringing artifacts Ringing artifacts cannot be removed by re-blurring image with the same kernel used for deblurring f k 1 g; ( k, f ) k g f g : estimate of g by Wiener filter

49 Residual function for detecting ringing artifacts Ringing artifacts cannot be removed by re-blurring image with the same kernel used for deblurring f k 1 g; ( k, f ) k g f g : estimate of g by Wiener filter Image: left half clear, right half blurred

50 Residual function for detecting ringing artifacts Ringing artifacts cannot be removed by re-blurring image with the same kernel used for deblurring f k 1 g; ( k, f ) k g f g : estimate of g by Wiener filter Image: left half clear, right half blurred Deblurrd by blur kernel of right half

51 Residual function for detecting ringing artifacts Ringing artifacts cannot be removed by re-blurring image with the same kernel used for deblurring f k 1 g; ( k, f ) k g f g : estimate of g by Wiener filter Image: left half clear, right half blurred Reblurred by the same kernel

Residual function for detecting ringing artifacts Ringing artifacts cannot be removed by re-blurring image with the same kernel used for deblurring f

52 Residual function for detecting ringing artifacts Ringing artifacts cannot be removed by re-blurring image with the same kernel used for deblurring f k 1 g; ( k, f ) k g f g : estimate of g by Wiener filter Image: left half clear, right half blurred Residual between blured and reblurred image

53 Blind defocus deblurring Gaussian is a rough approximation to defocus blur kernel

54 Blind defocus deblurring Gaussian is a rough approximation to defocus blur kernel Existing parametric form of defocus kernel

55 Blind defocus deblurring Gaussian is a rough approximation to defocus blur kernel Existing parametric form of defocus kernel Real defocus kernel

56 Blind defocus deblurring Gaussian is a rough approximation to defocus blur kernel Existing parametric form of defocus kernel Real defocus kernel Observation Low rank priori for regularizing defocus blur kernel Optimization model for blind defocus deblurring min ( u k f ) Wu k r 2 2 k, u F F k[ r] 1, k[ r] 0, Rank( k) r 0

57 Demonstration Input Dai et al. Shen et al. ours

58 Demonstration Input Dai et al. Shen et al. ours

59 Demonstration Input Dai et al. Shen et al. ours

60 Demonstration Input Dai et al. Shen et al. ours

61 Demonstration on image refocus Input

62 Demonstration on image refocus Input All-in-focus

63 Demonstration on image refocus Input Image re-focus

64 List of co-authors Blind deconvolution for removing motion blur Jianfeng Cai, Chaoqiang Liu and Zuowei Shen Non-stationery blind motion deblurring Wang Kang Defocus blurring estimator and image refocus Xu Guodong and Yuhui Quan

65 Thank You

Toward Non-stationary Blind Image Deblurring: Models and Techniques

Toward Non-stationary Blind Image Deblurring: Models and Techniques Ji, Hui Department of Mathematics National University of Singapore NUS, 30-May-2017 Outline of the talk Non-stationary Image blurring