Spline wavelet based blind image recovery

Spline wavelet based blind image recovery Ji, Hui ( 纪辉 ) National University of Singapore Workshop on Spline Approximation and its Applications on Carl de Boor's 80 th Birthday, NUS, 06-Nov-2017

Spline function [ref] B-splines (plot by MATLAB curvefit toolbox) Linear B-spline Refinable functions k 1 1 k 1 Bk() t Bk(2 t j) k 2 j 0 j Cubic B-spline [ref] Carl de Boor, A Practical Guide to Splines, Springer, 2001.

Spline wavelet tight frames [ref] MRA-based spline wavelet tight frames 1 { )} 2,, : ( 2 j j k k j, j, k Linear B-spline Linear spline framelet Wavelet filter bank a0 1 [1,2,1]; a 2 1 1 [ 1,0,1]; a2 [ 1,2, 1] 4 4 4 [ref] I. Daubechies, B.Han, A. Ron, and Z. Shen, Framelets:MRA-based constructions of wavelet frames, Applied and Computational Harmonic Analysis, 14 (2003), 1-46

Discrete tensor Gabor tight frames w/. Optimal orientation selectivity [1] Gabor system: { g ( m) g(( m ak) mod N) e,0 m N} 2 i bm k, k K, L Gabor filter bank { g } L Real part Imaginary part A good choice of window : square root of B-spline B ( ) 1, for B-splines on knots {0,,, } k k an a ak [1] Hui Ji, Zuowei Shen,Yufei Zhao, Directional Frames for Image Recovery: Multiscale Discrete Gabor Frames, Journal of Fourier Analysis and Applications, 2017

Discrete wavelet decomposition and reconstruction Cascade algorithm a 0 ( ) 2 w O,n 2 a 0 ( ) w O,n-1 a 1 ( ) 2 w 1,n 2 a 1 ( ) w O,n-1 a L ( ) 2 w L,n 2 a L ( ) Tight frame property Analysis operator: W Synthesis operator : W T W T W I; WW T I

Sparsity prompting regularization in wavelet domain An ill-posed inverse problem that estimating f from g f Ag n Suppose f is compressible under wavelet system g ck k most values of c k are zero or close to zero l 1 -norm relating regularization 1 2 g : argmin g f Ag Wg 2 k f : images { k }: wavelet system {c k}: coefficients 2 1

Single particle analysis in Electron Microscopy Computing 3D structure of macromolecules from TEM images TEM images A large-scale linear problem with low very SNR ratio data Ax b

Spline wavelet based method 3D reconstruction in EM [2] Sparsity-based forward-backward projection scheme u ( k 1 : ( I A A) W T Wuk) A g ~500 TEM images of Dengue virus NUS CBIS 3D structures of Dengue virus CWAIP, NUS [2] M. Li, Z. Fan, H. Ji and Z. Shen, Wavelet frame based algorithm for 3D reconstruction in electron microscop, SIAM Journal on Scientific Computing, 2 012

Image blurring Degradation of sharpness and contrast of the image, causing loss of image details (high frequency information) Motion blurring Out-of-focus blurring

Blind image deblurring Blind image de-convolution: recovering the clear image from a blurred one without given information on how it is blurred

Motion blurring Blurring caused by the relative motion between camera or object during shutter time Larger motion; more blurring Object point t image sensor t+δt lens

Type of motion blurring Blurring effect caused by 2D image motion determined by Scene depth configuration 3D motion between camera and scene Constant scene depth Image-plane translation Varying scene depth Camera roation Image-plane camera translation Stationary: blurring is same everywhere. Nonstationary: blurring is different at different pixels

Blind motion deblurring Blind image de-convolution: recovering the clear image from a blurred one without knowing how it is blurred

Stationary blind motion deblurring Convolution: shift-invariant blurring f p g = + f g p A ill-posed bi-liner inverse problem Estimating (p, g) form f Many mathematical feasible solutions e.g. f p g f

Regularization for blind image deconvolution [5] Optimization model 1 2 min f p g ( g) ( p) s. t. p g, p 1 1 2 2 l 1 -norm relating regularization in wavelet transform 1( g) Wg ; ( p) Wp h j 2 2 1 2 { p : p[ j] 1, p[ j] 0} Alternating iteration based numerical solver 2 F 2 Remark: h 2 is for avoiding convergence to δ 1 2 1 arg min h s.t. h 2 n [5] Jianfeng Cai, Hui Ji, Chaoqiang Liu and Zuowei. Shen, Framelet based blind image deblurring from a single image, IEEE TIP 2012

Demonstration Input Fergus et al. Shan et al. Tzikas Cai et al. Ours

Demonstration Real blurred image Our result

Non-stationary image blurring Problem formulation f Kg, K n n K: a block-wise band matrix Stationary: all rows of K are same, up to a shift Nonstationary: each row of K might be different Two key questions How to efficiently approximate measurement matrix K How to estimating clear image f with a non-perfect K

A piece-wise stationary model based framework [2] Input blurred image Piece-wise uniform motion-blur approx. Estimate one kernel for each region Removing erroneous estimation PCA-based Interp. for blurring matrix Interpolation matrix PCA-based Interp. for blurring matrix [2] Hui Ji and Kang Wang, A two-stage approach to remove spatially-varying motion blur from a single photograph, CVPR 12

Sensitivity of deconvolution to blur kernel error Clear image Image blurred by horizontal constant kernel of size 10 pixels

Sensitivity of deconvolution to blur kernel error Clear image Image blurred by horizontal constant kernel of size 10 pixels Image de-blurred by l 1 -norm based regularization, and an erroneous kernel (horizontal constant of size 12 pixels

Convex minimization model Key idea Error induced by kernel error δ K g is sparse Artifacts (ringing artifacts) are sparse in DCT domain Model for robust image deconvolution 1 T 1 ( c, h, u) f K( W c D u) h ( I W W ) c 2 2 2 T 2 2 2 Clear image Artifacts Error induced by kernel W: framelet transform, D: DCT transform [3] Hui Ji and Kang Wang, Robust image de-convolution with an inaccurate blur kernel. IEEE Trans. Image Proc.. 2012

Demo. Blurry image Stationary blind deconvolution Whyte et al. CVPR 10 (nonstationary) Our nonstationary method

Out-of-focus (defocus) blurring Blurring caused by objects away from focal plane More away from focal plane; more blurring Defocus plane Focal plane Lens Image sensor c d d f f 0 Circle of Confusion c d d f f 2 0 d n ( d f ) s f 0

De-focus blurring: usually nonstationary Image usually contains several depth layer Different layer has different blurring De-focus blurring amount f Kg Ordinal scene depth 2 1 r r0 2 Each row of K is a Gaussian kernel pr ( ) exp( ) 2 2 2 ( r0 )

Defocus map estimation A two-stage approach Defocus amount estimation at edge points Completing defocus map by propagation Matting Laplacian method for map completion [8] Completion is done by keeping the defocus amount close to the given ones at edge points, and keeping the discontinuities consistent with that of image edges. [8] Anat, Alex Rav-Acha, and Dani Lischinski. Spectral matting, IEEE PAMI 2008

Estimating defocus blur by rank of local patches [9] Proposion 1. convolution: I=U Consider three matrices U,I,G related by 2D G. Suppose U is positive (negative) definite and G gg. Then, Rank(I)= gˆ, where gˆ is DFT of g. 0 Two observations The rank of a positive/negative definite patch after defocus blurring determines s.t.d. of a Gaussian kernel Rotation can convert a rank-deficit patch to a full-rank patch [9] Gudong Xu, Yuhui Quan, and Hui Ji, Estimating defocus blur through rank of local patches, ICCV 17

Rank-based estimator Sampling symmetric patches in gradient domain along different orientations: Q0[ i, j] I[ i0 p i, j0 p j]: horizontal Q1[ i, j] I[ i0 p j, j0 p i]: vertical Q2[ i, j] I[ i0 i j, j0 p i]: diagonal Q3[ i, j] I[ i0 p i, j0 i j]: anti-diagonal T P Q Q, k 1,2,3,4 k k k Defocus amount estimator ln(1 max rank( P ) / n), n P 0 k 3 k 0

Demonstration Input image defocus map at edges

Demonstration Input image defocus map at edges Complete defocus map

Foreground/background segmentation Input image defocus map at edges Complete defocus map Foreground segmentation

More Input image Bae et al. Tang et al. ours

Evaluation on fore/background segmentation Test defocus dataset from CUHK: 704 images Manually segmented in-focus foreground and out-of-focus background Precision and recall curves of foreground/background segmentation using the defocus maps generated by different methods

Blind defocus deblurring and Recocus Blind defocus de-convolution: recover a all-in-focus image from an image with both in-focus and defocus region

Blind defocus deblurring and Recocus Blind defocus de-convolution: recover a all-in-focus image from an image with both in-focus and defocus region Applications Surveillance Photography Robotics

From an image with defocus to an all-in-focus image Problem formulation f: input image with multiple de-focus regions α i : binary mask of i-th defocus region, i.e. 1 for related pixels and 0 otherwise u i : i-th in-focus region k i : the defocus blur kernel of i-th de-focus region η: noise Unknowns: everything in right sides Output: f L f i ( ki ui ) i 0 L i i 0 u i {, k, u, } i i i i

Removing defocus blurring from image [10] Alternating scheme between blind deconvolution and segmentation { k, u } { } i i i [10] Guodong Xu, Chaoqiang Liu and Hui Ji, Removing partial out-of-focus blur from images. Preprint, 2017

Refining α i, given {k i t, u i t } Observation A region is deblurred by an inaccurate kernel will lead to noticeable ringing artifacts Basic idea: deblurring image with the estimated kernel The pixels with ringing artifacts in estimated defocus region should be in in-focus region The pixels with ringing artifacts in the estimated in-focus region should be in de-focus region Key question? How to detect the pixels with ringing artifacts?

Residual function for detecting ringing artifacts Ringing artifacts cannot be removed by re-blurring image with the same kernel used for deblurring f k 1 g; ( k, f ) k g f g 2 2 2 2 : estimate of g by Wiener filter

Blind defocus deblurring Gaussian is a rough approximation to defocus blur kernel

Blind defocus deblurring Gaussian is a rough approximation to defocus blur kernel Existing parametric form of defocus kernel

Blind defocus deblurring Gaussian is a rough approximation to defocus blur kernel Existing parametric form of defocus kernel Real defocus kernel

Blind defocus deblurring Gaussian is a rough approximation to defocus blur kernel Existing parametric form of defocus kernel Real defocus kernel Observation Low rank priori for regularizing defocus blur kernel Optimization model for blind defocus deblurring min ( u k f ) Wu k r 2 2 k, u F 1 1 2 F k[ r] 1, k[ r] 0, Rank( k) r 0

Demonstration Input Dai et al. Shen et al. ours

Demonstration on image refocus Input

Demonstration on image refocus Input All-in-focus

Demonstration on image refocus Input Image re-focus

List of co-authors Blind deconvolution for removing motion blur Jianfeng Cai, Chaoqiang Liu and Zuowei Shen Non-stationery blind motion deblurring Wang Kang Defocus blurring estimator and image refocus Xu Guodong and Yuhui Quan

Thank You