Total Variation Blind Deconvolution: The Devil is in the Details*

Size: px

Start display at page:

Download "Total Variation Blind Deconvolution: The Devil is in the Details*"

Tobias Wright
5 years ago
Views:

1 Total Variation Blind Deconvolution: The Devil is in the Details* Paolo Favaro Computer Vision Group University of Bern *Joint work with Daniele Perrone

2 Blur in pictures When we take a picture we expose the sensor of our camera to the incoming light through the lens The lens needs to be placed at the right distance between the scene and the sensor, otherwise

3 Out of focus blur

4 Blur in pictures When we take a picture we expose the sensor of our camera to the incoming light through the lens The camera or the scene should not move during the exposure otherwise

5 Motion Blur

6 A blur model When the captured image is blurry then we have no choice but to try and remove the degradation computationally The first step is to model blur degradation f = k u + n blurry image kernel sharp image noise = * +

7 Deblurring When the kernel k is known then we are essentially inverting a linear system Deblurring can be posed as a convex optimization problem min u kuk BV kf k uk2 2

8 Kernel k is known: Deblurring

9 Kernel k is known: Deblurring

10 Blind deconvolution Neither the kernel nor the sharp image are known We need to recover both the blur and the sharp image min u,k kuk BV kf k uk2 2 The problem is non convex

11 Blind deconvolution Neither the kernel nor the sharp image are known We need to recover both the blur and the sharp image min u,k kuk BV kf k uk2 2 The problem is non convex

12 Prior Work Before the general belief was that blind deconvolution was not just impossible, but that it was hopelessly impossible How can we extract more data than we observe?

13 Ambiguities The main difficulty in solving blind deconvolution is that the problem is ill-posed

14 Ambiguities The main difficulty in solving blind deconvolution is that the problem is ill-posed For example, if (u,k) is a solution, then also (au,k/a) and (u(x+d),k(x-d)) for any d and for any a>0 are solutions

15 Ambiguities The main difficulty in solving blind deconvolution is that the problem is ill-posed For example, if (u,k) is a solution, then also (au,k/a) and (u(x+d),k(x-d)) for any d and for any a>0 are solutions Consider the Fourier transform: F = KU, where F, K and U are Fourier transforms of f, k, and u respectively

16 Ambiguities The main difficulty in solving blind deconvolution is that the problem is ill-posed For example, if (u,k) is a solution, then also (au,k/a) and (u(x+d),k(x-d)) for any d and for any a>0 are solutions Consider the Fourier transform: F = KU, where F, K and U are Fourier transforms of f, k, and u respectively Then, for any K that is not 0 at any frequency there exists always a U such that F = KU (simply let U = F/K)

17 The role of the image prior To reduce the set of ambiguities to a unique sensible answer one can use a regularization term One of the first regularization terms proposed in blind deconvolution was the H 1 prior (You and Kaveh 1996) kruk 2 2 Total Variation (strongly related to sparse gradient and natural image prior) was also proposed at the same time (You and Kaveh 1996) kruk 2

18 Chan and Wong (1998) Total Variation Blind Deconvolution (similar work appeared earlier in You and Kaveh, 1996) Solve min u,k kuk BV kf k uk2 2 Use an alternating minimization algorithm (fix the blur and compute the sharp image, then fix the sharp image and compute the blur)

19 Chan and Wong (1998) it works! sharp image out of focus restored image Gaussian blur restored image

20 Fergus et al (2006) Alternating minimization does not work (MAPu,k)

21 Fergus et al (2006) Alternating minimization does not work (MAPu,k)

22 Fergus et al (2006) Alternating minimization does not work (MAP u,k ) Use instead a MAP k approach (based on Miskin and McKay 2000) Marginalize wrt a distribution of the sharp images Compute k by maximizing the marginalized dist. Compute u by solving deblurring given k Technical details: Use a variational bayesian approach (Jordan et al 1999) and a Gaussian mixture model

23 Fergus et al (2006) motion blurred restored motion blurred restored

24 Shan et al (2008) Impose that noise is iid Use alternating minimization (MAPu,k) but on the image gradients Impose that sharp image and blurry image should coincide where the blurry image is very smooth Then estimate sharp image given kernel k

25 Shan et al (2008) motion blurred iteration 1 iteration 6 iteration 10

26 Cho and Lee (2009) Success of prior work is: Sharp edge restoration and noise suppression in smooth regions Blur can be estimated reliably at edges Try and predict edges with a shock filter Use a modified alternating minimization (MAP u,k )

27 Cho and Lee (2009) motion blurred restored

28 Xu et al (2013) Use a saturated L1 prior (they call it unnatural L0) Use alternating minimization (MAPu,k) Technical details: Many intermediate steps

29 Xu et al (2013)

30 Levin et al (2011) Stop using MAPu,k! It should not work! Use MAPk Compare the following true solution (u,k) with the no-blur solution (f, ) f f k u Then, solution is based only on the image prior; however, the prior favors the no-blur solution! krfk 2 apple kruk 2

31 Levin et al (2011)

32 MAPk After marginalization Levin et al obtain which is an alternating minimization weights are updated sequentially

33 A conundrum On the one hand many MAPu,k implementations and (heuristic) variants work very well, and on the other hand they are not supposed to work at all

34 A conundrum On the one hand many MAPu,k implementations and (heuristic) variants work very well, and on the other hand they are not supposed to work at all Rather than developing yet another blind deconvolution algorithm, should we not try to understand what is going on first?

35 A conundrum On the one hand many MAPu,k implementations and (heuristic) variants work very well, and on the other hand they are not supposed to work at all Rather than developing yet another blind deconvolution algorithm, should we not try to understand what is going on first? Could MAPk be just another recipe for MAPu,k?

36 Recent analysis Wipf and Zhang arxiv 2013: MAP k equivalent to a MAP u,k MAPk MAPu,k See also Babacan et al and Krishnan et al. 2014

37 Recent analysis So, current conclusion is that it is not about MAPk vs MAPu,k, but rather about the choice of priors Still, this does not explain why current so-called MAPu,k approaches (that use TV-like priors) work

38 Removing the bells and whistles We start by applying the golden rule in analysis: Remove all the unnecessary Result: Total Variation Blind Deconvolution (1996!) min u,k J(u)+kk u fk 2 2 subject to k < 0, kkk 1 =1

39 Attempt #1: Exact solution The alternating minimization (AM) algorithm Actually, it does not work!

40 AM does not work

41 A toy example in 1D Let us consider a 1D signal (a hat function) and a 1D blur of 3 pixels

42 A toy example in 1D Let us consider a 1D signal (a hat function) and a 1D blur of 3 pixels Because the blur components add to 1, we only have 2 free parameters

43 A toy example in 1D Let us consider a 1D signal (a hat function) and a 1D blur of 3 pixels Because the blur components add to 1, we only have 2 free parameters For each possible combination of these parameters we minimize the TV problem wrt the sharp image (a deblurring problem)

44 A toy example in 1D Let us consider a 1D signal (a hat function) and a 1D blur of 3 pixels Because the blur components add to 1, we only have 2 free parameters For each possible combination of these parameters we minimize the TV problem wrt the sharp image (a deblurring problem) We show the energy at the minimum wrt the sharp image for each possible blur

45 A toy example in 1D k[2] k[1]

46 A toy example in 1D k[2] k[1] true minimum 1

47 A toy example in 1D k[2] initial solution k[1] true minimum 1

48 A toy example in 1D k[2] initial solution k[1] true minimum 1 value of energy at no-blur solutions is lower than at the true minimum

49 Attempt #2: Approximate solution Projected alternating minimization (PAM) implementation of Chang and Wong (1998) It works!

50 Where s Wally? What is the difference between AM and PAM that makes PAM work? Why does it make it work?

51 Comparison of AM and PAM The first step (image deblurring) is identical The second step separates the normalization and the positivity constraints from the minimization step

52 A gradient descent?? k[2] initial solution k[1] 0.6 final solution =

Normalization is the key k[2] 1 2 k[2] 1 2 k[2] 1 2 1 1

53 Normalization is the key k[2] 1 2 k[2] 1 2 k[2] k[1] k[1] k[1] 2 = kkk 1 =1 kkk 1 =1.5 kkk 1 =2.5

54 Normalization is the key k[2] 1 2 k[2] 1 2 k[2] k[1] k[1] k[1] 2 2 =1.5 2 kkk 1 =1 kkk 1 =1.5 kkk 1 =2.5

55 Normalization is the key k[2] 1 2 k[2] 1 2 k[2] k[1] k[1] k[1] =2.5 kkk 1 =1 kkk 1 =1.5 kkk 1 =2.5

56 AM on a step function 0.5 Blurred signal Sharp Signal TV Signal Blurred TV Signal f[x] Engineering 5 Science University 0 of Oxford5 10 x

57 AM on a step function 0.5 Blurred signal Sharp Signal TV Signal Blurred TV Signal f[x] Engineering 5 Science University 0 of Oxford5 10 x

58 AM on a step function 0.5 Blurred signal Sharp Signal TV Signal Blurred TV Signal f[x] Engineering 5 Science University 0 of Oxford5 10 x

59 AM on a step function 0.5 Blurred signal Sharp Signal TV Signal Blurred TV Signal f[x] Engineering 5 Science University 0 of Oxford5 10 x

60 AM on a step function 0.5 Blurred signal Sharp Signal TV Signal Blurred TV Signal f[x] 0 no-blur error Engineering 5 Science University 0 of Oxford5 10 x

61 AM on a step function 0.5 Blurred signal Sharp Signal TV Signal Blurred TV Signal f[x] Engineering 5 Science University 0 of Oxford5 10 x

62 AM on a step function 0.5 Blurred signal Sharp Signal TV Signal Blurred TV Signal f[x] 0 additional true-blur error Engineering 5 Science University 0 of Oxford5 10 x

63 PAM on a step function 0.5 Blurred signal Sharp Signal Scaled TV Signal TV Signal f[x] Engineering 5 Science University 0 of Oxford 5 10 x

64 PAM on a step function 0.5 Blurred signal Sharp Signal Scaled TV Signal f[x] Engineering 5 Science University 0 of Oxford5 10 x

65 PAM on a step function 0.5 Blurred signal Sharp Signal Scaled TV Signal f[x] 0 Detailed proofs of convergence of PAM in CVPR Engineering 5 Science University 0 of Oxford5 10 x

66 Technical details As in most current implementations we used a pyramid scheme Adaptation of the regularization parameter is needed Boundary conditions: None as we use the exact blur model

67 The PAM algorithm

68 Experiments our Levin Cho Fergus 2 Engineering 3 Science University of 4Oxford 5 error ratio

69 Blurry image

70 Cho and Lee (2009)

71 Fergus et al (2006)

72 Hirsch et al (2011)

73 Shan et al (2008)

74 Whyte et al (2011)

75 Xu and Jia (2010)

76 Our (PAM)

77 Blurred

78 Xu and Jia (2010)

79 Our (PAM)

80 One more example blurry Cho and Lee (2009) Goldstein and Fattal (2012)

81 One more example our (PAM) Zhong et al (2013) Levin et al (2011) be wary of the results of others!

82 One more example our (PAM) Zhong et al (2013) Levin et al (2011)

83 Conclusions We have shown (with theory and experiments) why many alternating minimization algorithms work The reason lies in the normalization of blur (scaling) + regularization parameter This 1998 algorithm competes very well with recent more sophisticated algorithms Perhaps we should rethink our formulation of blind deconvolution?

Recent Advances in Image Deblurring. Seungyong Lee (Collaboration w/ Sunghyun Cho)

Recent Advances in Image Deblurring Seungyong Lee (Collaboration w/ Sunghyun Cho) Disclaimer Many images and figures in this course note have been copied from the papers and presentation materials of previous