Centre for Vision, Speech and Signal Processing. University of Surrey. United Kingdom.

Transcription

1 INEXACT PROXIMAL OPERATORS FOR l p -QUASINORM MINIMIZATION Cian O Brien Mark D. Plumbley Centre for Vision, Speech and Signal Processing. University of Surrey. United Kingdom. ABSTRACT Proximal methods are an important tool in signal processing applications, where many problems can be characterized by the minimization of an expression involving a smooth fitting term and a convex regularization term for example the classic l -Lasso. Such problems can be solved using the relevant proximal operator. Here we consider the use of proximal operators for the l p -quasinorm where p. Rather than seek a closed form solution, we develop an iterative algorithm using a Majorization-Minimization procedure which results in an inexact operator. Experiments on image denoising show that for p the algorithm is effective in the high-noise scenario, outperforming the Lasso despite the inexactness of the proximal step. Index Terms Proximal Methods, Compressed Sensing, Sparse Recovery, Majorization-Minimization.. INTRODUCTION Sparse methods have proven to be an effective tool in many signal processing applications ranging from compressed sensing to image denoising. A common aspect of such problems is the requirement for solving an optimization problem, usually involving a smooth fitting term plus a regularization term which encourages some structure on the solution typically sparsity). A popular choice is the l -norm penalty on the solution which encourages the solution vector to be sparse i.e. contain few non-zero entries). The l -norm acts as a convex surrogate to the true sparsity penalty which counts the number of non-zero entries, making the optimization problem tractable. Proximal methods are a class of algorithms which can solve problems involving non-smooth terms such as l regularization []. For the l -norm, this results in the Iterative Shrinkage and Thresholding Algorithm ISTA) []. Relatively less works have focused on the use of general l p -norm when p is less than one []. However, this case is closer to the true sparse penalty and may lead to improved results in sparse problems. A proximal method has been derived for the special cases p {,, } [], which was further developed to the general case by Chen et al []. Reweighted l methods have also been shown to be related to the problem of l p -norm minimization [] [7][8]. In this work we develop a simple algorithm for l p regularized problems using a majorization-maximization technique. By relaxing the need for an exact solution to the proximal problem, we derive an iterative procedure which results in an approximate or inexact proximal operator that works well in practice. Unlike reweighted l approaches, our algorithm is generic and can be used as part of any algorithm requiring the proximal operator such as accelerated) proximal gradient or the Alternating Direction Method of Multipliers ADMM) [9] and works for any p, ).. PROXIMAL METHODS Consider the optimization problem min fx) + λgx) ) x R n where f : R n R is convex and smooth and g : R n R is convex and possibly non-differentiable. Equation ) represents a wide class of problems found in signal processing and machine learning. Generally, f is taken to be a smooth fitting term for example the Euclidean distance) which penalizes the difference between the estimated signal and the desired signal. The second function g often assumes the role of a regularizer which promotes some desired structure in the solution. A common choice for g is the l -norm which penalizes large values in the solution x, or the l -norm which promotes sparsity. The problem of estimating x under an l penalty is known as the Lasso and has seen widespread use in areas such as feature selection, denoising and compressed sensing. Proximal methods are a class of algorithms for solving problems with the form specified by ). Given a convex function g, the proximal operator associated with g is { } prox λg x) arg min u R n u x + λgu). ) The theory of proximal operators includes projection operators onto convex sets and can deal with a wide range of regularization terms. We are interested in proximal operators for the so-called l p -norms, which include the Lasso as a special case p = ): n ) gx) = x p = x i p p. ) i=

2 Fig.. The l p penalty for several values of p left) and their associated approximate shrinkage functions right). The dashed line is the exact l shrinkage function u t) = signu t) ) u t) λ ) Fig.. Difference function F = G λ p,x h λ p,x see Proposition.). Here F obtains its minima at u = ±θ = ±.7 dashed lines) R R h λ p,xu) u x) + λ u p. ) Then since p is separable, Fig.. Majorizing auxiliary functions for h λ p,x with p = left), p =. middle) and p =. right). The black line is the graph of h λ p,x for x =. and λ =.. Generally, the l p -norm is only defined for p. Since it is easier to work without the fraction in the exponent, in the following we take x p to mean ) raised to the p-th power. For p the l p -norm will be convex see for example the left panel of Fig., black line). Given the proximal operator prox λg ) for g, the problem given by equation ) can be solved using proximal gradient descent via the iteration x t+) := prox λg x t) ) η t f x t) which for g ) = p and p = results in the widely used Iterative Shrinkage and Thresholding algorithm.. AN INEXACT PROXIMAL OPERATOR The l -norm is the tightest convex approximation to the true sparsity penalty which counts the number of non-zero entries in a vector and for this reason there exist many algorithms for l minimization. For p the function p fails to be a norm, instead defining a quasinorm. In this case p is no longer convex see left panel in Fig. ). However, for p the corresponding p-quasinorm is a closer approximation to the true sparsity penalty. In this section we propose a method of approximating the solution to ) in the p case using a Majorization- Minimization MM) scheme. We first define a function h λ p,x : ) x = prox λ p x) i, x i = arg min h λ p,x i u) u R ) and in order to solve the proximal problem ), we seek the minimizers of h λ p,x i for i =,..., n... Optimization via Auxiliary Functions Given a function h : R R to be minimized, a majorizing auxiliary function G : R R R is such that hu) = Gu, u) u R hu) Gu, θ) u R, θ R. The idea behind the MM scheme is that minimization of h can be replaced with iterative minization of G via u t+) := arg min u Gu, u t) ). 7) The two conditions above then guarantee a monotone sequence h so long as Gu t+), u t) ) Gu t), u t) ). For the l p -quasinorm proximal operator, we define the following function G λ p,x : R R R Gp,xu, λ θ) u x) + λp θ p u θ) ) + λ signθ)pu θ p p ) θ p). Proposition.. For fixed x R n, p and λ, the function G λ p,xu, θ) defined by 8) is a majorizing auxiliary function for h λ p,xu). Proof. The first condition follows by substituting θ = u in 8) and using the fact that signu)u = u. 8)

3 For the second condition, let Fu, θ) : R R R be the function given by the difference Fu, θ) G λ p,xu, θ) h λ p,xu). 9) First note that F is symmetric in u and θ, i.e. Fu, θ) = F u, θ) = Fu, θ), ) so without loss of generality we can work in the positive halfplane u, + ) by replacing x with x. For positive θ and u we have Fu, θ) = λp )p u p θ u)θ + u) ) and since p, the factor p ) is negative and the overall quantity is positive or negative depending on the sign of the factor θ u). In general: u, θ ) Fu, θ) <, u θ, + ) Fu, θ) >, u = θ Fu, θ) =. ) Therefore on, + ) R and, ) R, the function Fu, θ) is convex and equals zero at its minimum. Furthermore, Fu, θ) is symmetric in u about for all θ R. Having determined G λ p,x we can define an MM scheme by iteratively solving u = arg min θ G λ p,xu, θ) and setting θ = u. Equating the derivative θ G λ p,xu, θ) with zero, we can find the solution θ which globally minimizes G λ p,xu, θ) and hence h λ p,xu) h λ p,xθ ). For x R n and p this gives a non-linear shrinkage function u := xθ pλ θ p + θ ) Applying ) iteratively will monotonically decrease the objective ). Additionally, we need to check the value of the objective function at the extremal point u =. Comparing hu) and h), we find that hu) < h) u u x ) + λ u p <. ) Putting all of this together, we define an inexact or approximate l p -quasinorm proximal operator: [ ] aprox λ p x) i=,...,n { ) u i if u i u := i x i + λ u i p ) < otherwise where u i is found by iterating ). The number of iterations used to find u can be adjusted to balance accuracy and speed. Algorithm describes the full proximal operator calculation, while algorithm demonstrates the use of the operator as part of a proximal gradient procedure to minimize a smooth function with l p -quasinorm regularization. The right panel of Fig shows plots of the shrinkage function ) after several iterations for different values of p. Initial numerical experiments in compressed sensing showed that using the proposed approach with small p values alleviates the bias towards zero seen in l sparse coding. The choice of p introduces a tradeoff between identifiability and bias; smaller values result in less shrinkage of the coefficients, while larger values are better able to recover small signal values. The following result provides a bound in terms of p and λ below which signal components cannot be recovered using l p -minimization. Proposition. Identifiability). Let λ > and p be fixed. A necessary condition for exact recovery of x R n is that for all i {j : x j } in its support where κ := λp p). x i λpκ p p + κ p + λ[ fx ] i ) Proof. A solution x of ) satisfies x = aprox λ p x λ fx ) 7) and each x must be either a fixed point of h λ p,x or else have u = note that u = is not a fixed point in general). For x >, fixed points of h λ p,x occur at the solution to d du hλ p,xu) = λpu p + u x =. 8) This derivative is anti-symmetric and convex resp. concave) for u > resp. u < ). The minimum of this function is achieved at arg min u> { } d du hλ p,xu) Substituting 9) into 8) gives = λp p) ) p = κ p. 9) λpκ p p + κ p x ) and this function has zero-values if and only if x λpκ p p + κ p. ) Therefore, if condition ) is not met for x > then h λ p,x has no fixed points. The inequality in ) follows by taking x = x i λ[ f x ] i. By symmetry, a similar argument holds for x <. Exact recovery depends on this condition being met for every component, which completes the proof.

4 Algorithm Inexact l p Proximal Operator Input: Signal x R n, λ, p [, ] and iteration count T >. Initialize u ) i := for i =,..., n. for t = to T Update each u t+) i according to ). end for for i = to n Update each u T ) i according to ). end for Algorithm Inexact l p Proximal Gradient with Linesearch Input: Objective F = f + λ p, regularizer λ, p [, ], decay term c, ) and tolerance ɛ >. Initialize x :=. while F x t) ) F x t+) ) ɛ x := x t). while f x) fx t) )+ f x x t) )+ x t) λ f x xt) ) x := aprox λ p x t) λ f x t)). λ := cλ. end while x t+) := x. end while. DICTIONARY LEARNING FOR IMAGE DENOISING To validate the proposed algorithm, it was tested as part of a dictionary-based image denoising system. Dictionary learning has proven an effect method for a variety of image processing applications [][][]. Here our goal is not to present state-of-the-art denoising results, but to i) validate the performance of the proposed algorithm and ii) examine the effect of p on the denoising results. Given an input image which has been corrupted by noise, dictionary learning proceeds by learning a collection of dictionary atoms which can be used to reconstruct the original signal. First, the input image is divided into a collection of m patches of size n n. The dictionary learning problem is then given by min D S n d,x i=,...,m m y i Dx i + λ x i p ) i= where D R n d is a matrix whose columns consist of individual dictionary atoms, S n d is the set of all n d matrices whose columns live on the unit sphere in R n and x i=,...,m are the sparse coefficients for the m image patches y i=,...,m arranged as column vectors). After learning, each image patch is reconstructed as Dz i and arranged to recover the full denoised image. Since the dictionary learning problem in nonconvex in all of its parameters jointly, we take the standard p σ =. σ =. MSE PSNR MSE PSNR Lasso Decay Table. Image denoising results for several noise and p- values. alternating update approach; for the sparse coding step, we use the inexact proximal gradient algorithm ) to recover x i for each i {,..., m}. Next, holding the codes fixed, we update the dictionary []) D t+) := P S n d YX t) X t) X t) ) ) ) where P S n d ) : R n d S n d projects the columns of its argument into S n d. This process is then iterated until convergence. For the input signal we used the Barbara test image corrupted with varying amounts of Gaussian noise with variance σ {.,.} and a patch size of 8 8. The dictionary size was set to atoms, initialized with exemplars from the training set. We used p {.,.7} as well as the Lasso p = ), which was solved using ISTA. Additionally, we test a continuation strategy where p was initialized to and decayed at each iteration until a final value of. last row in Table ). The results are summarized in Table where we record the performance in terms of Peak Signal to Noise Ratio PSNR) and Mean Squared Error MSE). We see that values of p < lead to improved performance, particularly in the high noise scenario.. CONCLUSION We have presented an algorithm for solving l p -norm regularized problems in signal processing. The proposed approach is generic, in the sense that it can be used as part of any algorithm requiring the l p proximal operator. We find that values of p < result in less bias in the solution. The approach was validated on a small dictionary learning and image denoising experiment, where we find that tuning the value of p can improve the performance over the l case.. ACKNOWLEDGEMENT The research leading to these results has received funding from the European Union s Seventh Framework Programme FP7-PEOPLE--ITN) under Grant Agreement no. 79 SpaRTaN.

5 7. REFERENCES [] Neal Parikh, Stephen Boyd, et al., Proximal algorithms, Foundations and Trends in Optimization, vol., no., pp. 7 9,. [] Ingrid Daubechies, Michel Defrise, and Christine De Mol, An iterative thresholding algorithm for linear inverse problems with a sparsity constraint, Communications on Pure and Applied Mathematics, vol. 7, no., pp. 7,. [] Rick Chartrand and Valentina Staneva, Restricted isometry properties and nonconvex compressive sensing, Inverse Problems, vol., no., pp., 8. [] Michal Aharon, Michael Elad, and Alfred Bruckstein, k-svd: An algorithm for designing overcomplete dictionaries for sparse representation, IEEE Transactions on signal processing, vol., no., pp.,. [] Kjersti Engan, Sven Ole Aase, and J Hakon Husoy, Method of optimal directions for frame design, in Acoustics, Speech, and Signal Processing, 999. Proceedings., 999 IEEE International Conference on. IEEE, 999, vol., pp.. [] Thomas Blumensath, Mehrdad Yaghoobi, and Mike E Davies, Iterative hard thresholding and l regularisation, in Acoustics, Speech and Signal Processing, 7. ICASSP 7. IEEE International Conference on. IEEE, 7, vol., pp. III 877. [] Feishe Chen, Lixin Shen, and Bruce W Suter, Computing the proximity operator of the lp norm with < p <, IET Signal Processing, vol., no., pp. 7,. [] Emmanuel J Candes, Michael B Wakin, and Stephen P Boyd, Enhancing sparsity by reweighted l minimization, Journal of Fourier analysis and applications, vol., no., pp , 8. [7] Feishe Chen, Composite Minimization: Proximity Algorithms and Their Applications, Ph.D. thesis, Syracuse University,. [8] Ming-Jun Lai, Yangyang Xu, and Wotao Yin, Improved iteratively reweighted least squares for unconstrained smoothed \ell q minimization, SIAM Journal on Numerical Analysis, vol., no., pp ,. [9] Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, and Jonathan Eckstein, Distributed optimization and statistical learning via the alternating direction method of multipliers, Foundations and Trends in Machine Learning, vol., no., pp.,. [] Ling Shao, Ruomei Yan, Xuelong Li, and Yan Liu, From heuristic optimization to dictionary learning: A review and comprehensive comparison of image denoising algorithms, IEEE Transactions on Cybernetics, vol., no. 7, pp.,. [] Ruomei Yan, Ling Shao, and Yan Liu, Nonlocal hierarchical dictionary learning using wavelets for image denoising, IEEE transactions on image processing, vol., no., pp ,.