ONE of the most challenging experiences in photography. Removing Camera Shake via Weighted Fourier Burst Accumulation

Transcription

1 Removing Camera Shake via Weighted Fourier Burst Accumulation Mauricio Delbracio and Guillermo Sairo arxiv:55.v [cs.cv] Dec 5 Abstract Numerous recent aroaches attemt to remove image blur due to camera shake, either with one or multile inut images, by exlicitly solving an inverse and inherently illosed deconvolution roblem. If the hotograher takes a burst of images, a modality available in virtually all modern digital cameras, we show that it is ossible to combine them to get a clean shar version. This is done without exlicitly solving any blur estimation and subsequent inverse roblem. The roosed algorithm is strikingly simle: it erforms a weighted average in the Fourier domain, with weights deending on the Fourier sectrum magnitude. The method can be seen as a generalization of the align and average rocedure, with a weighted average, motivated by hand-shake hysiology and theoretically suorted, taking lace in the Fourier domain. The method s rationale is that camera shake has a random nature and therefore each image in the burst is generally blurred differently. Exeriments with real camera data, and extensive comarisons, show that the roosed Fourier Burst Accumulation (FBA) algorithm achieves state-of-the-art results an order of magnitude faster, with simlicity for on-board imlementation on camera hones. Finally, we also resent exeriments in real high dynamic range (HDR) scenes, showing how the method can be straightforwardly extended to HDR hotograhy. Index Terms multi-image deblurring, burst fusion, camera shake, low light hotograhy, high dynamic range. I. INTRODUCTION ONE of the most challenging exeriences in hotograhy is taking images in low-light environments. The basic rincile of hotograhy is the accumulation of hotons in the sensor during a given exosure time. In general, the more hotons reach the surface the better the quality of the final image, as the hotonic noise is reduced. However, this basic rincile requires the hotograhed scene to be static and that there is no relative motion between the camera and the scene. Otherwise, the hotons will be accumulated in neighboring ixels, generating a loss of sharness (blur). This roblem is significant when shooting with hand-held cameras, the most oular hotograhy device today, in dim light conditions. Under reasonable hyotheses, the camera shake can be modeled mathematically as a convolution, v = u k + n, () where v is the noisy blurred observation, u is the latent shar image, k is an unknown blurring kernel and n is additive white noise. For this model to be accurate, the camera movement has This work was artially funded by: ONR, ARO, NSF, NGA, and AFOSR. The authors are with the Deartment of Electrical and Comuter Engineering at Duke University. {mauricio.delbracio, guillermo.sairo}@duke.edu to be essentially a rotation in its otical axis with negligible in-lane rotation, e.g., []. The kernel k results from several blur sources: light diffraction due to the finite aerture, out-offocus, light integration in the hoto-sensor, and relative motion between the camera and the scene during the exosure. To get enough hotons er ixel in a tyical low light scene, the camera needs to cature light for a eriod of tens to hundreds of milliseconds. In such a situation (and assuming that the scene is static and the user/camera has correctly set the focus), the dominant contribution to the blur kernel is the camera shake mostly due to hand tremors. Current cameras can take a burst of images, this being oular also in camera hones. This has been exloited in several aroaches for accumulating hotons in the different images and then forming an image with less noise (mimicking a longer exosure time a osteriori, see e.g., []). However, this rincile is disturbed if the images in the burst are blurred. The classical mathematical formulation as a multiimage deconvolution, seeks to solve an inverse roblem where the unknowns are the multile blurring oerators and the underlying shar image. This rocedure, although it roduces good results [], is comutationally very exensive, and very sensitive to a good estimation of the blurring kernels, an extremely challenging task by itself. Furthermore, since the inverse roblem is ill-osed it relies on riors either, or both, for the calculus of the blurs and the latent shar image. Camera shake originated from hand tremor vibrations is essentially random [], [5], [6]. This imlies that the movement of the camera in an individual image of the burst is indeendent of the movement in another one. Thus, the blur in one frame will be different from the one in another image of the burst. Our work is built on this basic rincile. We resent an algorithm that aggregates a burst of images (or more than one burst for high dynamic range), taking what is less blurred of each frame to build an image that is sharer and less noisy than all the images in the burst. The algorithm is straightforward to imlement and concetually simle. It takes as inut a series of registered images and comutes a weighted average of the Fourier coefficients of the images in the burst. Similar ideas have been exlored by Garrel et al. [] in the context of astronomical images, where a shar clean image is roduced from a video affected by atmosheric turbulence blur. With the availability of accurate gyroscoe and accelerometers in, for examle, hone cameras, the registration can be obtained for free, rendering the whole algorithm very efficient for on-board imlementation. Indeed, one could envision a mode transarent to the user, where every time he/she takes

2 a icture, it is actually a burst or multile bursts with different arameters each. The set is then rocessed on the fly and only the result is saved. Related modes are currently available in ermanent oen cameras. The exlicit comutation of the blurring kernel, as commonly done in the literature, is comletely avoided. This is not only an unimortant hidden variable for the task at hand, but as mentioned above, still leaves the ill-osed and comutationally very exensive task of solving the inverse roblem. Evaluation through synthetic and real exeriments shows that the final image quality is significantly imroved. This is done without exlicitly erforming deconvolution, which generally introduces artifacts and also a significant overhead. Comarison to state-of-the-art multi-image deconvolution algorithms shows that our aroach roduces similar or better results while being orders of magnitude faster and simler. The roosed algorithm does not assume any rior on the latent image; exclusively relying on the randomness of hand tremor. A reliminary short version of this work was submitted to a conference [8]. The resent version incororates a more detailed analysis of the burst aggregation algorithm and its imlementation. We also introduce a detailed comarison to lucky image selection techniques [9], where from a series of short exosure images, the sharest ones are selected, aligned and averaged into a single frame. Additionally, we resent exeriments in real high dynamic range (HDR) scenes showing how the method can be extended to hand-held HDR hotograhy. The remaining of the aer is organized as follows. Section II discusses the related work and the substantial differences to what we roose. Section III exlains how a burst can be combined in the Fourier domain to recover a sharer image, while in Section IV we resent an emirical analysis of the algorithm erformance through the simulation of camera shake kernels. Section V details the algorithm imlementation and in Section VI we resent results of the roosed aggregation rocedure in real data, comaring the algorithm to state-of-theart multi-image deconvolution methods. Section VI-D resents exeriments in real high dynamic range (HDR) scenes, showing how the method can be straightforwardly extended to HDR hotograhy. Conclusions are finally summarized in Section VII. II. RELATED WORK Removing camera shake blur is one of the most challenging roblems in image rocessing. Although in the last decade several image restoration algorithms have emerged giving outstanding erformance, their success is still very deendent on the scene. Most image deblurring algorithms cast the roblem as a deconvolution with either a known (non-blind) or an unknown blurring kernel (blind). See e.g., the review by Kundur and Hatzinakos [], where a discussion of the most classical methods is resented. Single image blind deconvolution. Most blind deconvolution algorithms try to estimate the latent image without any other inut than the noisy blurred image itself. A reresentative work is the one by Fergus et al. []. This variational method sarked many cometitors seeking to combine natural image riors, assumtions on the blurring oerator, and comlex otimization frameworks, to simultaneously estimate both the blurring kernel and the shar image [], [], [], [5], [6]. Fergus et al. [] aroximated the heavy-tailed distribution of the gradient of natural images using a Gaussian mixture. In [], the authors exloited the use of sarse riors for both the shar image and the blurring kernel. Cai et al. [] roosed a joint otimization framework, that simultaneously maximizes the sarsity of the blur kernel and the shar image in a curvelet and a framelet systems resectively. Krishnan et al. [] introduced as a rior the ratio between the l and the l norms on the high frequencies of an image. This normalized sarsity measure gives low cost for the shar image. In [5] the authors discussed unnatural sarse reresentations of the image that mainly retain edge information. This reresentation is used to estimate the motion kernel. Michaeli and Irani [6] recently roosed to use as an image rior the recurrence of small natural image atches across different scales. The idea is that the cross-scale atch occurrence should be maximal for shar images. Several attemt to first estimate the degradation oerator and then alying a non-blind deconvolution algorithm. For instance, [] accelerates the kernel estimation ste by using fast image filters for exlicitly detecting and restoring strong edges in the latent shar image. Since the blurring kernel has tyically a very small suort, the kernel estimation roblem is better conditioned than estimating the kernel and the shar image together. In [8], [9], the authors concluded that it is better to first solve a maximum a osteriori estimation of the kernel than of the latent image and the kernel simultaneously. However, even in non-blind deblurring, i.e., when the blurring kernels are known, the roblem is generally ill-osed, because the blur introduces zeros in the frequency domain. Thereby avoiding exlicit inversion, as here roosed, becomes critical. Multi-image blind deconvolution. Two or more inut images can imrove the estimation of both the underlying image and the blurring kernels. Rav-Acha and Peleg [] claimed that Two motion-blurred images are better than one, if the direction of the blurs are different. In [] the authors roosed to cature two hotograhs: one having a short exosure time, noisy but shar, and one with a long exosure, blurred but with low noise. The two acquisitions are comlementary, and the shar one is used to estimate the motion kernel of the blurred one. Close to our work are aers on multi-image blind deconvolution [], [], [], [], [5]. In [] the authors showed that given multile observations, the sarsity of the image under a tight frame is a good measurement of the clearness of the recovered image. Having multile inut images imroves the accuracy of identifying the motion blur kernels, reducing the illosedness of the roblem. Most of these multi-image algorithms introduce cross-blur enalty functions between image airs. However this has the roblem of growing combinatorially with the number of images in the burst. This idea is extended in [] using a Bayesian framework

3 for couling all the unknown blurring kernels and the latent image in a unique rior. Although this rior has numerous good mathematical roerties, its otimization is very slow. The algorithm roduces very good results but it may take several minutes or even hours for a tyical burst of 8- images of several megaixels. The very recent work by Park and Levoy [6] relies on an attached gyroscoe, now resent in many hones and tablets, to align all the inut images and to get an estimation of the blurring kernels. Then, a multi-image non-blind deconvolution algorithm is alied. By taking a burst of images, the multi-image deconvolution roblem becomes less ill-osed allowing the use of simler riors. This is exlored in [] where the authors adoted a total variation rior on the underlying shar image. All these aers roose kernel estimation and to solve an inverse roblem of image deconvolution. The main inconvenience of tackling this roblem as a deconvolution, on to of the comutational burden, is that if the convolution model is not accurate or the kernel is not accurately estimated, the restored image will contain strong artifacts (such as ringing). Lucky imaging. A oular technique in astronomical hotograhy, known as lucky imaging or lucky exosures, is to take a series of thousands of short-exosure images and then select and fuse only the sharer ones [8]. Fried [9] showed that the robability of getting a shar lucky short-exosure image through turbulence follows a negative exonential. Thus, when the catured series or video is sufficiently long, there will exist such a frame with high robability. Classical selection techniques are based on the brightness of the brightest seckle [8]. The number of selected frames is chosen to otimize the tradeoff between sharness and signalto-noise ratio required in the alication. Others roose to measure the local sharness from the norm of the gradient or the image Lalacian [9], [], [], []. Joshi and Cohen [] engineered a weighting scheme to balance noise reduction and sharness reservation. The sharness is measured through the intensity of the image Lalacian. They also roosed a local selectivity weight to reflect the fact that more averaging should be done in smooth regions. Haro and colleagues [] exlored similar ideas to fusion different acquisitions of ainting images. The weights for combining the inut images rely on a local sharness measure based on the energy of the image gradient. The main disadvantage of these aroaches is that they only rely on sharness measures (which by the way is not necessarily trivial to estimate) and do not rofit the fact that camera shake blur can be in different directions in different frames. Garrel et al. [] introduced a selection scheme for astronomic images, based on the relative strength of signal for each satial frequency in the Fourier domain. From a series of realistic image simulations, the authors showed that this rocedure roduces images of higher resolution and better signal to noise ratio than traditional lucky image fusion schemes. This rocedure makes a much more efficient use of the information contained in each frame. Our aer is based on similar ideas but in a different Fig.. When the camera is set to a burst mode, several hotograhs are catured sequentially. Due to the random nature of hand tremor, the camera shake blur is mostly indeendent from one frame to the other. An image consisting of white dots was hotograhed with a DSLR handheld camera to deict the camera motion kernels. The kernels are mainly unidimensional regular trajectories that are not comletely random (erfect random walk) nor uniform. scenario. The idea is to fuse all the images in the burst without exlicitly estimating the blurring kernels and subsequent inverse roblem aroach, but taking the information that is less degraded from each image in the burst. The estimation of the less degraded information is done in a trivial fashion as exlained next. The entire algorithm is based on hysical roerties of the camera (hand) shake and not on riors or assumtions on the image and/or kernel. III. F OURIER B URST ACCUMULATION A. Rationale Camera shake originated from hand tremor vibrations has undoubtedly a random nature [], [5], [6]. The indeendent movement of the hotograher hand causes the camera to be ushed randomly and unredictably, generating blurriness in the catured image. Figure shows several hotograhs taken with a digital single-lens reflex (DSLR) handheld camera. The hotograhed scene consists of a lato dislaying a black image with white dots. The catured icture of the white dots illustrates the trace of the camera movement in the image lane. If the dots are very small mimicking Dirac masses their hotograhs reresent the blurring kernels themselves. As one can see, the kernels mostly consist of unidimensional regular random trajectories. This stochastic behavior will be the key ingredient in our roosed aroach. Let F denote the Fourier Transform and k the Fourier Transform of the kernel k. Images are defined in a regular grid indexed by the D osition x and the Fourier domain is indexed by the D frequency ζ. Let us assume, without loss of generality,rthat the kernel k due to camera shake is normalized such that k(x)dx =. The blurring kernel is nonnegative since the integration of incoherent light is always nonnegative. This imlies that the motion blur does not amlify the Fourier sectrum: R Claim. Let k(x) and k(x) =. Then, k (ζ), ζ. (Blurring kernels do not amlify the sectrum.) Proof. Z k (ζ) = k(x)eix ζ dx Z Z k(x) dx = k(x)dx =.

4 inut inut inut inut 5 inut 6 inut inut 8 inut 9 inut inut inut inut inut = = = = 5 k v inut real(k ) imag(k ) Fig.. Real camera shake kernels were comuted using a shar snashot catured with a triod as a reference. The first row shows a cro of each inut image (frames to ) and the roosed Fourier Burst Accumulation (from (), no additional sharening). As noted before, the kernels are generally regular unidimensional trajectories (second row). The four last columns in the second row show the resultant oint sread functions (PSF) after the Fourier weighted average for different values of. The kernel due to the Fourier average with > is closer to a Delta function, showing the success of the method. The two bottom rows show the Fourier real and imaginary arts of each blurring kernel (the red box indicates the π/ frequency). The real art is mostly ositive and significantly larger than the imaginary art, imlying that the blurring kernels do not introduce significant hase distortions. This might not be the case for large motion kernels, uncommon in standard hand shakes. Most modern digital cameras have a burst mode where the hotograher is allowed to sequently take a series of images, one right after the other. Let us assume that the hotograher takes a sequence of M images of the same scene u, vi = u? k i + n i, for i =,..., M. () The movement of the camera during any two images of the burst will be essentially indeendent. Thus, the blurring kernels ki will be mostly different for different images in the burst. Hence, each Fourier frequency of u will be differently affected on each frame of the burst. The idea is to reconstruct an image whose Fourier sectrum takes for each frequency the value having the largest Fourier magnitude in the burst. Since a blurring kernel does not amlify the Fourier sectrum (Claim ), the reconstructed image icks what is less attenuated, in Fourier domain, from each image of the burst. Choosing the least attenuated frequencies does not necessarily guarantee that those frequencies are the least affected by the blurring kernel, as the kernel may introduce changes in the Fourier image hase. However, for small motion kernels, the hase distortion introduced is small. This is illustrated in Figure, where several real motion kernels and their Fourier sectra are shown. B. Fourier Magnitude Weights Let be a non-negative integer, we will call Fourier Burst Accumulation (FBA) to the Fourier weighted averaged image,! M X u (x) = F wi (ζ) v i (ζ) (x), () i= v i (ζ) wi (ζ) = PM, j= v j (ζ) where v i is the Fourier Transform of the individual burst image vi. The weight wi := wi (ζ) controls the contribution of the frequency ζ of image vi to the final reconstruction u. Given ζ, for >, the larger the value of v i (ζ), the more v i (ζ) contributes to the average, reflecting what we discussed above that the strongest frequency values reresent the least attenuated u comonents. The integer controls the aggregation of the images in the Fourier domain. If =, the restored image is just the arithmetic average of the burst (as standard for examle in the case of noise only), while if, each reconstructed frequency takes the maximum value of that frequency along the burst. This is stated in the following claim; the roof is straightforward and it is therefore omitted. Claim. Mean/Max aggregation. Suose that v i (ζ) for i =,..., M are such that v i (ζ) = v i (ζ) =... = v iq (ζ) > v iq+ (ζ) v iq+ (ζ)... v im (ζ) and wi (ζ) is given by (). If =, then wi (ζ) = M, i (arithmetic mean ooling), while if, then wi (ζ) = q for i = i,..., iq and wi (ζ) = otherwise (maximum ooling). The Fourier weights only deend on the Fourier magnitude and hence they are not sensitive to otential image misalignment. However, when doing the average in (), the images vi have to be correctly aligned to mitigate Fourier hase intermingling and get a shar aggregation. The images in our exeriments are aligned using SIFT corresondences and then finding the dominant homograhy between each image in the burst and the first one (imlementation details are given in Section V). This re-alignment ste can be done exloiting the camera gyroscoe and accelerometer data. Dealing with noise: The images in the burst are blurry but also contaminated with noise. In the ideal case where the inut images are not contaminated with noise, () is reduced to v i k i u k i wi = PM = PM = PM, j= v j j= k j u j= k j () as long as u >. This is what we would like to have: a rocedure for weighting more the frequencies that are less attenuated by the different camera shake kernels. Since camera shake kernels have tyically a small suort, of maximum only a few tenths of ixels, their Fourier sectra magnitude vary smoothly. Thus, v i can be smoothed out before comuting the weights. This hels to remove noise and also to stabilize the weights (see Section V).

5 5 C. Equivalent Point Sread Function The aggregation rocedure can be seen as the convolution of the underlying shar image u with an average kernel kfba given by the Fourier weights in (), u = u? kfba + n, (5) where kfba (x) = F M X (a) Frames cro - and the Fourier weights wi for =.! wi (ζ) k i (ζ) (x), (6) i= and n is the weighted average of the inut noise. The FBA kernel can be seen as the final oint sread function (PSF) obtained by the aggregation rocedure (assuming a erfect alignment). The closer the FBA kernel is to a Dirac function, the better the Fourier aggregation works. By construction, the average kernel is made from the least attenuated frequencies in the burst given by the Fourier weights. However, since arbitrary convolution kernels may also introduce hase distortion, there is no guarantee in general that this aggregation rocedure will lead to an equivalent PSF that is closer to a Dirac mass. In Figure we show a series of inut images and the resective motion kernels. The motion kernels were estimated using a shar snashot, catured with a triod. Using the shar reference uref we solve for a blurring kernel ki minimizing the least squares distance to the blurred acquisition vi, namely, uref? ki vi (see e.g., [] for a similar setu). The figure also shows the real and imaginary arts of the Fourier kernels sectra. As one can see for most of the kernels the real art is mostly ositive and considerably larger than the imaginary art. This imlies that the less attenuated frequencies will not resent significant hase distortion. This assumtion may not be accurate for large motion kernels, an unexected scenario in ordinary camera shake. In this examle, as increases the equivalent oint sread function gets closer to a Dirac function. In articular, the FBA kernel for > attenuates significantly less the high frequencies than the regular arithmetic average ( = ), leading to a sharer aggregation. = = = = = = 5 (b) Fourier Aggregation results for different values..5.5 = = = = = = Frame 5 6 (c) Weights Renergy distribution: wei = wi (ζ) dζ = = = = = =5..5 Frame 5 6 (d) WeightedRframes energy distribution: wfei = wi (ζ) v i (ζ) dζ Fig.. Weights distribution of the Fourier Burst Aggregation when changing the value of. As increases, the weights are concentrated in fewer images and the aggregated image becomes sharer but also noisier. The difference between (c) and (d) lies in the fact that a large weight may not necessarily reflect a large contribution to the final image, although this is generally the case. To make exlicit the contribution of each frame to the final image, the FBA weighted average () can be decomosed into its contributions: M M X X F wi (ζ) v i (ζ) (x) := v i (x). u (x) = i= i= Each term v i is the result of alying the corresonding Fourier weights wi (a filter) to the resective inut frame vi. Figure shows each of these terms for a cro of a real burst. As the reader may notice, each frame contributes differently. None of the frames cature all the structure resent in the final aggregated image. IV. F OURIER B URST ACCUMULATION A NALYSIS A. Anatomy of the Fourier Accumulation B. Statistical Performance Analysis The value of sets a tradeoff between sharness and noise reduction. Although one would always refer to get a shar image, due to noise and the unknown Fourier hase shift introduced by the aggregation rocedure, the resultant image would not necessary be better as. Figure shows an examle of the roosed FBA for a burst of images, and the amount of contribution of each frame to the final aggregation. As grows, the weights are concentrated in fewer images (Figure c) and d)). Also, the weights mas clearly show that different Fourier frequencies are recovered from different frames (Figure a)). In this examle, the high frequency content is uniformly taken from all the frames in the burst. This roduces a strong noise reduction behavior, in addition to the sharening effect. To show the statistical erformance of the Fourier weighted accumulation, we carried out an emirical analysis alying the roosed aggregation with different values of. We simulated motion kernels following [5], where the (exected value) amount of blur is controlled by a arameter related to the exosure time Tex. The simulated motion kernels were alied to a shar clean image (ground truth). We also controlled the number of images in the burst M and the noise level in each frame s. The kernels were generated by simulating the random shake of the camera from the ower sectral density of measured hysiological hand tremor []. All the images were aligned by re-centering the motion kernels before blurring the underlying shar image. Figure 5 shows several different realizations for different exosure values Tex. Actually, the

6 6 inut inut inut inut 5 inut 6 inut inut 8 inut 9 FBA Weights wi Filtered v i Image vi inut Fig.. Anatomy of the Fourier aggregation. The first row shows a cro of each inut image vi (frames to 9) and the roosed Fourier Burst Accumulation for = (from (), no additional sharening). The second row shows the contribution of each frame to the final aggregation v i (rescaled for better visualization). The reader can check that the FBA results from the aggregation of different comonents resented in different frames. This is also confirmed by the Fourier weights distribution shown in the bottom row. The bar lot shown on the right indicates the total contribution of each frame to the FBA image. None of the inut images contain all the structure resented in the final aggregation. amount of blur not only deends on the exosure time but also on the focal distance, user exertise, and camera dimensions and mass []. However, for simlicity, all these variables were controlled by the single arameter Tex. We randomly samled thousands of different motion kernels and Gaussian noise realizations, and then alied the Fourier aggregation rocedure for each different configuration (Tex, M, s) several times. The emirical mean square error (MSE) between the ground truth reference and each FBA restoration was comuted and averaged over hundreds of indeendent realizations. We decomosed the mean square error into the bias and variance terms, namely MSE(u ) = bias(u ) + var(u ), to hel visualize the behavior of the algorithm. Figure 5 shows the average algorithm erformance when changing the acquisition conditions. In general, the larger the value of the smaller the bias and the larger the variance. There is a minimum of the mean square error for [, ]. This is reasonable since there exists a tradeoff between variance reduction and bias. Although both the bias and the variance are affected by the noise level, the qualitative erformance of the algorithm remains the same. The bias is not altered by the number of frames in the burst but the variance is reduced as more images are used, imlying a gain in the exected MSE as more images are used. On the other hand, the exosure time mostly affects the bias, being much more significant for larger exosures as exected. C. Imact of Burst Misalignment Misalignment of the burst will certainly have an imact on the quality of the aggregated image. For the general case where images are noisy but also degraded by anisotroic blur the roblem of defining a correct alignment is not well defined. In what follows we consider that the burst is correctly aligned if each vi satisfies vi = u? ki + ni, with the blurring kernel ki having vanishing first moment. That is, R ki (x)xdx =. This constraint on the kernel imlies that the kernel does not drift the image u, so each vi is aligned to u (see Aendix). To evaluate the imact of misalignment, we considered the articular setting in which the error due to registration is a ure shift. Although being a simlified case, this hels to understand the general algorithm erformance as a function of the arameter and the level of misalignment. In this articular case the translation error can be absorbed in a hase shift of the kernel. Although the weights will not change, since they only deend on the Fourier magnitude, the average in () will be out-of-hase due to the misalignment of the images vi. This will result in blur but also on image artifacts. We carried out a similar emirical analysis as before, where several thousands kernels were simulated and centered by forcing to have vanishing first moment. We introduce a Gaussian random D shift to each blurring kernel (i.e., the first moment of the kernel is shifted) with zero mean and standard deviation controlled by a arameter to simulate the misalignment. Figure 6 shows the algorithm erformance when changing the amountof error in the registration (from 5 ixels in average) and the value of controlling the FBA aggregaion. As the alignment error is more significant, the mean square error increases as exected. When the misalignment error is large, the best is to use low values (close to arithmetic mean). On the other hand, when the misregistration error is low, large values will roduce the best erformance in terms of reconstruction error (MSE). For shift errors in the order of ixel, the value giving the minimum MSE is in [, ]. In general, the bias is always reduced with while the variance is increased. This is a direct consequence of the fact that smoothness is reduced when increasing the value (see in Fig. how the energy is concentrated in fewer images as, indicating less smoothing). D. Comarison to Classical Lucky Imaging Techniques Lucky imaging techniques, very oular when imaging through atmosheric turbulence, seek to select and average the sharest images in a video. The Fourier weighting scheme can be seen as a generalization of the lucky imaging family. Lucky imaging selects (or weights more) frames/regions that are

7 - Bias s=. s=. s=. s=.8 s=.6 / - Variance 6 - MSE / (c) Noise level s / 5 - Bias.5 M= M=9 M=6 M=5 M=6 M=9 M=6 - Variance 5 MSE.5 / / (d) Number of images M.5 - Bias.5 Tex=/ Tex=/5 Tex=/ Tex=/ Tex=/ Tex= (a) Examles of simulated kernels (exosure Tex ).5 s =. s =. s =. s =.8 s = Variance MSE 5 (e) Exosure time Tex (b) Examles of noisy simulations (noise level s) Fig. 5. Bias-Variance tradeoff. (a) Simulated kernels due to hand tremor following [5]. Each row shows a set of simulated kernels (left anel) for different exosures Tex = /, /5, /, /, /, and the resective Fourier sectrum magnitude (bottom anel). The arameter Tex controls the amount of exected blur. (b) Simulated noise levels. Average algorithm erformance with resect to when changing (c) the amount of noise s in the inut images, (d) the number of images in the burst M, and (e) the exosure time Tex. The rest of the arameters are set to M = 6, s =. and Tex = /, unless other secified. With short exosures, the arithmetic average ( = ) roduces the best MSE since the images are not blurred. The bias does not deend on M, but the variance can be significantly reduced by taking more images (light accumulation rocedure). Noise affects the bias and the variance terms (with the excetion of = where the bias is unaffected). The MSE lots show the existence of a minimum for [, ], indicating that the best is to balance a erfect average and a max ooling. Avg. error (ixels) / MSE.5.5 Bias Variance ǫ= ǫ = ǫ= ǫ= ǫ= ǫ= (e) Registration error Fig. 6. Imact of misalignment. We simulated shifts following a D Gaussian distribution of zero mean and standard deviation. The average dislacement is aroximately.5 (shown in the table). The to-right lot shows the average algorithm erformance (in MSE) with resect to when changing the amount of registration error. Misalignment deteriorates the algorithm erformance. When the error is large ( ), low values (aggregation is close to arithmetic average) roduces the best MSE, while when the registration error is low ( ), large values roduce lower reconstruction errors. Using a large roduces results that are less smoothed, so in general, variance is increased while bias is reduced. sharer but without aying attention to the characteristics of the blur. Thus, when dealing with camera shake, where frames are randomly blurred in different directions, classical lucky imaging techniques will have a subotimal erformance. In contrast, trying to detect the Fourier frequencies that were less affected by the blur and then build an image with them makes much more sense. This is what the Fourier Burst Accumulation seeks. Recently, Garrel et al. [] introduced a selection scheme for astronomic images, based on the relative strength of signal for each frequency in the Fourier domain. Given a satial frequency, only the Fourier values having largest magnitude are averaged. The user arameter is the ercentage of largest frames to be averaged (tyically ranging from % %). This method was develoed for the articular case of of astronomical images, where astronomers cature videos having thousands of frames (for examle 9 frames in []). Our algorithm is built on similar ideas, but we do not secify a constant number of frames to be averaged. We let the Fourier weighting scheme select the total contribution of each frame deending on the relative strength of the Fourier magnitudes. The authors showed that this generalized lucky imaging rocedure roduces astronomical images of higher resolution and better signal to noise ratio than traditional lucky

8 8 Inut frames cros ordered from highest (left) to lowest (right) gradient energy Arithmetic average Sharness selectivity [] LFA LFA LFA FBA = Fig.. Comarison to lucky imaging techniques on a real data burst (building). The arithmetic average roduces the best noise reduction but comletely removes the details. The lucky imaging algorithm roosed in [] (sharness selectivity) slightly imroves the arithmetic average. In this articular exeriment there is blur in different directions and no single frame shar in all directions. The sharest frame detected (LFA ) is still blurred and significantly noisier than the result given by the Fourier Burst Accumulation with = (final sharening ste has been disabled to do a fair comarison). As more shar frames are averaged (LFA and LFA ) noise is reduced at the exense of blurring the final image. image fusion schemes, further stressing our findings. Traditional Lucky imaging techniques, are based on evaluating the quality of a given frame. In astronomy, the most common sharness measure is the intensity of the brightest sot, being a direct measure of the concentration of the system s oint sread function. However, this is not alicable in the context of classical hotograhy. Haro et al. [] roose to locally estimate the level of sharness using the integral of the energy of the image gradient in a surrounding region (i.e., the Dirichlet energy). If all the images in the series have similar noise level, the Dirichlet energy rovides a direct way of ordering the images according to sharness. (i.e., large Dirichlet energy imlies sharness). Let vi i =,..., M, be a series of registered images of the same scene, the er-ixel Dirichlet energy weights are [] Z i wdirichlet (x) = vi (x) dx, () Ωx where Ωx is a block of ( ) ixels around the ixel x. In ractice, the Dirichlet weights vary oorly with camera shake blur, so although blurry images will contribute less to the final image, their contribution will be still significant. Joshi and Cohen [] roose to use a combination of sharness and selectivity er-ixel weights to determine the contribution of each ixel to the restored image. The sharness weight is built from the local intensity of the image Lalacian and it is ondered by a local selectivity term. The selectivity term enforces more noise reduction in smooth/flat areas. The final sharness selectivity weights are i i wshar-sel (x) = wtex (x)γ(x) (8) i (x) i where wtex (x) = max v is the local sharness measure x vi (x) and γ(x) = λ v (x) /maxx v (x) is the selectivity term controlled by a arameter λ. We have denoted by v the average of all inut frames. In the resent analysis we did not consider the terms due to the resamling error nor the sensor dust that were originally included in the formulation resented in []. In all cases, the final image is comuted as a er-ixel weighted average of the inut images, PM i w (x) vi (x) Pn. vlucky (x) = i= i i= w (x) Figure shows a comarison of these two aroaches to the roosed Fourier Burst Accumulation. We did not include a final sharening ste to faithfully comare all the aroaches, as this last ste could be included in all of them (see Section V). Since the weighted averaged image using () did not show any difference with resect to the arithmetic average, we instead used the total Dirichlet energy to rank all the inut images and then average only the to K (the sharest ones). We named this method Lucky frame average (LFA) and tested different values of K =,,. The weights given by (8) lead to an over-smoothed image with significant less noise. The sharest frame detected (LFA, K = ) is still blurred and noisier than the result given by the Fourier Burst Accumulation with =. As more lucky frames are averaged (LFA and LFA ), more noise is eliminated at the exense of introducing blur in the final image. V. A LGORITHM I MPLEMENTATION The roosed burst restoration algorithm is built on three main blocks: Burst Registration, Fourier Burst Accumulation, and Noise Aware Sharening as a ost-rocessing. These are described in what follows. Burst Registration. There are several ways of registering images (see [5] for a survey). In this work, we use image corresondences to estimate the dominant homograhy relating every image of the burst and a reference image (the first one in the burst). The homograhy assumtion is valid if the scene is lanar (or far from the camera) or the viewoint location is fixed, e.g., the camera only rotates around its otical center. Image corresondences are found using SIFT features [6] and then filtered out through the ORSA algorithm [], a variant of the so called RANSAC method [8]. To mitigate the effect of the camera shake blur we only detect SIFT features having a larger scale than σmin =.8.

9 9 Recall that as in rior art, e.g., [6], the registration can be done with the gyroscoe and accelerometer information from the camera. Fourier Burst Accumulation. Given the registered images {vi }M i= we directly comute the corresonding Fourier transforms {v i }M i=. Since camera shake motion kernels have a small satial suort, their Fourier sectrum magnitudes vary very smoothly. Thus, v i can be lowass filtered before comuting the weights, that is, v i = Gσ v i, where Gσ is a Gaussian filter of standard deviation σ. The strength of the low ass filter (controlled by the arameter σ) should deend on the assumed motion kernel size (the smaller the kernel the more regular its Fourier sectrum magnitude). In our imlementation we set σ = min(mh,mw )/ks, where ks = 5 ixels and the image size is mh mw ixels. Although this low ass filter is imortant, the results are not too sensitive to the exact value of σ (see Figure 8). The final Fourier burst aggregation is (note that the smoothing is only alied to the weights calculation)! M X v i. (9) u = F wi v i, wi = PM v j are still available. The sharening must contemlate that the reconstructed image may have some remaining noise. Thus, we first aly a denoising algorithm (we used the NLBAYES algorithm [9] ), then on the filtered image we aly a Gaussian sharening. To avoid removing fine details we finally add back a ercentage of what has been removed during the denoising ste. The comlete method is detailed in Algorithm. Algorithm : Aggregation of Blurred Images Inut: A series of images v, v,..., v n of size m n c. An integer value. Outut : The aggregated image u w = zeros(m, n); u = zeros(m, n, c); for image i in {,..., n} do Burst Registration 5 The extension to color images is straightforward. The accumulation is done channel by channel using the same Fourier weights for all channels. The weights are comuted by arithmetically averaging the Fourier magnitude of the channels before the low ass filtering. Mi set of corresonding oints. Hi dominant homograhy in Mi. Image resamling. Fourier Burst Accumulation j= i= Mi = SIFT(v i, v ) ; Hi = ORSA(Mi ) ; vi = v i Hi ; v i = FFT(vi ); Pc wi = c j= v ij ; wi = Gσ wi ; u = u + wi v i ; w = w + wi ; Mean over color channels Gaussian smoothing Weighted Fourier accumulation u = IFFT(u /w); Noise Aware Sharening (Otional) u = DENOISE(u ); Gaussian sharening, ρ [, ] u s = u Gρ u ; u = u s + δ(u u ); Add a fraction of removed noise, δ =. [*] c is the number of color channels, tyically. The color channels of v are denoted by v j, for j =,..., c. All the regular oerations (e.g, +, /, ) are oint-wise. no smoothing σ σ 6σ Fig. 8. Imact of smoothing the Fourier weights. To eliminate image artifacts and noise, v i are smoothed before comuting the weights wi. To row shows the results of the FBA average (for the burst shown in Figure ), middle row the Fourier weights, and the bottom row a cro of the Fourier weights, when considering different levels of Gaussian smoothing (no smoothing, σ, σ, 6σ). The strength of the low ass filter is controlled by the arameter σ = min(mh,mw )/ks, where ks = 5 ixels and the image size is mh mw ixels. As shown in the left column (no smoothing), this filtering ste is very imortant. It rovides stabilization to the Fourier weights and also hels to remove noise. The results are stable for a large range of smoothing levels. Noise Aware Sharening. While the results of the Fourier burst accumulation are already very good, considering that the rocess so far has been comutationally non-intensive, one can otionally aly a final sharening ste if resources Memory and Comlexity Analysis. Once the images are registered, the algorithm runs in O(M m log m), where m = mh mw is the number of image ixels and M the number of images in the burst. The heaviest art of the algorithm is the comutation of M FFTs, very suitable and oular in VLSI imlementations. This is the reason why the method has a very low comlexity. Regarding memory consumtion, the algorithm does not need to access all the images simultaneously and can roceed in an online fashion. This kees the memory requirements to only three buffers: one for the current image, one for the current average, and one for the current weights sum. VI. E XPERIMENTAL R ESULTS We catured several handheld bursts with different number of images using a Canon D DSLR camera and the back A variant of this is already available on camera hones, so we stay at the level of otential on-board imlementations.

10 arking night imgs ISO 6, / Canon D woods imgs 6, /8 Canon D ISO ortrait imgs ISO 8, /8 Canon D auvers imgs, /, ipad ISO anthroologie [6] 8 imgs ISO, 5 ms bookshelf imgs ISO, /6 Canon D tequila [6] 8 imgs ISO, ms Fig. 9. Restoration of image bursts catured with two different cameras. The number of frames, the ISO sensitivity and the exosure time is indicated for each burst. Note that in the case of the ipad tablet, the exosure time is the one indicated by the a., and there is no guarantee that this is the real exosure time. Full images are available at the roject s website. camera of an ipad tablet. The full restored images and the details of the camera arameters are shown in Figure 9 and Figure. The hotograhs contain comlex structure, texture, noise and saturated ixels, and were acquired under different lighting conditions. All the results were comuted using =. The full high resolution images are available at the roject s website. The roosed method clearly outerforms [] in all the sequences. This algorithm introduces strong artifacts that degraded the erformance in most of the tested bursts. Tuning the arameters was not trivial since this algorithm relies on arameters that the authors have linked to a single one (named γ). We swet the arameter γ to get the best ossible erformance. Our aroach is concetually similar to a regular align and average algorithm, but it roduces significantly sharer images while keeing the noise reduction ower of the average rincile. In some cases with numerous images in the burst (e.g., see the arking night sequence), there might already be a relatively shar image in the burst (lucky imaging). Our algorithm does not need to exlicitly detect such best frame, and naturally uses the others to denoise the frequencies not containing image information but noise. B. Execution Time Once the images are registered, the roosed aroach runs in only a few seconds in our Matlab exerimental code, while [] needs several hours for bursts of 8- images. Even if the estimation of the blurring kernels is done in a croed version (i.e., ixels region), the multi-image nonblind deconvolution ste is very slow, taking several hours for 6-8 megaixel images. A. Comarison to Multi-image Blind Deblurring Since this roblem is tyically addressed by multi-image blind deconvolution techniques, we selected two state-of-theart algorithms for comarison [], []. Both algorithms are built on variational formulations and estimate first the blurring kernels using all the frames in the burst and then do a ste of multi-image non-blind deconvolution, requiring significant memory for normal oeration. We used the code rovided by the authors. The algorithms rely on arameters that were manually tuned to get the best ossible results. We also comare to the simle align and average algorithm (which indeed is the articular case = ). Figures, and show some cros of the restored images by all the algorithms. In addition, we show two inut images for each burst: the best one in the burst and a tyical one in the series. The roosed algorithm obtains similar or better results than the one by Zhang et al. [], at a significantly lower comutational and memory cost. Since this algorithm exlicitly seeks to deconvolve the sequence, if the convolution model is not erfectly valid or there is misalignment, the restored image will have deconvolution artifacts. This is clearly observed in the bookshelf sequence where [] roduces a slightly sharer restoration but having ringing artifacts (see Jonquet book). Also, it is hard to read the word Women in the sine of the red book. Due to the strong assumed riors, [] generally leads to very shar images but it may also roduce overshooting/ringing in some regions like in the brick wall (arking night). htt://dev.iol.im/ mdelbra/fba/ htt://zoi.utia.cas.cz/files/fastmbd.zi htts://drive.google.com/file/d/bzobvkfrhe5buf9jqzswxryskk/ C. Multi-image Non-blind Deconvolution Figure shows the algorithm results in two sequences rovided in [6]. The algorithm roosed in [6] uses gyroscoe information resent in new camera devices to register the burst and also to have an estimation of the local blurring kernels. Then a more exensive multi-image non-blind deconvolution algorithm is alied to recover the latent shar image. Our algorithm roduces similar results without exlicitly solving any inverse roblem nor using any information about the motion kernels. D. HDR imaging: Multi-Exosure Fusion In many situations, the dynamic range of the hotograhed scene is larger than the one of the camera s image sensor. A oular solution to this roblem is to cature several images with varying exosure settings and combine them to roduce a single high dynamic range high-quality image (see e.g., [], []). However, in dim light conditions, large exosure times are needed, leading to the resence of image blur when the images are catured without a triod. This resents an additional challenge. A direct extension of the resent work is to cature several bursts, each one covering a different exosure level. Then, each of the bursts is rocessed with the FBA rocedure leading to a clean sharer reresentation of each burst. The obtained shar images can then be merged to roduce a high quality image using any existent exosure fusion algorithm. Figure shows the results of taking two image bursts with two different exosure times, and searately aggregating them using the roosed algorithm. We then alied the exosure

11 Tyical Shot Best Shot Align and average S roubek & Milanfar [] Zhang et al. [] Proosed Method (no final shar.) Proosed method Fig.. Real data burst deblurring results and comarison to state-of-the-art multi-image blind deconvolution algorithms (woods rows -, arking night rows -6, bookshelf rows -8, sequences).

12 Tyical Shot Best Shot Align and average S roubek & Milanfar [] Zhang et al. [] Proosed method Tyical Shot Best Shot Align and average S roubek & Milanfar [] Zhang et al. [] Proosed method Fig.. Real data burst deblurring results and comarison with multi-image blind deconvolution methods (auvers). Fig.. Real data burst deblurring results and comarison with multi-image blind deconvolution methods (ortrait). fusion algorithm of [] to get a clean tone maed image from the two burst reresentations. The fusion is much sharer and cleaner when using the Fourier Burst Accumulation for combining each burst than the one given by the arithmetic average or the best frames in each burst. camera. As a future work, we would like to incororate a gyroscoe registration technique, e.g., [6], to create a realtime system for removing camera shake in image bursts. A very related roblem is how to determine the best cature strategy. Giving a total exosure time, would it be more convenient to take several ictures with a short exosure (i.e., noisy) or only a few with a larger exosure time (i.e., blurred)? Variants of these questions have been reviously tackled [6], [], [] in the context of denoising / deconvolution tradeoff. We would like to exlore this analysis using the Fourier Burst Accumulation rincile. VII. C ONCLUSIONS We resented an algorithm to remove the camera shake blur in an image burst. The algorithm is built on the idea that each image in the burst is generally differently blurred; this being a consequence of the random nature of hand tremor. By doing a weighted average in the Fourier domain, we reconstruct an image combining the least attenuated frequencies in each frame. Exerimental results showed that the reconstructed image is sharer and less noisy than the original ones. This algorithm has several advantages. First, it does not introduce tyical ringing or overshooting artifacts resent in most deconvolution algorithms. This is avoided by not formulating the deblurring roblem as an inverse roblem of deconvolution. The algorithm roduces similar or better results than the state-of-the-art multi-image deconvolution while being significantly faster and with lower memory footrint. We also resented a direct alication of the Fourier Burst Accumulation algorithm to HDR imaging with a hand-held A PPENDIX We consider that a burst is correctly aligned if each vi satisfies vi = u? ki + ni, with the R blurring kernel ki having vanishing first moment. That is, ki (x)xdx =. This constraint on the kernel imlies that the kernel does not drift the image u, so each vi is aligned to u. An intuitive way to motivate this requirement is by analyzing the result of iteratively alying the blurring kernel a large number of times. Let k(x) be a non-negative blurring kernel, R R with µ = k(x)xdx and Σ = k(x)(x µ)(x µ)t dx. Then, k?n := k? k?? k G (nµ, nσ),

13 building, imgs, /8 and /, ISO 8, Canon D FBA - short exosure (SE) FBA - exosure fusion FBA - long exosure (LE) FBA - short exosure (SE) FBA - exosure fusion FBAs fusion best frames fusion align & average fusion tyical frame LE best frame LE tyical frame SE best frame SE FBA - long exosure (LE) indoors, imgs, / and /8, ISO 5, ipad Fig.. HDR burst exosure fusion. Two different examles of fusion of two -image bursts catured with two different exosure levels. On to, the FBA average of each burst (left and middle) and the HDR fusion of these two images using [] (right). Below, image cros of the best shot (sharest, manually selected) in each burst and another tyical in the series; the exosure fusion using the regular align and average to combine all the images of a burst, the exosure fusion using the best shot in each series, and the fusion using the roosed Fourier weighted average. Some of the cros have been rescaled to imrove their contrast.

14 Inut Park & Levoy [6] Proosed method Fig.. Restoration results with the data rovided in [6] (anthroologie and tequila sequences). where G (nµ, nσ) is a Gaussian function with mean nµ and variance nσ. This is a direct consequence of the Central Limit Theorem. This Gaussian function can be decomosed into two different comonents: a centered Gaussian kernel (the low ass filter comonent) and a shifting kernel given by a Dirac delta function, that is, G (nµ, nσ) = G (, nσ)? δnµ. This means that iteratively alying n times the kernel k is (asymtotically) equivalent to alying a Gaussian blur with variance nσ and then shifting the image an amount nµ. Thus, we can say that the original kernel k(x) drifts the image in average an amount µ. Therefore, if we do not want the image to be shifted, the blurring kernel should have zero first moment. Following this argument all the simulated kernels R were centered by forcing µ = k(x)xdx =. ACKNOWLEDGMENT The authors would like to thank Cecilia Aguerrebere and Tomer Michaeli for fruitful comments and discussions. R EFERENCES [] O. Whyte, J. Sivic, A. Zisserman, and J. Ponce, Non-uniform deblurring for shaken images, Int. J. Comut. Vision, vol. 98, no., ,. [] T. Buades, Y. Lou, J.-M. Morel, and Z. Tang, A note on multi-image denoising, in In Proc. of Local and Non-Local Arox. in Image Proc. (LNLA), 9. [] H. Zhang, D. Wif, and Y. Zhang, Multi-image blind deblurring using a couled adative sarse rior, in In Proc. IEEE Conf. Comut. Vis. Pattern Recognit. (CVPR),. [] B. Carignan, J.-F. Daneault, and C. Duval, Quantifying the imortance of high frequency comonents on the amlitude of hysiological tremor, Ex. Brain Res., vol., no.,. 99 6,. [5] F. Gavant, L. Alacoque, A. Duret, and D. David, A hysiological camera shake model for image stabilization systems, in IEEE Sensors,. [6] F. Xiao, A. Silverstein, and J. Farrell, Camera-motion and effective satial resolution, in In Proc. Int. Cong. of Imag. Sci. (ICIS), 6. [] V. Garrel, O. Guyon, and P. Baudoz, A highly efficient lucky imaging algorithm: Image synthesis based on fourier amlitude selection, Publ. Astron. Soc. Pac., vol., no. 98, ,. [8] M. Delbracio and G. Sairo, Burst Deblurring: Removing Camera Shake Through Fourier Burst Accumulation, in In Proc. IEEE Conf. Comut. Vis. Pattern Recognit. (CVPR), 5, [9] D. L. Fried, Probability of getting a lucky short-exosure image through turbulence, J. Ot. Soc. Am. A, vol. 68, no., , 98. [] D. Kundur and D. Hatzinakos, Blind image deconvolution, IEEE Signal Process. Mag., vol., no.,. 6, 996. [] R. Fergus, B. Singh, A. Hertzmann, S. T. Roweis, and W. T. Freeman, Removing camera shake from a single hotograh, ACM Trans. Grah., vol. 5, no.,. 8 9, 6. [] Q. Shan, J. Jia, and A. Agarwala, High-quality motion deblurring from a single image, ACM Trans. Grah., vol., no., 8. [] J.-F. Cai, H. Ji, C. Liu, and Z. Shen, Blind motion deblurring from a single image using sarse aroximation, in In Proc. IEEE Conf. Comut. Vis. Pattern Recognit. (CVPR), 9. [] D. Krishnan, T. Tay, and R. Fergus, Blind deconvolution using a normalized sarsity measure, in In Proc. IEEE Conf. Comut. Vis. Pattern Recognit. (CVPR),. [5] L. Xu, S. Zheng, and J. Jia, Unnatural l sarse reresentation for natural image deblurring, in In Proc. IEEE Conf. Comut. Vis. Pattern Recognit. (CVPR),. [6] T. Michaeli and M. Irani, Blind deblurring using internal atch recurrence, in In Proc. IEEE Eur. Conf. Comut. Vis. (ECCV),. [] S. Cho and S. Lee, Fast motion deblurring, ACM Trans. Grah., vol. 8, no. 5,. 5: 5:8, 9. [8] A. Levin, Y. Weiss, F. Durand, and W. T. Freeman, Understanding and evaluating blind deconvolution algorithms, in In Proc. IEEE Conf. Comut. Vis. Pattern Recognit. (CVPR), 9. [9], Efficient marginal likelihood otimization in blind deconvolution, in In Proc. IEEE Conf. Comut. Vis. Pattern Recognit. (CVPR),. [] A. Rav-Acha and S. Peleg, Two motion-blurred images are better than one, Pattern Recogn. Lett., vol. 6, no.,., 5. [] L. Yuan, J. Sun, L. Quan, and H.-Y. Shum, Image deblurring with blurred/noisy image airs, ACM Trans. Grah., vol. 6, no.,. [] J.-F. Cai, H. Ji, C. Liu, and Z. Shen, Blind motion deblurring using multile images, J. Comut. Phys., vol. 8, no.,. 55 5, 9. [] J. Chen, L. Yuan, C.-K. Tang, and L. Quan, Robust dual motion deblurring, in In Proc. IEEE Conf. Comut. Vis. Pattern Recognit. (CVPR), 8. [] F. Sroubek and P. Milanfar, Robust multichannel blind deconvolution via fast alternating minimization, IEEE Trans. Image Process., vol., no.,. 68,. [5] X. Zhu, F. S roubek, and P. Milanfar, Deconvolving PSFs for a better motion deblurring using multile images, in In Proc. IEEE Eur. Conf. Comut. Vis. (ECCV),. [6] S. H. Park and M. Levoy, Gyro-based multi-image deconvolution for removing handshake blur, In Proc. IEEE Conf. Comut. Vis. Pattern Recognit. (CVPR),. [] A. Ito, A. C. Sankaranarayanan, A. Veeraraghavan, and R. G. Baraniuk, Blurburst: Removing blur due to camera shake using multile images, ACM Trans. Grah., Submitted. [8] N. Law, C. Mackay, and J. Baldwin, Lucky imaging: high angular resolution imaging in the visible from the ground, Astron. Astrohys., vol. 6, no.,. 9 5, 6. [9] S. John and M. A. Vorontsov, Multiframe selective information fusion from robust error estimation theory, IEEE Trans. Image Process., vol., no. 5,. 5 58, 5.

15 5 [] M. Aubailly, M. A. Vorontsov, G. W. Carhart, and M. T. Valley, Automated video enhancement from a stream of atmoshericallydistorted images: the lucky-region fusion aroach, in In Proc. SPIE Ot. Eng. Al., 9. [] N. Joshi and M. F. Cohen, Seeing Mt. Rainier: Lucky imaging for multiimage denoising, sharening, and haze removal, in In Proc. IEEE Int. Conf. Comut. Photogr. (ICCP),. [] G. Haro, A. Buades, and J.-M. Morel, Photograhing aintings by image fusion, SIAM J. Imag. Sci., vol. 5, no.,. 55 8,. [] M. Delbracio, P. Musé, A. Almansa, and J.-M. Morel, The Nonarametric Sub-ixel Local Point Sread Function Estimation Is a Well Posed Problem, Int. J. Comut. Vision, vol. 96,. 5 9,. [] G. Boracchi and A. Foi, Modeling the erformance of image restoration from motion blur, IEEE Trans. Image Process., vol., no. 8,. 5 5,. [5] B. Zitova and J. Flusser, Image registration methods: a survey, Image Vision Comut., vol., no.,. 9,. [6] D. Lowe, Distinctive image features from scale-invariant keyoints, Int. J. Comut. Vision, vol. 6,. 9,. [] L. Moisan, P. Moulon, and P. Monasse, Automatic homograhic registration of a air of images, with a contrario elimination of outliers, J. Image Proc. On Line (IPOL), vol.,. 56,. [8] M. A. Fischler and R. C. Bolles, Random samle consensus: a aradigm for model fitting with alications to image analysis and automated cartograhy, Comm. ACM, vol., no. 6,. 8 95, 98. [9] M. Lebrun, A. Buades, and J.-M. Morel, Imlementation of the Non- Local Bayes image denoising algorithm, J. Image Proc. On Line (IPOL), vol.,.,. [] E. Reinhard, W. Heidrich, P. Debevec, S. Pattanaik, G. Ward, and K. Myszkowski, High dynamic range imaging: acquisition, dislay, and image-based lighting. Morgan Kaufmann,. [] F. Banterle, A. Artusi, K. Debattista, and A. Chalmers, Advanced high dynamic range imaging: theory and ractice. AK Peters (CRC Press),. [] W. Zhang and W.-K. Cham, Gradient-directed comosition of multiexosure images, in In Proc. IEEE Conf. Comut. Vis. Pattern Recognit. (CVPR),. [] L. Zhang, A. Deshande, and X. Chen, Denoising vs. deblurring: HDR imaging techniques using moving cameras, in In Proc. IEEE Conf. Comut. Vis. Pattern Recognit. (CVPR),. Guillermo Sairo was born in Montevideo, Uruguay, on Aril, 966. He received his B.Sc. (summa cum laude), M.Sc., and Ph.D. from the Deartment of Electrical Engineering at the Technion, Israel Institute of Technology, in 989, 99, and 99 resectively. After ost-doctoral research at MIT, Dr. Sairo became Member of Technical Staff at the research facilities of HP Labs in Palo Alto, California. He was with the Deartment of Electrical and Comuter Engineering at the University of Minnesota, where he held the osition of Distinguished McKnight University Professor and Vincentine Hermes-Luh Chair in Electrical and Comuter Engineering. Currently he is the Edmund T. Pratt, Jr. School Professor with Duke University. G. Sairo works on theory and alications in comuter vision, comuter grahics, medical imaging, image analysis, and machine learning. He has authored and co-authored over aers in these areas and has written a book ublished by Cambridge University Press, January. G. Sairo was awarded the Gutwirth Scholarshi for Secial Excellence in Graduate Studies in 99, the Ollendorff Fellowshi for Excellence in Vision and Image Understanding Work in 99, the Rothschild Fellowshi for Post-Doctoral Studies in 99, the Office of Naval Research Young Investigator Award in 998, the Presidential Early Career Awards for Scientist and Engineers (PECASE) in 998, the National Science Foundation Career Award in 999, and the National Security Science and Engineering Faculty Fellowshi in. He received the test of time award at ICCV. G. Sairo is a Fellow of IEEE and SIAM. G. Sairo was the founding Editor-in-Chief of the SIAM Journal on Imaging Sciences. Mauricio Delbracio received the graduate degree from the Universidad de la Reública, Uruguay, in electrical engineering in 6, the MSc and PhD degrees in alied mathematics from École normale suérieure de Cachan, France, in 9 and resectively. He currently has a ostdoctoral osition at the Deartment of Electrical and Comuter Engineering, Duke University. His research interests include image and signal rocessing, comuter grahics, hotograhy and comutational imaging.