Restoration and resolution enhancement of a single image from a vibration-distorted image sequence

Transcription

1 Restoration and resolution enhancement of a single image from a vibration-distorted image sequence Adrian Stern, MEMBER SPIE Elad Kempner Avi Shukrun orman S. Kopeika, FELLOW SPIE Ben-Gurion University of the egev Department of Electrical and Computer Engineering P.O. Box Beer-Sheva, Israel stern@eesrv.bgu.ac.il kopeika@bguee.bgu.ac.il Abstract. In many applications such as in airborne and terrestrial reconnaissance, robotics, medical imaging, and machine vision systems, the images of a video sequence are severely distorted by vibrations. Superresolution algorithms are suitable for restoring an image from a lowfrequency vibrated sequence because of high correlation between the frames and inherent interframe motion. However, we show that superresolution algorithms, which were developed for general types of blur, should be adapted to the specific characteristics of low-frequency vibration blur. We demonstrate that in the case of image sequences distorted by vibration, the images should be selected prior to processing. We find empirical selection criteria and propose a selection procedure Society of Photo-Optical Instrumentation Engineers. [S (00) ] Subject terms: vibration-blurred images; superresolution; iterative backprojection method; vibrations; image restoration; point spread function; optical transfer function. Paper received Oct. 22, 1999; revised manuscript received Apr. 12, 2000; accepted for publication Apr. 13, Introduction Vibration blur may hide vital information in images taken with dynamic imaging systems. Image vibration is common when the imaging system is located on aircraft, ships, and other vehicles because of turbines and motors or mechanical structural resonances. Vibration can be minimized by proper design. In practice, however, it is often the most serious source of image distortion despite even state of the art stabilization. In this paper, we address the problem of restoration of images from sequences distorted by low-frequency vibration. Low-frequency vibrations are defined as those vibrations for which the exposure time t e is less than the vibration period T 0 Fig. 1. Low-frequency vibrations are more common and more complicated to deal with than highfrequency vibrations for which t e T 0 ) because the point spread function PSF due to such vibrations exhibits a strong random behavior. The random behavior of the vibration PSF is illustrated in Fig. 1. We can see that at different instants of exposure t x, the PSF differs in shape and displacement x. Image degradation caused by low-frequency vibration has been analyzed in detail previously. 1 3 The lost information due to vibration smear can be recovered by using the information from adjacent images in the video sequence by applying superresolution restoration SR algorithms. Vibrated sequences possess two inherent characteristics required for efficient restoration using SR algorithms. The first is the inherent motion between the images because the PSF location changes in each image Fig. 1. Even though motion is not strictly required for SR restoration, 4 it enables better restoration. The second characteristic is the high correlation between consecutive images in the sequence because vibrated images possess similar fields of view. umerous SR approaches and algorithms were developed in the last decade and a half. Typical applications of SR methods can be found in the fields of remote sensing, medical imaging, reconnaissance, etc. They can be used for frame freezing in a video sequence and video standard conversion. Superresolution reconstruction algorithms consists of three basic components: 1 motion compensation, 2 interpolation, and 3 blur and noise removal, which are implemented separately or simultaneously. The first SR algorithm proposed by Tsai and Huang 5 addressed the interpolation component. They demonstrated the ability to reconstruct one improved resolution image from several downsampled noise-free versions of it. Their frequency approach, which makes explicit use of the aliasing effect, was extended by others. 6 8 A spatial domain alternative was suggested by Ur and Gross. 9 A class of iterative image domain algorithms that solves simultaneously the restoration and interpolation steps has also been proposed These methods pose a model relating low-resolution LR images to the desired SR images and then use iterative reconstruction techniques to estimate the SR image. Peleg et al. 13 and Peleg and Irani 14,15 suggested a restoration approach based on the iterative backprojection IBP method used in computer-aided tomography. It can be shown 4 that the IBP method can be viewed as maximum likelihood ML or least-squares estimation method without regularization. Extension and application of the IBP method for IR image restoration was developed by Hardie et al. 16 and Cohen and Dinstein 17 improved the IBP method by integrating polyphase filtering with it. Opt. Eng. 39(9) (September 2000) /2000/$ Society of Photo-Optical Instrumentation Engineers 2451

2 Fig. 1 Motion functions during exposures of a low-frequency vibrating imaging platform. The spread function depends on the instant of the exposure (t x ), which is random. Iterative SR restoration algorithms based on the method of projection onto convex sets POCS was suggested by several authors The POCS method incorporates efficiently nonlinear constraints such as spatial dependence of the PSF but makes the computations more complex. Elad and Feuer 4 proposed a unified methodology toward the SR problem. They show the equivalencies between the ML, maximum a posteriori, and POCS methods and propose a hybrid algorithm that combines the simplicity of the ML estimator and the ability of POCS to incorporate nonlinear constraints. We point out that even though most of the abovementioned SR methods consider blur due to nonzero aperture size, most of them except 10,21 ignore blurring due to nonzero aperture time motion blur. Several methods were developed to restore a single image from a sequence of images blurred by motion. Hadar et al. 22 use a sequence of images distorted by vibration to estimate the exact motion during the exposure of a specific image. Then the motion function is used to calculate the motion optical transfer function OTF, which is used together with the Wiener filter to restore the specific image. The algorithm is suboptimal since the information in consecutive images is used only for motion estimation and not for deblurring and resolution enhancement. An iterative method for deblurring an image from a motion-distorted sequence was developed by Trussel and Fogel. 10 The method uses Landweber iteration for debluring the image but not for superresolution enhancement. The method is demonstrated by restoring an image from only two successive images distorted by space variant motion blur. The SR recovery method developed by Patti et al. 21 is formulated for the most general case. It considers image sampling on any spatiotemporal grid with nonzero aperture size and exposure time. It can be applied to space-varying motion blur in different kinds of sampling lattices. The restoration is based on the POCS method, which makes it computationally complex. In this paper, we do not propose a new method for recovering an image from a vibrated sequence. We do not address the problem of how the data captured images should be processed. Instead, we deal with the question of which data should be processed. We show that in the case of vibration degradation, some images should be discarded from the sequence to improve restoration quality. Selection of images prior to processing is not required in the applications for which most of the SR methods were designed. To the contrary, SR methods, which were developed for sequences with approximately similar blur in each image, are more effective as more captured images are used more data is available. As shown in Fig. 1, the vibration PSF differs from other common types of blur such as due to defocus, finite aperture size, lens aberration, constant velocity motion, atmospheric turbulence and light scatter in that they have variable size and shape in each sequence frame. Because of this, some of the images are significantly worse than the others. This leads to the basic idea in this paper that some of the images in the sequence should be excluded before the restoration process. By doing this, a better restored image is achieved with significantly less computational effort. Analysis of lucky shots probabilities and frame selection procedures were demonstrated in other domains for other kinds of blur or applications. Lucky shot probabilities for short-exposure imagery through atmospheric turbulence were calculated by Fried. 23 Wulich and Kopeika 2 calculated probabilities of lucky shots in ensembles of low-frequency vibrated images. Frame selection techniques were developed for images captured through the atmosphere and using adaptive optics AO systems see, for example, Ref. 24. These frame selection techniques differ from that suggested in this paper in the types of blur that are applied and in the postprocessing technique used. The postprocessing used for AO imagery consists only of registration, averaging frames, and deblurring with a pseudo-wiener filter. This paper is organized as follows. In Sec. 2, we describe vibrated sequence simulation and restoration algorithms. We use the IBP algorithm because of its efficiency and relative simplicity. By using simulated sequences we are able to give a qualitative measure for each restoration by comparing the restored image to the original one. In Sec. 3, we demonstrate the advantage achieved by using the selection principle. In Sec. 4, we propose a method to select the images to be discarded and those to be used from the given vibrated sequence. We assume here that resolution is limited primarily by blur spatial frequency bandwidth rather than by noise. This is especially relevant when lownoise sensors such as modern camera are used Optical Engineering, Vol. 39 o. 9, September 2000

3 f n1 f n 1 T 1 k g k g n k s k *p, k1 2 where s k is the inverse operator of s k and p is a backprojection kernel, determined by h k. In the case where T k consists only of translation, the kernel p must obey the following constraint: h*p Constraint 3 can be written in Fourier Domain as 01HP1. 4 Fig. 2 Distorted LR frames g k k1 2 Vibrated Sequence Simulation and the Restoration Algorithm In our simulations, we considered sinusoidal vibrations with amplitudes in the range of 5 to 20% of the field of view and frequencies between 2 and 10 Hz. The motion function for each exposure was obtained by taking 1/60-s long samples from each sinusoidal vibration function at intervals of 20 ms. Then LR sequences were composed of blurred and downsampled images appropriate to each exposure. The sequence formation model used in our simulation is described in Fig. 2. Each vibrated image g k was simulated by convolving a pixel high-resolution image f with the particular PSF appropriate to each instant of exposure h k. The vibration PSFs are the histogram of the displacement function. 1,3,22 Each image was downsampled with a factor of 4 using the method of pyramids, obtaining pixel images. The length of the sequences varied between 15 to 200 frames. Long sequences were simulated to contain approximately uniform distributions of the vibration PSFs so that practically all the possible types of lowfrequency vibration PSFs were included. Using the images of the sequence, or part of the images, a single image was restored. An ideal restoration would yield the original image f. The restoration was carried out using the IBP algorithm 13,15 described briefly as follows. The restoration starts with an arbitrary guess fˆ (0) for the high-resolution image. At each iteration step, the imaging process Fig. 2 is simulated to obtain a set of LR images g (n) k k1 corresponding to the sequence of blurred frames g k k1. The k th simulated image at the iteration step is given by g k n T k f n *h 1k s k, simulation model. where T k is the geometric transformation from f to g k, h k is the PSF of the k th image, s k is the decimation operator for downsampling, and * denotes the convolution operator. At each step, the difference images g k g (n) k k1 are used to improve the previous guess fˆ (n) by using the following update scheme: 1 Since the purpose of this paper is only to check the dependence of the restoration quality on the PSF distribution of the images used, the only geometric transformation we assumed is translation so T k is due only to translation. However, it is difficult to define the translation of the scene for the case of vibration. The motion of the objects in vibrated images can be viewed as the motion of nonrigid bodies. Objects are blurred differently in each image and therefore their shapes change. In other words, not all the object points in the first images possess the same translation to the next image. In this paper, we define the translation of the images as the translation of the PSF s center of mass ( 1 and 2 in Fig. 1. The center of mass of the k th image is given by kx ky xhk x,y dx dy, yhk x,y dx dy, where x, y are the image coordinates. The IBP algorithm as proposed by Irani and Peleg 15 estimates the motion also, but since this is not the objective of this work we assume that the motion function is known and therefore the translation, too. 3 Restoration Using Selected Images from the Sequence The basic idea presented in this work is that, in the case of low-frequency vibrated images, the use of more images in the restoration process does not necessarily add information. To the contrary, if a severely blurred image is added to the set of the images used for restoration, the restored image obtained may be poorer. This is demonstrated in Figs. 3a and 3b, which show some typical results of high-resolution restoration of LR vibration-blurred images. Gaussian noise was added to the images, forming a signalto-noise SR of 37 db. Figures 3a and 3b are two examples of more severe and less severe blurred LR ( pixel) images from a horizontally blurred sequence of images. The vibrations are of amplitude equal to 12% of the field of view, the vibration frequency is 15 Hz, and the exposure time t e 1/60 s. Figure 3c shows the highresolution, pixel restoration of the image using a sequence of 16 frames. Figure 3d shows the restoration of 5 Optical Engineering, Vol. 39 o. 9, September

4 Fig. 3 (a) and (b) Example of two vibrated LR pixel images from the sequence. The images are distorted by horizontal vibration with amplitude 12% of the field of view, vibration frequency 15 Hz, and exposure time t e 1/60 s; (c) high-resolution pixel restored image using 16 images of the sequence; and (d) high-resolution restoration using the four least blurred images. the image using the four least blurred images from the 16 images used for Fig. 3c. The number of iterations used for both restorations is 60. It is evident that more details can be seen in Fig. 3d than in Fig. 3c. For example, the fifth line is readable in Fig. 3d, while in Fig. 3c it is essentially not. Clearly, also much more computation effort was required for the restoration shown in Fig. 3c based on 16 degraded images than for that shown in Fig. 3d based on four blurred images. For the restoration shown in Fig. 3c, approximately four times more memory was required than for that of Fig. 3d, and the restoration process was more than three times longer. The dependence of the restoration quality on the number of images used is shown in Fig. 4. Figure 4 shows an example of the restoration root mean square error RMSE as a function of the number of images used for restoration. The RMSE is calculated by RMSE 1 2 i1 j1 1/2 fˆi, j f i, j 2, 6 where f is the original high-resolution, pixel image, and fˆ is the restored high-resolution image. We recall that f is known since the experiment was simulated and f was used for this simulation of the sequence. The results shown in Fig. 4 were obtained from the following experiment. First the whole sequence consisting of 16 images was used to restore the image. The RMSE was calculated using Eq. 6. The restoration and RMSE calculation was repeated after excluding the most severely blurred images each time. We can see that the RMSE decreases as the number of images used is decreased, until it reaches a minimum when using the four least blurred images Optical Engineering, Vol. 39 o. 9, September 2000

5 Fig. 4 Dependence of the RMSE on the number of images used. 4 Selection Criteria for the Images to be Used in the Restoration Algorithm As demonstrated in the previous section, the images to be used for restoration depend on the severity of their blur. A common measure of the blur is the blur length PSF width. In the case of low-frequency vibration, the blur length is not a good measure because the vibration PSFs are not of identical shape Fig. 1. Instead we define the effective blur extent b as 3.46 times standard deviation of the PSF: b i 3.46 x ix 2 1/2 y iy 2 h i x,y dx dy, 7 where x, y are the Cartesian coordinates, h i (x,y) is the PSF of the i th image, and ix and iy are the PSF center of mass coordinates, as defined in Eq. 5. The multiplicator 3.46 (12) in Eq. 7 is chosen so that the effective blur equals the real blur extent in the case of constant velocity motion. In Eq. 7, it is assumed that the motion PSF has nonzero values only in a line in the direction of the motion linear smear. We define also the blur diversity ratio k n in a set of n images: b n k n 1/n1 n1, i1 b i where b n is the effective blur of the most severely blurred image and 1/(n1) n1 i1 b i is the mean effective blur of the other images in the set. We found empirically that the most blurred image in a set of n vibrated images can be excluded from the restoration process if the blur diversity of the set k n is higher than If k n is lower than 1.40, which means that the effective blur of the most blurred image is no larger than 40% of the mean of the others, the whole set of images should be used in the restoration process. The critical k n was found from the following experiment. A set of 200 frames distorted by vibration was simulated. Sequences of 30 frames were randomly sampled. Using each sequence, 15 restorations were carried out, each using four images. The images 8 Fig. 5 Restoration RMSE versus the blur diversity of the set k n. in the sequence differ by their mean blur and blur diversity ratio k n. Starting with an initial group of four images, the IBP restoration was carried out. After each restoration, the three least blurred images were kept for the next restoration and the worst image was changed with a less blurred image from the sequence. The blur diversity ratio k n and the RMSE were calculated for each experiment. In the preceding procedure the blur diversity ratio k n decreased with each restoration. An additional criteria for choosing the images was that after registration of the images in the highresolution grid, the relative translations between the images span approximately uniformly the LR pixel. Figure 5 shows a representative example for the k n behavior in the described procedure. The solid line in Fig. 5 shows a plot of the RMSE versus the blur diversity ratio k 4. We can see that the restoration error increases with k 4. The dashed line shows the RMSE by using only the three least blurred images. We can see that restorations using images with blur diversity ratio k 4 larger than approximately 1.40 are poorer than the restoration using the only three least blurred images out of the four. In others word, if the blur diversity ratio of the four given images is larger than 1.40, then the most severely blurred images should be removed and the restoration should be carried out with only three images. Based on the preceding discussion we suggest the following procedure to select the image prior to restoration. Given a sequence of n images distorted by vibration: 1. Calculate the effective blur b i of all the images (i 1,...,n) in the sequence using Eq Calculate the blur diversity ratio k n of the n images using Eq If k n 1.40, then perform the restoration using all n images. If k n 1.40, exclude the most severely blurred image having the largest effective blur, let nn1 and repeat steps 2 and 3. The density of the high-resolution grid to be used is constrained by the number of images remaining after the Optical Engineering, Vol. 39 o. 9, September

6 preceding selection procedure and by the span of their relative displacements. The interpolation factor, which is the ratio between the densities of the high-resolution and the LR grids, should be no larger than the number of the images used for the restoration. It is also preferable that the relative displacements of the images after registration span the LR pixel approximately uniformly. By maintaining this, to each high-resolution pixel at least one LR pixel is registered. Using denser high-resolution grids that do not apply to this constraint does not improve the restored image resolution but increases the processing effort and time and demands more memory capacity. 5 Conclusions We demonstrated that contrary to most applications for which SR algorithms were developed, in which when more images were used yielded better restoration, in the case of a sequence distorted by vibration, only some of the images should be used. The best images from the sequence should be chosen prior to restoration. We found empirically that if the worst image in the set of images to be used for restoration has an effective blur larger by approximately 40% than the average of the others, it should be discarded from the set. Attempts to use severely blurred images yield a poorer restoration with a larger calculation effort. We propose an image selection procedure to be used prior to restoration. By choosing the minimal number of images, better restoration is achieved with significantly less memory resources and shorter processing time. We demonstrated the selection principle using the common IBP restoration method. However, we believe that our conclusions hold in general for other restoration methods, since other methods known to us do not address the vibration blur characteristics. We believe that even though other methods may converge faster or more precisely, their relative performance will depend on the set of the images used in the process. References 1.. S. Kopeika, A System Engineering Approach to Imaging, chap. 14, pp , and chap. 18, pp , SPIE Optical Engineering Press, Bellingham, WA D. Wulich and. S. Kopeika, Image resolution limits resulting from mechanical vibration, Opt. Eng. 26, O. Hadar, I. Dror, and. S. Kopeika, Image resolution limits resulting from mechanical vibration. Part IV: real time numerical calculation of optical transfer functions and experimental verification, Opt. Eng. 332, M. Elad and A. Feuer, Restoration of a single superresolution image from several blurred, noisy, and undersampled measured images, IEEE Trans. Image Process. 612, R. Y. Tsai and T. S. Huang, Multiframe image restoration and registration, in Advances in Computer Vision and Image Processing, T. S. Huang, Ed., JAI Press, Greenwich, CT S. P. Kim,. K. Bose, and H. M. Valenzuela, Recursive reconstruction of high resolution image from noisy undersampled multiframes, IEEE Trans. Acoust., Speech, Signal Process. 386, S. P. Kim and W. Y. Su, Recursive high-resolution reconstruction of blurred multiframe images, IEEE Trans. Image Process. 2, K. Bose, S. P. Kim, and H. M. Valenzuela, Recursive implementation of least squares algorithm for image reconstruction from noisy, undersampled multiframes, in Proc. IEEE Int. Conf. Acoustics Speech, and Signal Processing (ICASSP), Vol. 5, pp , Minneapolis, M H. Ur and D. Gross, Improved resolution from subpixel shifted pictures, CVGIP Graph. Models Image Process. 542, H. J. Trussell and S. Fogel, Identification and restoration of spatially variant motion blurs in sequential images, IEEE Trans. Image Process. 11, M. G. Choi, O. E. Erdogan,. P. Galatsanos, and A. K. Katsaggelos, Multichannel regularized iterative restoration of image sequences, Visual Communication and Image Processing, Proc. SPIE 2094, M. Elad and A. Feuer, Super-resolution restoration of continuous image sequence using the LMS algorithm, in Proc. IEEE 18th Convention of Electrical and Electronics Engineers, Chap , pp. 1 5, Tel Aviv S. Peleg, D. Keren, and L. Schweitzer, Improving image resolution using subpixel motion, Pattern Recogn. Lett. 5, A. Irani and S. Peleg, Improving resolution by image registration, CVGIP: Graph. Models Image Process. 53, A. Irani and S. Peleg, Motion analysis for image enhancement: resolution, occlusion, and transparency, J. Visual Commun. Image Represent 4, R. C. Hardie, K. J. Barnard, J. G. Bognar, E. E. Armstrong, and E. A. Watson, High-resolution image reconstruction from a sequence of rotated and translated frames and its application to an infrared imaging system, Opt. Eng. 371, B. Cohen and I. Dinstein, Resolution enhancement by polyphase back-projection filtering, in Proc. IEEE Int. Conf. on Image Processing, Vol. 5, pp H. Stark and P. Oskoui, High-resolution image recovery from image-plane arrays, using convex projections, J. Opt. Soc. Am. A 611, A. M. Teklap, M. K. Ozkan, and M. I. Sezan, High-resolution image reconstruction from lower-resolution image sequences and space varying image restoration, in Proc. IEEE Int. Conf. Acoustics, Speech and Signal Processing (ICASSP), Vol. III, pp , San Francisco, CA A. J. Patti, M. I. Sezan, and A. M. Teklap, High-resolution image in presence of time-varying blur, in Proc. ICIP, pp , Austin, TX A. J. Patti, M. I. Sezan, and A. M. Teklap, Superresolution video reconstruction with arbitrary sampling lattices and nonzero aperture time, IEEE Trans. Image Process. 68, O. Hadar, M. Robbins, Y. ovogrozky, and D. Kaplan, Image motion restoration from a sequence of images, Opt. Eng. 3510, D. L. Fried, Probability of getting a lucky short-exposure image through turbulence, J. Opt. Soc. Am. 6, M. C. Roggemann, C. A. Stoudt, and B. M. Welsh, Image-spectrum signal-to-noise-ratio improvement by statistical frame selection for adaptive optics imaging through atmospheric turbulence, Opt. Eng. 33, Adrian Stern received his BSc and MSc (cum laude) degrees in electrical and computer engineering from Ben-Gurion University of the egev, Israel, in 1988 and He is now a PhD student in the Electrical and Computer Engineering Department and part of the electro-optics research group. His current research interests are in effects of motion and vibrations on sequences of images, influence of video camera vibration on the human visual system, image restoration, target aquiaition, image compression, imaging through atmosphere. Mr. Stern is a member of SPIE. orman S. Kopeika received the BS, MS, and PhD degrees in electrical engineering from the University of Pennsylvania, Philadelphia, in 1966, 1968, and 1972, respectively. His PhD dissertation, supported by a ASA Fellowship, dealt with detection of millimeter waves by glow discharge plasmas and the utilization of such devices for detection and recording of millimeter wave holograms. In 1973 he joined the Department of Electrical Engineering, Ben- Gurion University of the egev, Beer-Sheva, Israel as a lecturer. He was appointed a professor in 1988 and incumbent of the Reuben and Frances Feinberg Chair in electrooptics in During he was a visiting associate professor in the Department of Electrical Engineering, University of Delaware, ewark, Delaware. His research interests include atmospheric optics, target acquisition, effects of surface phenomena on optoelectric device properties, optical communication, electronic properties of plasmas, laser breakdown of gases, the optogalvanic effect, electromagnetic waveplasma interaction in various portions of the EM spectrum, and utilization of such phenomena in EM wave detectors and photopreionzation lasers. He has published over 130 reviewed journal papers and over 90 conference papers in the above areas. He was particularly active in research of time response and impedance properties of plasmas and authored a general unified theory to ex Optical Engineering, Vol. 39 o. 9, September 2000

7 plain EM wave-plasma interactions all across the electromagnetic spectrum. His earlier published work on the optogalvanic effect preceded the naming of the effect. He has done extensive work on the wavelength and weather dependences of image resolution through the open atmosphere, particularly with regard to spatial coherence degradation and spatial frequency dependence resulting from forward light scattering by relatively large airborne particulates. He and his students have developed methods to predict atmospheric modulation transfer function, including both turbulence and aerosol MTF components, according to weather, and to use that information in image restoration so as to deblur such effects. Also, methods to calculate numerically in real time optical trnasfer function for any type of image motion and vibration have been developed, and these too have also been used in image restoration. He contributed actively toward the development of postfabrication techniques for improvement of photodiode responsivity and uniformity and for wavelength-tuning of semiconductor light sources by external means. His textbook A System Engineering Approach to Imaging was published by SPIE Optical Engineering Press (1998). Recently he has been involved in adaptive techniques for satellite optical communication networks and effects of the atmosphere and image motion and vibration on target acquisition. During he served two terms as department chair. Presently, he is chair of the electrooptics engineering unit, which is the first to grant graduate degrees in electrooptics in Israel. Dr. Kopeika is a fellow of SPIE, a Senior Member of the IEEE, and a member of the Optical Society of America, and the Laser and Electrooptics Society of Israel. Biographies of the other authors not available. Optical Engineering, Vol. 39 o. 9, September