Restoration of images captured by a staggered time delay and integration camera in the presence of mechanical vibrations

Transcription

1 Restoration of images captured by a staggered time delay and integration camera in the presence of mechanical vibrations Gadi Hochman, Yitzhak Yitzhaky, Norman S. Kopeika, Yair Lauber, Meira Citroen, and Adrian Stern Staggered time delay and integration TDI scanning image acquisition systems are usually employed in low signal-to-noise situations such as thermal imaging. Analysis and restoration of images acquired by thermal staggered TDI sensors in the presence of mechanical vibrations that may cause space-variant image distortions severe geometric warps and blur are studied. The relative motion at each location in the degraded image is identified from the image when a differential technique is used. This information is then used to reconstruct the image by a technique of projection onto convex sets. The main novelty is the implementation of such methods to scanned images columnwise. Restorations are performed with simulated and real mechanically degraded thermal images Optical Society of America OCIS codes: , , , Introduction Time delay and integration TDI cameras are a wellknown implementation of an optical sensor array and are used in various applications. 1 3 The TDI sensor offers better sensitivity compared with a regular CCD sensor, an advantage that is significant in lowlight-level imaging and in thermal imaging in particular. The imaging process of a TDI camera was described in detail by Wong et al. 4 and is briefly described here. A TDI sensor is constructed from columns of N photosensitive cells also known as TDI stages. The object is scanned along the horizontal direction while the charge accumulated in each cell is synchronically transferred opposite the scan direction. The TDI array is assembled with M rows the image vertical dimension containing N TDI stages each. The imaging process causes multiple exposures of the object and sums them throughout the G. Hochman, Y. Yitshaky, N. S. Kopeika, and A. Stern are with the Department of Electro-Optics Engineering, Beer Sheva 84105, POB 653, Israel for Y. Yitzhaky, itzik@ee.bgu.ac.il. Y. Lauber and M. Citroen are with ELOP Electrooptics Industries Ltd., Advanced Technology Park, POB 1165, Rehovot 76111, Israel. Received 8 January 2004; revised manuscript received 26 April 2004; accepted 5 May $ Optical Society of America TDI stages. This results in an improvement in the signal-to-noise ratio of N 1 2 compared with a regular staring CCD array. The emerging need for high-resolution scanning systems able to scan large areas quickly creates a requirement for large multielement detectors. 5 Such a detector structure requires a great deal of circuitry surrounding each detector pixel. The staggered configuration illustrated in Fig. 1 allows more space for the circuitry adjacent to each detector element. 6 This configuration permits scanning without loss of information between neighboring areas in the scene. In this configuration the scanning sensor is divided into two spatially separated fields; one captures the odd, while the other captures the even, image rows, forming an interleaved image with a full spatial resolution perpendicular to the scanning direction. As a result of the distance between the odd and even fields margin N pixels as shown in Fig. 1, a constant time delay occurs between the image captured by the odd and the even fields. Barbe 7 analyzed a number of phenomena affecting the resolution of TDI cameras. Owing to the long integrated exposure time of the TDI cameras and the wide margin between the odd and the even sensors in the staggered structure, the recorded images may be degraded significantly by the vibrations. Thermal-imaging systems are frequently affected by vibrations caused by electrical cooling systems, 1 August 2004 Vol. 43, No. 22 APPLIED OPTICS 4345

2 Fig. 1. Staggered TDI sensor configuration. servo engines, or platform airplane, tank, etc vibrations. Efforts to constrain these vibrations are usually part of the system s development process, but residual motion due to vibrations always exists. The influence of vibrations on staring imaging systems has been well investigated, 8 and the basic tools developed for staring systems apply in the case of scanning arrays. The point-spread function PSF resulting only from relative motion between the sensor, and the scene is equal to the histogram of the sensor position during exposure 8 : PSF f x,y x, y, (1) where x and y are the spatial orthogonal axes and the histogram f x,y x, y equals the probability density function of the motion during exposure. In the case of low-frequency sinusoidal motion where the vibration period is longer than the exposure time, the PSF is random, depending on the initial time of exposure. Yitzhaky et al. 9 performed blind restoration of a single progressive image degraded by vibrations. The geometrical distortion problem that occurs when a composite frame constructed from two, odd and even, fields is degraded by motion was addressed by Yitzhaky and Stern. 10 Miettinen and Ailisto 11 examined the effects of vibrations on images captured by a TDI camera at a nonzero viewing angle relative to the object s surface and presented a method for estimating the image quality in vibrations conditions. In the case of highfrequency vibrations the image degradation is space invariant. In low-frequency vibrations the sensor is affected by different portions of the vibration sine wave along the scanning process, causing spacevariant image degradation. 2. Vibrated Time Delay and Integration Imaging Process Wolberg and Loce 12 developed a one-dimensional model describing the quality of images captured by a linear scanner in the presence of vibrations. This model reflects the influences of vibrations along the scan direction on the local PSF and on the sample points of the sensor, which are arbitrarily spaced owing to vibrations. Neglecting the degrading effects of the imaging system such as aberration and diffraction, the pixel size is significantly the most dominant cause, limiting the system s resolution. Therefore the shape of the PSF of a staring system can be approximated as the shape of the pixel a two-dimensional rectangular. When each scan step has a one-pixel extent, the Fig. 2. Effect of vibrations on the column PSF in the TDI imaging process. nonvibrated PSF in the scan direction is a onedimensional convolution of two identical rectangular functions, forming a triangular shape, and a rectangular shape vertical to the scan direction. 13 For other scan step extents the camera s PSF in the scan direction will have a trapezoidal shape. When the vibration velocities are low relative to the scan velocity the single TDI-stage PSF approximately maintains its triangular shape. As illustrated in Fig. 2 the effects of the vibrations on the resulting column PSF are 1 to be broaden the PSF as a result of the spatial spread of the sampling points of all the integrated TDI stages and 2 to shift the PSF s center of gravity position from its nominal position according to the average location of the N sampling points. The first effect causes the shift-variant motion blur, while the second causes geometrical deformations in each field. The PSF s center of gravity shift, combined with the spatial shift between the odd and even fields the staggered structure, cause the staircase or comb effect in the combined image of both fields. These effects apply also to the vertical axis in which the single TDI-stage PSF has a rectangular shape. 3. Simulation of a Time Delay and Integrated Camera in the Presence of Vibrations A main element of the simulation is the formation of random motion vectors of the camera, which represent the camera s space-variant displacements in the scan direction and vertical to this direction with respect to the nominal positions without vibrations. Frequently the given information in the case of mechanical vibrations is an estimation of the power spectral density PSD of the vibrations. The PSD is given as the power spectrum of the angular velocity of the image plane. A transformation from that angular-velocity PSD to spatial displacements can be carried out as follows. The spatial amplitude component at the ith frequency interval can be calculated according to the characteristics of the PSD of the camera vibrations: A i pixels 2PSD i Freq i 1 2 pix, (2) 4346 APPLIED OPTICS Vol. 43, No August 2004

3 where Freq i is the quantized frequency interval for which the PSD can be approximated as constant, pix is the pixel angular size, and PDS i is the value of the PSD in the ith frequency interval. Based on these spectral characteristics of the vibrations, the horizontal and vertical random-displacement vectors of the camera X t and Y t, respectively can be simulated: X t i A x,i sin 2 f x,i t x,i, (3) Y t A y, j sin 2 f y, j t y, j, (4) j where x,i and y,j are random variables equally distributed on 0,2 and A x,i and A y,j are amplitudes at temporal frequencies f x,i and f y,i, respectively, as calculated in Eq. 2. The time parameter t sec is related to the spatial coordinates by x nominal pixel V nominal pixel s t sec, (5) where V nominal is the nominal scan velocity without vibrations and x nominal is the equally spaced spatial sample positions of the sensor array. Substituting Eq. 5 into the temporal displacement vectors, Eqs. 3 and 4 yield, respectively, the spatial displacement vectors X x nominal i Y x nominal j x nominal A x,i sin 2 f x,i V nominal x,i, (6). (7) A y, j sin 2 f y, j y nominal V nominal y, j Equations 6 and 7 hold only for V nominal V vibs, where V vibs is the vibration-induced velocity, for which the expression x nominal V nominal approximates the time of sampling at x nominal in the presence of vibrations. Equations 6 and 7 show that the spatial sensor vibrations are affected by the nominal scan velocity. The sensor array was modeled in the simulation by an M N matrix Fig. 1. This matrix is used to scan a high-resolution target image, representing the continuous before sampling object. The location of the matrix with respect to the target image is calculated for each scanning step according to the scan velocity V nominal and the vibration-induced displacement vectors X x nominal and Y x nominal. The motion of the matrix is discrete at 1 D jumps, where D is the minimal interval of the camera motion in pixels. When the array is placed on the target image representing the scene each cell averages the highresolution values over D D pixels, representing the sampling features of the CCD camera. Two target images used in this work for demonstrations are presented in Fig. 3. In Fig. 4 we demonstrate the effects of two-dimensional vibrations on staggered TDI images for two extremely different scan velocities: 2200 pixels s in Fig. 4 a and 350 pixels s in Fig. 4 b. It is clear that for lower scan velocities the spatial frequencies of the vibration increase, and the geometrical distortion and image Fig. 3. Target original high-resolution images: a stripes, b port. warping are more substantial. The reason is that, in the practical case of low-frequency vibrations, for lower scan velocities the sensor is subjected to a bigger extent of motion due to the longer periods of integration time and time between acquisitions of the odd and even fields. As discussed in Section 2 the vibrations cause three major effects in the staggered TDI image; all are horizontally space variant: a Staircase or comb effects, which are shifted locations of the image objects in one field with regard to the other, as a result of the time delay between the exposures of the fields. As can be seen in the examples in Fig. 4, these artifacts are more apparent in regions with vertical features. b Nonuniform sampling of each field as a result of the shifted locations of the sensor in each column in the TDI process. This effect is demonstrated in the image of a single odd field shown in Fig. 5 b compared with its nonvibrated version shown in Fig. 5 a. The nonuniform sampling resulting from the horizontal component of the vibrations is noticeable in the vertical lines region, and the transverse warping due to the vertical component of the vibrations can be seen by the shape distortion of the horizontal lines. c Blur as a result of the smearing of image points caused by the motion during the exposures. The resulting spread of each pixel is an average of the spreads from all the N integrated exposures. 4. Motion Estimation Estimating the motion of the vibrations causing the image artifacts is the key element in the task of restoring the staggered TDI images. Motion estimation is a thoroughly investigated task with many applications image restoration, video compression, medical imaging. Stiller and Konrad 14 and Brown 15 reviewed various motion models and the different applications relevant for each model. These papers essentially refer to video sequences of staring arrays, but the basic concepts are applicable to the staggered TDI imaging as well. It is assumed here that the camera motion is a two-dimensional rigid-body motion. However, rotational motions can be considered with the tools given by Irani and Peleg. 16 Also, the maximum horizontal 1 August 2004 Vol. 43, No. 22 APPLIED OPTICS 4347

4 Fig. 4. Effects of two-dimensional vibrations on staggered TDI images for different scan rates: a 2200 pixels sc and b 350 pixels sc. Both images were simulated with N 16 and margin 3. velocity induced by vibrations is assumed to be small compared with the scan velocity. This assumption allows the use of the first-order approximation to the time parameter noted in Eq. 5. The motion-estimation process is divided into two main stages: estimating the field displacement vector FDV and assembling the camera-motion vector. The constant separation between the odd and the even sensors that equals N margin pixels illustrated in Fig. 1 causes a known constant time delay between the capture of the odd and the even fields. Therefore the horizontal and vertical vectors of the displacements between the fields, DX and DY, can be determined according to the displacement vectors of the camera as follows: DX i X i X i N margin, (8) DY i Y i Y i N margin, (9) where i is the column index. It was shown in Section 3 that each captured image column is an integration of N sensor positions. Therefore the camera-motion vector that forms the FDV s should be averaged with a kernel of the TDI length. The FDV can be estimated by various methods. 15,17 Here the differential method was chosen since it produces high-accuracy results with low computational cost. The general method applied was described by Horn and Schunck 18 and implemented as an iterative procedure 19 adequate for large displacements. The method is based on minimizing the squared error of the Taylor s series expansion of the difference between the captured odd and even fields. Based on the assumptions above the even column, f even x, y resembles its matched column in the odd field displaced DX, DY from it, f odd x DX, y DY, given that the object s intensity is not time dependent. For small displacements a Taylor ap- Fig. 5. a Single odd field simulated as captured by a nonvibrated TDI sensor with N 16. b Field degraded by two-dimensional vibrations, demonstrating the resulting nonuniform sampling. The distorted locations of the vertical lines result from the horizontal vibrations component, and the distorted shapes of the horizontal curves result from the vertical component APPLIED OPTICS Vol. 43, No August 2004

5 proximation is convenient for estimating the squared error Err DX, DY of the two matched columns. After the second-order terms are neglected, Err DX, DY f even x, y f odd x, y f oddx x, y DX f oddy x, y DY 2, (10) where R is the column size, f even x,y and f odd x,y are the captured even and odd columns, and f x f x and f y f y are partial derivatives of the image f in the scan direction and perpendicular to it, respectively. Taking the derivatives of Eq. 10 in both axes and finding the minimum error by equating each to zero yield the basic equation for the estimation process: Equations 12 and 13 can be solved recursively to form the camera-motion vectors given knowledge of the camera position at N margin 1 steps. Since most of the vibration energy is at low frequencies, it can be assumed that in regions where the FDV is close to linear the camera s displacement vector is approximately linear too. Therefore construction of the camera s displacement vectors at each axis is based on these following stages: 1 locating the N margin 1 section of the FDV, which is the closest to a straight line in the variance sense, 2 establishing the camera s position in this region as equal to the linear fit of the displacement vector in the chosen region, 3 constructing the camera s displacement vectors forward and backward from the section chosen above by using Eq. 12 and 13. f 2 oddx x, y f oddx x, y f oddy x, y f oddx x, y x, f oddy x, y y R DX f 2 oddy x, y DY f even x, y f odd x, y f oddx x, y f even x, y f odd x, y f oddy x, y. (11) Equation 11 is solved for each column in the odd field, thus producing its unique displacement DX, DY from its matching column in the even field. Since the Taylor expansion holds only for sufficiently small displacements, an iterative procedure is implemented to allow larger displacements 19 : a Initially assume zero displacement estimation between the matched columns in the fields. b Calculate the displacement between the columns by using Eq. 11 and add it to the existing displacement estimate. c Shift the column of the even field toward the location of the column of the odd field according to the current displacement estimate. d Return to phase b until the residual difference between the consecutive displacement estimations approaches zero. Since this method is sensitive to noise, a prior step of field smoothing is performed with a Gaussian low-pass filter. 19 The first-order horizontal and vertical derivative operators used were h x 1 1 and h y 1 1 T, respectively. Detailed considerations for choosing these operators are presented by Elad et al. 20 Constructing the camera-motion vectors is based on the FDVs estimated above, when Eqs. 8 and 9 are used, rearranged to form the following recursion equations: X i N margin X i DX i, (12) Y i N margin Y i DY i. (13) Figure 6 shows examples of two sections chosen from an FDV. Region 1 is the most linear section in the FDV, whereas region 2 is a nonlinear section. The influence of a proper selection of a linear region in the FDV on the final estimated camera s motion vector is demonstrated in Fig. 7, where this vector was constructed from the region choices of Fig. 6. Both of the estimated curves have a dc error, which is meaningless in the restoration procedure. However, constructing the vector based on region 2 yielded a cyclic error with amplitudes as great as 2 pixels, while using region 1 resulted in a flat error curve. Fig. 6. Examples of two sections chosen from an FDV for construction of the initial camera displacement guesses: 1, linear region; 2, nonlinear region. Top, camera motion vector. Bottom, error. 1 August 2004 Vol. 43, No. 22 APPLIED OPTICS 4349

6 which is the set of all images having the property i. These sets are often closed and convex. The estimation of f belongs to the solution set m C s C i, i 1 assuming that this intersection is not empty. Denoting P i the projection operator of an arbitrary image onto the convex set C i, the POCS theorem main recursion is f l 1 P m P m 1...P 1 f l, l 0, 1, 2,.... (15) Fig. 7. Influence of region selection shown in Fig. 6 as the basis for the construction of the camera-motion vector N 16 and margin Image Restoration As mentioned above the PSF is space variant along the scan axis. An initial stage of the restoration process is calculation of the space-variant PSF. The PSF is calculated for each column according to Eq. 1 by using its displacement given by the estimated cameramotion vectors. The total PSF is a convolution between the PSFs of the vibrations and the camera: PSF total PSF vibrations * PSF camera. (14) PSF vibrations is calculated for each column according to Eq. 1 by use of its displacement given by the estimated camera motion vectors. We use here the projection onto convex sets POCSs restoration technique because it is suitable for space-variant motion blur. Implementation for restoration of an image sequence degraded by vibrations by superresolution is shown by Stern et al. 24 With this method the high-resolution ideal image f is assumed to be an element of an appropriate Hilbert space. It is assumed that m properties constraints 1, 2,..., m of the image f are known a priori. For each a priori piece of information i there is an attributed set C i, We use the following closed, convex constraint set for each pixel of the low-resolution image g k m 1, m 2 where k 1, 2 for the two interlaced fields : C k m 1, m 2 f n 1, n 2 : f r m k 1, m 2 0 m 1, m 2, k, (16) where f n 1, n 2 is the estimated high-resolution image and r k f m 1, m 2 represents the residual image of each simulated image field g k m 1, m 2 with respect to the observed original field g k m 1, m 2 : r k f m 1, m 2 g k m 1, m 2 g k m 1, m 2. (17) The simulated image fields are formed according to the simulation described above, when the estimated high-resolution image f l at the lth iteration and the estimated motion vectors are used. The quantity 0 m 1, m 2,k represents the statistical confidence with which the actual image is a member of the set C k. Here it reflects the uncertainty of the PSF in addition to the noise statistics. The projection operator P k m 1, m 2 on the estimated highresolution image f l n 1, n 2 can be described as follows: P k m 1, m 2 f l n 1, n 2 f l n 1, n 2 f l r k m 1, m 2 0 m 1, m 2, k h PSF,k n 1, n 2 ; m 1, m 2 ; 2 h PSF,k o 1, o 2 ; m 1, m 2 o 1 o 2 r k f l m 1, m 2 0 m 1, m 2, k 0; 0 m 1, m 2, k r k f l m 1, m 2 0 m 1, m 2, k, (18) f l r k m 1, m 2 0 m 1, m 2, k h PSF,k n 1, n 2 ; m 1, m 2 ; 2 h PSF,k o 1, o 2 ; m 1, m 2 o 1 o 2 r k f l m 1, m 2 0 m 1, m 2, k 4350 APPLIED OPTICS Vol. 43, No August 2004

7 Fig. 8. a True versus the estimated FDV in the horizontal scan direction; b error between the true and estimated vectors in a ; c and d same as a and b but for the vertical displacements. Fig. 9. a True versus the estimated camera displacement vector in the horizontal scan direction; c error between the true and estimated vectors in a ; b, d same as a and c but for the vertical displacements. 1 August 2004 Vol. 43, No. 22 APPLIED OPTICS 4351

8 where h PSF,k n 1, n 2 ; m 1, m 2 is the space-variant PSF and o 1, o 2 are high-resolution coordinates for which h PSF,k n 1, n 2 ; m 1, m 2 is not zero. In addition to the principal constraint set Eq. 16 a second projection was used to bound the estimated pixel gray levels in the range 0,1 : C A f n 1, n 2 :0 f n 1, n 2 1. (19) To compensate for oversampled and undersampled regions due to vertical field displacements, a smoothing operation is performed by an additional simple projection that bandlimits the signal 25 : C F f n 1, n 2 : 1D,n1 f n 1, n 2 0, for n1 ε, (20) where 1D,n1 is the one-dimensional Fourier transform of a column and ε is the band limitation of the spatial frequency n1, which forces the smoothness. The POCS procedure converges as the iterative procedure progresses. A stopping condition can be set on the residual energy e l : l e k l g k g k, (21) where is a sufficiently small number. 6. Results The restoration process developed in this study was implemented to both simulated and real-degraded images. The purpose of the simulation is to enable a comparative evaluation of the results given the original image and the true motion vectors. A. Simulated Images Figures 8 a and 8 c present, respectively, the horizontal and vertical true FDVs in Fig. 4 a versus their estimations. It can be seen that the estimation errors shown in Figs. 8 b and 8 d are bound within 0.4 pixels. Figure 9 a and 9 b present, respectively, the horizontal and the vertical true camera displacement vectors in Fig 4 a versus their estimations. It can be seen that the amplitudes of the estimation errors for both the horizontal and the vertical directions shown in Figs. 9 c and 9 d, respectively, are bound within 0.5 pixels with dc errors of approximately 0.4 pixels for the horizontal direction and 1.4 pixels for the vertical direction. Figure 10 a is the bicubic interpolation of the odd field to the highresolution dimensions, presented here for warping and resolution comparison with the restored image. Figure 10 b is the POCS-restored image when the true motion vectors are used, and Fig. 10 c is the restored image when the estimated vectors are used. In Figs. 10 b and 10 c the geometrical artifacts are corrected and the improvement in resolution is noticeable. The resolution of the image restored by the true motion vectors is slightly better than the one restored by the estimated vectors. Fig. 10. a Odd field of Fig. 4 a interpolated to the highresolution dimensions for comparison; b, c restored highresolution images with the true and the estimated motion vectors, respectively. B. Real-Degraded Images Figure 11 a presents an image captured by a Tadir thermal-imaging system in the m wavelength range manufactured by ELOP, Electro Optic Systems 4352 APPLIED OPTICS Vol. 43, No August 2004

9 7. Conclusions The staggered TDI scanning sensors offer high resolution and signal-to-noise-ratio characteristics. However, vibrations during the imaging process that frequently occur in thermal-imaging systems can severely degrade the captured image quality. The degradation properties are space variant in the scanning direction. The effects of vibrations on images captured by staggered TDI sensors have been analyzed, and a methodology of image reconstruction has been proposed. This methodology includes two main steps: motion estimation by using the differential method with subpixel accuracy and image restoration by using the POCS algorithm, which utilizes the estimated motion vectors. The tools used here for the image reconstruction have been tailored to the specific characteristics of the staggered TDI problem. The three major effects of vibrations, the staircase comb effects, image warping, and shiftvariant blur, have been eliminated or reduced by the restoration process. The algorithm presented here has been tested on simulated images and on realdegraded images. Both cases resulted in imagequality improvement. The authors thank Sagi Faran, Aviv Levi, and Yoav Zimerman for important contribution to this research. The authors acknowledge the support of the Wolfson Foundation for the purchase of the Tadir thermal-imaging system and the support of the Magneton Program of the Ministry of Commerce and Industry with our partner ELOP Israel Electrooptics Industries Ltd for supporting this specific research. Fig. 11. Restoration of a real-degraded image; a image captured by a staggered TDI thermal-imaging system with 6 TDI stages and a margin of 29 pixels, b image restored by the POCS method. Ltd. This is a staggered TDI camera with 6 integration stages and a margin of 29 pixels between the odd and the even sensors. The electrical cooling system of the detectors vibrates the camera. It can be seen in the restored image presented in Fig. 11 b that the geometric distortions are corrected while the blur correction is hardly noticeable. The reason is that in the degraded image itself the relatively big margin of 29 pixels causes significant geometric distortions several pixels, while the relatively small number of integration stages 6 pixels causes less significant blur at subpixel extent. References 1. J. D. Spinhirne, V. S. Scott, J. Cavanaugh, S. Palm, K. Manizade, J. W. Hoffman, and R. C. Grush, Preliminary space flight results from the uncooled infrared spectral imaging radiometer ISIR on shuttle mission STS-85, in Infrared Detectors and Focal Plane Arrays V, E. L. Dereniak and R. E. Sampson, eds., Proc. SPIE 3379, A. Fenster, D. W. Holdsworth, and M. Drangova, Threedimensional imaging using TDI CCD sensors, in Charge- Coupled Devices and Solid State Optical Sensors II, M. M. Blouea, eds., Proc. SPIE 1447, V. Sankaran, C. M. Weber, and K. W. Tobin, Inspection in Semiconductor manufacturing, in Webster s Encyclopedia of Electrical and Electronic Engineering Wiley, New York, 1999, Vol. 10, pp H. S. Wong, Y. L. Yao, and E. S. Schlig, TDI charge coupled devices: design and applications, IBM J. Res. Dev. 36, M. Berger, Y. Lauber, M. Citroen, and J. M. Topaz, Cos 4 compensation and optimal TDI operation in multielement linear TDI IR detectors, in Infrared Technology and Applications XXVIII, B. F. Andresen, G. F. Fulop, and M. Strojnik, eds., Proc. SPIE 4820, M. Zucker, I. Pivnik, E. Malkinson, J. Haski, T. Reiner, D. Admon, M. Keinan, M. Yassen, I. Sapiro, N. Sapir, and A. Fraenkel, Long mid-wave infrared detector with time delayed integration, in Infrared Technology and applications XXVIII, B. F. Andersen, G. F. Fulop, and M. Strojnik, eds., Proc. SPIE 4820, D. F. Barbe, Charge-coupled imaging devices, in Charge- Coupled Devices: Technology and Applications, R. Melen and D. Buss, eds. Institute of Electrical and Electronics Engineers, New York, 1977, pp O. Hadar, I. Dror, and N. S. Kopeika, Image resolution limits resulting from mechanical vibrations. Part IV: Real time nu- 1 August 2004 Vol. 43, No. 22 APPLIED OPTICS 4353

10 merical calculation of the OTF and experimental verification, Opt. Eng. 33, Y. Yitzhaky, G. Boshusha, Y. Levi, and N. S. Kopeika, Restoration of an image degraded by vibrations using only a single frame, Opt. Eng. 38, Y. Yitzhaky and A. Stern Restoration of interlaced images degraded by variable velocity motion, Opt. Eng. 42, J. Miettinen and H. Ailisto, Using a time delay and integration camera at nonzero viewing angle under vibration condition, Opt. Lasers Eng. 31, G. Wolberg and R. P. Loce, Restoration of images scanned in the presence of vibrations, J. Electron. Imaging 5, W. L. Barnes, T. S. Pagano, and V. V. Salomonson, Prelaunch characteristics of the moderate resolution imaging spectroradiometer MODIS on EOS-AM1, IEEE Trans. Geosci. Remote Sens. 36, C. Stiller and J. Konrad, Estimating motions in image sequences a tutorial on modeling and computation of 2D motion, IEEE Signal Process. Mag. 16, G. L. Brown, A survey on image registration techniques, ACM Comput. Surv. CSUR 24, M. Irani and S. Peleg, Improving resolution by image registration, CVGIP: Graphi. Models Image Process. 53, S. Raiter, A. Stern, O. Hadar, and N. S. Kopeika, Image restoration from camera vibration and object motion blur in infrared staggered time-delay and integration systems, Opt. Eng. 42, K. P. Horn and B. G. Schunck, Determining optical flow, Artif. Intell. 17, C. L. Luengo Hendriks and L. J. van Vliet, Improving resolution to reduce aliasing in an under sampled image sequence, in Sensors, Cameras, and Systems, for Scientific, Industrial, and Digital Photography Applications, M. M. Blouke, G. M. Williams, N. Sampat, and T. Yeh, eds., Proc. SPIE 3965, M. Elad, P. Teo, and Y. Hel-Or, Optimal filters for gradient based motion estimation, in Proceedings of the International Conference on Computer Vision, Corfu, Greece, 1999 Institute of Electrical and Electronics Engineers, New York, 1999, pp D. C. Youla and H. Webb, Image restoration by method of convex projections: Part 1 theory, IEEE Trans. Med. Imaging MI-1, H. Stark and P. Oskoui, High-resolution image recovery from image-plane arrays, using convex projections, J. Opt. Soc. Am. A 6, A. J. Patti, M. I. Sezan, and A. M. Teklap, Super resolution video reconstruction with arbitrary sampling lattices and nonzero aperture time, IEEE Trans. Image Process. 6, A. Stern, Y. Porat, A. Ben-Dor, and N. S. Kopeika, Enhancedresolution image restoration from a sequence of low-frequency vibrated images by use of convex projections, Appl. Opt. 40, H. Stark and Y. Yang, Vector Space Projections: A Numerical Approach to Signal and Image Processing Neural Nets and Optics Wiley, New York, APPLIED OPTICS Vol. 43, No August 2004