Improving Signal- to- noise Ratio in Remotely Sensed Imagery Using an Invertible Blur Technique

Improving Signal- to- noise Ratio in Remotely Sensed Imagery Using an Invertible Blur Technique Linda K. Le a and Carl Salvaggio a a Rochester Institute of Technology, Center for Imaging Science, Digital Image and Remote Sensing Laboratory, Rochester, New Yor, USA ABSTRACT When planning an optimal image collection strategy, a choice needs to be made between a higher signal- to- noise ratio (SNR) and spatial fidelity. A higher SNR requires longer integration times; however, motion blur will be introduced into the image. An image collection methodology is proposed such that the benefits of both can be achieved. By collecting a sequence of images at varying integration times, the SNR is increased, while motion blur is purposefully allowed. The spatial resolution and geometric fidelity can be restored through the application of a null- filling deblurring technique that combines non- invertible MTFs of each image to create an invertible MTF [1]. The deblurring technique is applied to the removal of motion blur due to the movement of the imaging platform, rather than blur due to moving objects in the scene. Using longwave infrared images collected from RIT s WASP airborne imaging system with simulated motion blur, this methodology is successfully demonstrated with reasonable increases in integration time. Keywords: PSF estimation, deconvolution, deblur, SNR, motion blur, integration time, remote sensing, inverse filtering 1. INTRODUCTION One of the tradeoffs with imaging systems is choosing between the signal- to- noise ratio (SNR) and spatial blur due to motion. A longer integration time will yield a higher signal- to- noise ratio, as more photons will be collected by the detectors. However, this will be at the expense of spatial fidelity, as the resulting image would have increased motion blur. Motion blur can result from the movement of objects in the scene or stationary objects imaged by a moving imaging platform. Further author information: Emails: lindale@yahoo.com, salvaggio@cis.rit.edu

Agrawal et al. [1] proposed a technique to invert motion blur. A sequence of multiple frames is taen at different exposures, where each frame has an individual modulation transfer function (MTF), or frequency representation of the point spread function (PSF), which is not invertible. The nulls of each individual non- invertible MTF are filled with the data points from the other individual MTFs, thus resulting in a combined MTF that is invertible. The idea proposed in this deblurring technique can be applied to a proposed imaging collection methodology that improves the SNR by allowing for longer integration times while motion blur is removed to restore spatial fidelity. This proposed methodology will be applied to a sequence of longwave infrared images collected at various integration times using an airborne remote sensing imaging system. Instead of attempting to remove motion blur from moving objects within the scene, motion blur due to the movement of the imaging platform will be removed from stationary objects in the scene. This eliminates the need for segmentation of the blurred object from a static bacground. However, registration between the successive images will be necessary. In addition, the motion blur expressed in terms of the image space can be accurately determined by calculating the velocity of the airborne imaging platform. 2. BACKGROUND The observed image output from an imaging system can be simply described as: i = f * h + n (1) where denotes each image in the sequence taen of the same scene, but with varied exposures, f is the input, or scene, * denotes the convolution operator, h is the impulse response, or PSF, of the imaging system, and n is the uncorrelated additive noise introduced by the imaging system. Similarly, the frequency representation of the image can be described as: I ( w) = F( w) H ( w) N ( w) + Assuming that the motion of the imaging platform has a constant velocity, the PSF of each image in the sequence can be simply estimated as a rectangle function (or box function) in the spatial domain. The width of the rectangle function is proportional to the motion blur expressed in terms of the image space. With GPS systems, this can be accurately calculated using nowledge of the velocity of the imaging platform in conjunction with the integration time and ground sample distance (GSD) of the image to convert to the image space: (2) b v * T = (3) GSD

where b is the width of the rectangle function or PSF in the spatial domain, v is the velocity of the imaging platform, T is the integration time, and GSD is the ground sample distance of the images. Thus, the Fourier transform of the PSF (i.e. the MTF) is a sinc function with a width that is inversely proportional to the width of the PSF. Agrawal et al. [1] describes the deblurring process, which involves filling the nulls of the individual non- invertible MTFs to create a combined invertible MTF: V ( w) = N H = 1 *( w) H ( w) 2 (4) where V (w) is the frequency domain representation of the deconvolution filters, and H (w) are the individual non- invertible MTFs. Let the denominator of Equation 4 be defined as N P 2 ( w ) = = 1 H ( w ) 2 (5) It can be observed that the individual non- invertible MTFs, H (w), will contain nulls that cause the deconvolution filter, V (w), to become unstable due to the division operator. However, when the individual MTFs from multiple images with varying exposures are combined, an invertible MTF, P 2 (w), is obtained. Once the deconvolution filters with the combined invertible MTF are created, they can be simply applied to the Fourier transform of the image to obtain the final deblurred image [1]: ^ F( w) = N = 1 I ( w) V ( w) (6) The resulting deblurred image is, of course, still an estimate of the ideal, un- degraded image of the scene. It should be noted that the noise term in Equation 2 was eliminated, as all images in the sequence were assumed to have the same noise power. 3. IMAGE MOTION BLUR SIMULATION The proposed collection methodology was intended to be demonstrated with LWIR imagery collected by the Wildfire Airborne Sensor Program (WASP) sensor built and operated by the Center for Imaging Science at the Rochester Institute of Technology. Actual data collection using the proposed methodology was not available at the time of writing this paper. Thus, motion blur caused by increases in integration time was simulated to generate a sequence of images.

Original Image in spatial domain Image Simulation i Tae Fourier transform I Multiply with blurring sinc with width b_sim 1-1 Multiply with blurring sinc with width b_sim 2-1 Multiply with blurring sinc with width b_sim 3-1 I 1 I 2 I 3 Image Deblurring Deblurring sinc, H (w), with widths b_psf 1-1, b_psf 2-1, b_psf 3-1 ^ Deblur image using: V ( w) F( w) = = N = 1 N = 1 H I ( w) V ( w) *( w) H ( w) 2 ^ F( w) Tae inverse Fourier transform ^ f Resulting deblurred image Figure 1: Overview diagram of the process to simulate motion blurred imagery and the proposed deblurring process using a null- filling technique. Figure 1 shows an overview of the process to simulate blurred images and the deblurring process using the null- filling technique. Two sets of sinc functions are created one set is used to induce motion blur in the original image, and the other set is used for deblurring. As will be described later, the deblurring MTF (i.e. a sinc with a width that is inversely proportional to b_psf ) is not always made to equal the blurring MTF (i.e. a sinc with a width that is inversely proportional to b_sim ). Images blurred by motion from the imaging platform are simulated by multiplying the blurring PSFs with the Fourier transform of the original image. In order to demonstrate the proposed methodology while avoiding additional impacts from errors in image registration between three different images, only one LWIR WASP image is used

to generate the images with simulated blur. Thus, three simulated blurred images, with varying levels of blur, are created from the one original image (see Figures 4a, 4b, 4c, and 4d). The estimated deblurring MTFs, H (w), are created from sinc functions with widths that are inversely proportional to b_psf and are used to calculate the deconvolution filters V (w). That result is then applied to the simulated blurred images, I (w), to produce a deblurred image in the frequency domain. The inverse Fourier transform is taen to produce the final resulting deblurred image in the spatial domain. 4. IMPLEMENTATION AND RESULTS The motion blur simulations were approached in two ways. The first approach involved blurring all three images in the dataset. The second approach investigated the improvements when the dataset included one sharp image and two images with simulated motion blur. In the latter method, the sharp image helps the deblurring process for the other two blurred images, while the two blurred images with increased integration times improve the SNR. The results from both approaches are further described below. Figure 2 shows the original LWIR WASP image that was used for the simulated motion blurred images. Table 1 shows the image collection specifications for the original LWIR image and the calculated image- space motion blur (ref. Equation 3). (a) (b) Figure 2: The original LWIR image from the WASP sensor at the Rochester Institute of Technology (a) and a magnified area (b) are shown.

Ground sample distance GSD [m] Velocity v [m/s] Integration time T [s] PSF width b [pixels] Original image 0.91 68.91 0.016 1.21 Table 1: Specifications and calculated values for the original LWIR image from the WASP sensor at the Rochester Institute of Technology. In one of the simulation trials, the integration time of the original image was increased by 2x, 3x, and 4x. Figure 3 shows a plot of the blurring MTFs used to induce motion blur. In this case, the deblurring MTFs were made to match the blurring MTFs, and the plots for each image would just overlap each other. It can be observed that the deblurring MTFs for each of the three simulated blurred images contain zeros and would result in an unstable deconvolution filter. However, the combined MTF (see Figure 3) is invertible. As expected, the resulting deblurred image is exactly the same as the original image and has a root- mean- square error (RMSE) of 0.00 digital counts. Figure 4 shows a small region of the images with induced motion blur, the original image, and the resulting deblurred image. Figure 3: The 3 simulated blurring MTFs, H =[2x, 3x, 4x], that correspond to increases in the integration time. For this particular simulation trial, the deblurring MTFs were made equal to the blurring MTFs. P(w), the combined invertible MTF, is also shown.

(a) (b) (c) (d) (e) Figure 4: Magnified areas of the images with simulated blur resulting from increasing the integration time by 2x (image a), 3x (image b), and 4x (image c). The original image (d) and the resulting deblurred image (e) are also shown. When using a sequence of real imagery collected with varying integration times, some error in the PSF estimation is expected. This can be due to slight errors in the determination of the imaging platform s velocity, variations in GSD between the center and edge of the image, etc. Thus, simulation trials were run such that the deblurring MTFs were not made to match the blurring MTFs used to induce motion blur. Similar to simulation trial previously mentioned, Figure 5 shows a plot of the blurring MTFs that correspond to increases in integration time by 2x, 3x, and 4x of the original image. A systematic 2% error was introduced into the PSF estimation, thus the deblurring MTFs were made as sinc functions that are 2% wider than the blurring MTFs (see Figure 5).

Figure 5: The 3 simulated blurring MTFs, H =[2x, 3x, 4x], that correspond to increases in the integration time. For this particular simulation trial, a 2% error was introduced in the PSF estimation. Thus, the deblurring MTFs, H =[2.04x, 3.06x, 4.08x], are 2% wider than the corresponding blurring MTFs. The combined invertible MTF, P(w), is generated using the deblurring MTFs. The same three images with simulated blur were used as in the previous trial (see Figures 4a, 4b, and 4c). Figure 6 shows a comparison between the original image and the resulting deblurred image when a 2% error is introduced into the PSF estimation. The RMSE between the original image and the deblurred image is 0.09 digital counts. A visual comparison confirmed that the differences are slight and are difficult to observe unless the two images are flicered. In addition, the deblurred image is clearly sharper and an improvement over the three blurred images. A comparison of the frequency representations of the original image and the deblurred image show that, although the PSF estimation was off by 2%, the result is still very close to the original image even at high spatial frequencies (see Figure 7). The frequency representation of the deblurred image almost entirely overlaps with that of the original image.

(a) (b) Figure 6: A comparison between the original image (a) and the deblurred image (b) resulting from an increase in the integration time from the original image of 2x, 3x, and 4x and a 2% error introduced into the PSF estimation. (b) (a) Figure 7: The frequency representations of the original image, the three images with induced blur from integration times of 2x, 3x, and 4x ( Sim blur image1, image2, and image3, respectively), and the resulting deblurred image when a 2% error in the PSF estimation is introduced. An overview (a) is shown, as well as a magnified view of the lower frequencies (b) is shown in the inset.

As previously mentioned, the inclusion of one sharp image into the sequence of multiple frames was explored. Figure 8 shows the blurring MTFs used, where only subpixel motion blur was introduced into one of the images in the dataset (plotted as H1 PSF ). The other two images were induced with motion blur corresponding to increases in the integration time of 3x and 4x (plotted as H2 and H3 PSF, respectively). No PSF estimation errors were introduced. Figure 8: The simulated blurring MTFs, H =[0.6x, 3x, 4x], corresponding to the induced subpixel motion blur and motion blur that corresponds to the increases in the integration time. For this particular simulation trial, the deblurring MTFs were made equal to the blurring MTFs. P(w), the combined invertible MTF, is also shown. Figure 9 shows a small region of the images with induced motion blur. Since no PSF errors were introduced, the resulting deblurred image is exactly the same as the original image, as expected.

(a) (b) (c) (d) (e) Figure 9: Magnified areas of a sharp image (image a - only subpixel motion blur) and the images with simulated blur resulting from increasing the integration time by 3x (image b) and 4x (image c). The original image (d) and the resulting deblurred image (e) are also shown. Interestingly, the results are improved when one sharp image is included in the sequence of multiple frames, even when errors are introduced into the PSF estimation for all three images in the dataset. A PSF estimation error of 2% was introduced for a dataset consisting of one sharp image and two images with induced motion blur resulting from increases in the integration time of 3x and 4x. When compared to the original image, the resulting deblurred image has a RMSE of 0.04 digital counts, which is an improvement over the similar simulation previously described that resulted in a RMSE of 0.09 digital counts. Figure 10 shows the blurring MTFs and the deblurring MTFs with a PSF estimation error of 2%. Figure 11 shows the deblurred image that resulted from applying the deblurring process on the one sharp image (Figure 9a) and two simulated blurred images (Figures 9b and 9c). Differences between the two images were difficult to visually locate even when flicering.

Figure 10: The simulated blurring MTFs, H =[0.6x, 3x, 4x], corresponding to the induced subpixel motion blur and motion blur that corresponds to the increases in the integration time. For this particular simulation trial, a 2% error was introduced in the PSF estimation. Thus, the deblurring MTFs, H =[0.61x, 3.06x, 4.08x], are 2% wider than the corresponding blurring MTFs. The combined invertible MTF, P(w), is generated using the deblurring MTFs. (a) (b) Figure 11: A comparison between the original image (a) and the deblurred image (b) resulting from a sequence dataset consisting of one sharp image and increases in the integration time from the original image of 3x, and 4x and a 2% error introduced into the PSF estimation of all three images.

Figure 12 shows the frequency representations of the original image and the resulting deblurred image, which appear to almost completely overlap each other. The frequency representations of the dataset are also shown the one sharp image and two images induced with motion blur due to increasing the integration time by 3x and 4x. (b) (a) Figure 12: The frequency representations of the original image, the sharp image ( Image1 ), and the two images with induced blur from integration times of 3x and 4x ( Image2 and Image 3, respectively), and the resulting deblurred image when a 2% error in the PSF estimation is introduced. An overview (a) is shown, as well as a magnified view of the lower frequencies (b) is shown in the inset. Multiple simulation trials were run to determine the threshold for increases in the integration time that would still produce acceptable results; that is to determine the limit of how much blur the null- filling deblurring process could remove. Table 2 lists the integration time factors used for each simulation trial where all three images in each dataset were induced with motion blur. For example, Trial a had the least amount of blur induced, as the integration times were increased by 1x, 2x, and 3x from that of the original image. Conversely, Trial b had the most amount of blur induced, as the integration times were increased by 4x, 5x, and 6x.

Integration Time Factor Image 1 Image 2 Image 3 Simulated Trials more blur less blur a 1x 2x 3x b 2x 3x 4x c 3x 4x 5x d 4x 5x 6x Table 2: Integration time factors used for simulated trials where all three images were blurred. Trial a was induced with smaller amounts of blur, while Trial b was induced with larger amounts of blur. Figure 3 shows the root- mean- square error that was calculated between the resulting deblurred image and the original image. As expected, increases in the integration time, and thus increases in induced blur, cause increases in the RMSE value. Based on visual assessments of the resulting deblurred images, an approximate threshold was determined for the most blur that can still be successfully removed with the null- filling deblurring process. The effect of PSF estimation errors on RMSE and the blur threshold was also investigated. As expected, the integration time can be increased more, and consequently more blur is allowable, when the PSF estimation error is lower.

Figure 13: Simulation trials were run where all three images in each dataset were induced with simulated motion blur. The integration time factors used for each trial are shown in Table 2. The effect of PSF estimation errors is also shown. An approximate threshold was determined. As previously described, the results are better when one sharp image (or an image induced with a small amount of blur) is included in the sequence of multiple frames. For these simulation trials, one image was induced with a subpixel amount of blur that corresponds to 0.6x of the integration time of the original image. This equates to subpixel motion blur that is 4/5 of a pixel. Table 3 lists the integration time factors used for each simulation trial where the dataset consisted of one sharp image and two images in each dataset were induced with motion blur.

Integration Time Factor Image 1 Image 2 Image 3 Simulated Trials more blur less blur a 0.6x 2x 3x b 0.6x 3x 4x c 0.6x 4x 5x d 0.6x 5x 6x e 0.6x 6x 7x Table 3: Integration time factors used for simulated trials where the datasets consisted of one sharp image and two images induced with simulated blur. Trial a was induced with smaller amounts of blur, while Trial f was induced with larger amounts of blur. Figure 14 shows the RMSE values for multiple simulation trials where the dataset included a sharp image. Compared to the datasets where all three images are blurred, it can be observed that there is a higher allowance for increases in the integration time and the amount of motion blur while still producing a deblurred image that is very close to the original image. In addition, more error in the PSF estimation can be allowed. For example, the blurriest image in Trial c is a result from increasing the integration time by 5x (see Table 3). When including one sharp image, the resulting deblurred image has a low RMSE value even when the PSF estimation error is 2%, 3%, or 5% (see Figure 14). Conversely, when all three images are induced with motion blur, an acceptable deblurred image is only produced when the PSF estimation error is 2% (see Figure 13, Trial c ).

Figure 14: Simulation trials were run where the datasets consisted of one sharp image and two images induced with simulated motion blur. The integration time factors used for each trial are shown in Table 3. The effect of PSF estimation errors is also shown. An approximate threshold was determined. 5. LIMITATIONS There is a limit to how much blur can be removed, and thus, there is a limit to how much the integration time can be increased. Figure 15 shows examples of deblurred images that fell above the estimated blur threshold. In these cases, all three images in the dataset were induced with simulated motion blur and a 2% PSF estimation error was introduced. It can be observed that the noise increased and artifacts started to appear, especially in the low dynamic range areas. However, it should also be noted that these examples were induced with extreme amounts of blur, where the largest integration time factors are 9x (15b) and 12x (Figure 15c) more than the integration time of the original image. Thus, successful deblurring will be achieved for more reasonable increases in integration time.

(a) (b) (c) Figure 15: Examples of deblurred images resulting from simulated blur images that are induced with extreme amounts of blur. The original image is shown as a reference (a). (b) is the result from increasing the integration time by 7x, 8x, and 9x, while a 2% PSF estimation error was introduced. (c) is the result from increasing the integration time by 10x, 11x, and 12x, while a 2% PSF estimation error was introduced. 6. CONCLUSION The proposed collection methodology allows for the increase in SNR through a sequence of images taen at varying integration times. Thus, motion blur is purposefully allowed and then removed using the null- filling deblurring process. The simulations show that spatial fidelity can be restored when the integration times are reasonably increased, and the errors in PSF estimation are ept to a minimum. More leeway in both factors can be afforded when the sequence of images includes one sharp image. Furthermore, the entire deblurring process can be automated since the image- space motion blur can be calculated based on the recorded velocity of the imaging platform, and automated image registration algorithms can be utilized. 7. REFERENCES [1] A. Agrawal, Y. Xu, and R. Rasar, Invertible Motion Blur in Video, in ACM SIGGRAPH 2009 proceedings. [2] J. D. Gasill, Linear Systems, Fourier Transforms, and Optics, John Wiley & Sons, Inc., 1978. [3] J. R. Schott, Remote Sensing: The Image Chain Approach, Oxford University Press, 2nd Edition, 2007. [4] R. C. Gonzalez and R. E. Woods, Digital Image Processing, Pearson Education, Inc., 3rd Edition, 2008. [5] R. L. Easton, Jr., Fourier Methods in Imaging, Wiley, John & Sons, Incorporated, 2010.