Airborne test results for a smart pushbroom imaging system with optoelectronic image correction

Airborne test results for a smart pushbroom imaging system with optoelectronic image correction V. Tchernykh a, S. Dyblenko a, K. Janschek a, K. Seifart b, B. Harnisch c a Technische Universität Dresden, Department of Electrical Engineering and Information Technology, D-01062 Dresden, Germany b HTS GmbH, Am Glaswerk 3, D-01640 Coswig, Germany c European Space Agency, ESTEC, 2201 AZ Noordwijk, The Netherlands ABSTRACT Smart pushbroom imaging system (SMARTSCAN) solves the problem of image correction for satellite pushbroom cameras which are disturbed by satellite attitude instability effects. Satellite cameras with linear sensors are particularly sensitive to attitude errors, which cause considerable image distortions. A novel solution of distortions correction is presented, which is based on the real-time recording of the image motion in the focal plane of the satellite camera. This allows using such smart pushbroom cameras (multi-/hyperspectral) even on moderately stabilised satellites, e.g. small sat's, LEO comsat's. The SMARTSCAN concept uses in-situ measurements of the image motion with additional CCDsensors in the focal plane and real-time image processing of these measurements by an onboard Joint Transform Optical Correlator. SMARTSCAN has been successfully demonstrated with breadboard models for the Optical Correlator and a Smart Pushbroom Camera at laboratory level (satellite motion simulator on base of a 5 DOF industrial robot) and by an airborne flight demonstration in July 2002. The paper describes briefly the principle of operation of the system and gives a description of the hardware model are provided. Detailed results of the airborne tests and performance analysis are given as well as detailed tests description. Keywords: optoelectronic image correction, pushbroom scan camera, optical correlator, image motion tracking. 1. INTRODUCTION Observation satellites generally use pushbroom scanners with linear image sensors. The use of such scanners allows to reduce the size and cost of the imaging payload and has a number of other advantages, but requires high stability of the satellite attitude during scanning of the frame; otherwise the scanned image will be geometrically distorted. Figure 1 shows the results of a computer simulation of the attitude instability effect on a high resolution pushbroom scanned image. a b Fig. 1. Simulated pushbroom scanned images; a with stable attitude of the satellite; b in presence of attitude disturbances 550 Sensors, Systems, and Next-Generation Satellites VII, edited by Roland Meynart, Steven P. Neeck, Haruhisa Shimoda, Joan B. Lurie, Michelle L. Aten, Proceedings of SPIE Vol. 5234 (SPIE, Bellingham, WA, 2004) 0277-786X/04/$15 doi: 10.1117/12.510712

The simulation has been made with an attitude instability scenario, which is typical for a low orbit communication satellite. The sensitivity to the short-term attitude errors limits the application area of pushbroom scanners to dedicated imaging satellites with very precise attitude control. This compensates to a large extent the cost saving benefits of the pushbroom scanner solution. To solve this problem, the concept of a smart pushbroom imaging system (SMARTSCAN) has been proposed 1,2. It is based on the correction of the image distortions, caused by attitude disturbances, using the record of the image motion in the focal plane during scan of the frame. The image motion record is understood as two trajectories of the actual image motion with respect to both ends of the linear sensor. This record will be produced with auxiliary matrix sensors in the focal plane of the camera and an onboard optical correlator (Fig. 2) by determination and processing of the image shifts with respect to the moment t 0 of exposing of the first line. Camera field-ofview Main linear image sensor Auxiliary matrix image sensors Linear sensor image data Current image t=t 0 RAM Reference image Reference RAM image t=t 0 Current image Joint Transform Optical Correlator Joint Transform Optical Correlator Satellite Digital Processor Downlink Distorted image Image motion record Ground Computer Ground Station Corrected image Fig. 2. Smart pushbroom imaging system principle of operation The image motion record is derived from a 2D correlation of sequential image records generated by one or two matrix CCD sensors in the focal plane of the camera. As the processing of these sequential images (i.e. 2D correlation operation) has to be performed with high speed to cope with the complete spectrum of the satellites attitude motion, specific correlator hardware is required. SMARTSCAN incorporates a Joint Transform Optical Correlator 3, which is an optoelectronic device capable of performing extremely fast the spatial 2D correlation of two images. With currently available optoelectronic components it is possible to process up to 32000 pairs of image fragments (50x50 pixels) per second. This makes possible to determine the position of the focal plane image for the moments of every line exposing (pushbroom scan satellite camera with 1 meter resolution produces approximately 7000 image lines per second). To improve the mechanical robustness of the optical scheme, an innovative technique of self calibration has been proposed 4. This technique makes the optoelectronic correlator insensitive to mechanical deformations and thus in particular suitable for space applications. The SMARTSCAN pushbroom scan imaging system allows therefore to reduce the requirement to the satellite attitude stability. As a consequence, high quality images can be obtained also with satellites and imaging platforms which are not specially intended for imaging missions (low orbit communication satellites, micro- and nano-satellites with simplified attitude control, ISS). This expands the application area of pushbroom scanners to new domains of satellites with only moderate attitude stability. Proc. of SPIE Vol. 5234 551

2. HARDWARE MODEL OF SMART PUSHBROOM IMAGING SYSTEM 2.1 General structure of the model of smart pushbroom imaging system To prove the feasibility of the SMARTSCAN system concept and to estimate the expected performances, a hardware breadboard model of the smart pushbroom scan system has been produced. It includes a smart pushbroom camera model with linear and matrix image sensors, camera mount, an optical correlator model and a computer to record and process the image data (Fig. 3). Matrix Sensors Optical Correlator Model Camera Linear Model sensor Distorted image Image motion record Camera Mount Lens Portable PC (notebook) Visualisation and image correction software Corrected image 2.2 Model of smart camera with auxiliary image sensors Fig. 3. Hardware model of smart pushbroom imaging system The model of smart pushbroom camera (Fig. 4) consists of the body, lens, focusing elements and the set of image sensors in the focal plane. The main image acquisition sensor is a linear one (as in standard pushbroom scan camera). Besides it, two auxiliary matrix sensors are installed in the focal plane of a common lens (Fig. 4). To save development time and to cope with limited funding, standard video cameras have been used as the matrix sensors. Analogue video data (2x30 frames/s) Matrix Sensors (cameras) Linear sensor To Optical Processor Model Digital image data (240 lines/s) To PC Lens Fig. 4. Model of smart pushbroom scan camera During the imaging operations, the linear sensor produces a large image frame (line by line, as a normal pushbroom scan camera). The auxiliary matrix sensors produce two sequences of small images (video streams), which are processed by the optical correlator to produce the image motion record. Application of the standard video cameras as matrix image sensors limits the maximum sampling frequency of the image motion record to 30 Hz. To use this capability for both matrix sensors, the optical correlator should perform at least 60 correlations per second. Table 1. Main performances of the camera model Lens Linear sensor Auxiliary matrix sensors Dimensions / mass Tessar type: f = 75 mm; f/4; angular resolution - 187 µrad/pixel Resolution: 2048 pixels in line; lines rate: 240 lines per second Frame size: 640 x 480 pixels; frames rate: 30 frames per second 110 x 58 x 50 mm / 900 g 552 Proc. of SPIE Vol. 5234

2.3 Optical correlator model The optical correlator model (Fig. 5) consists of an optical unit and an electronic module. It performs the real time processing of two standard video signals from matrix sensors of the camera model and produces the image motion record. Electronics Standard cameras as image sensors Fig. 5. Optical processor model Optical unit with two optical Fourier processors The optical unit is based on the scheme of the Joint Transform Optical Correlator 3. To cope with the limited project funding and to save development time, it uses standard video cameras as the image sensors. This limits the image processing rate to 30 optical Fourier transforms per second or 15 correlations per second per one optical Fourier processors (one correlation requires two Fourier transforms). To provide the required 60 correlations per second, two optical Fourier processors have been manufactured and the image processing rate for each of them has been doubled by simultaneous processing of two image pairs (Fig. 6). Input image (two pairs of images) Correlation image (two pairs of correlation peaks) Fig. 6. Simultaneous processing of two image pairs Each pair of input images produce a pair of correlation peaks, which are then processed separately and two image shifts are determined. Table 2. Main performances of the optical correlator model Input Output Image processing rate Dimensions / mass (optical unit) Two standard video inputs (2 x 30 frames/s) Image motion record (via RS 232 interface) 60 correlations per second 210 x 62 x 30 mm / 500 g Proc. of SPIE Vol. 5234 553

3. AIRBORNE TESTS DESCRIPTION The tests have been performed on 31.07.2002 at the DLR (Deutsches Zentrum für Luft- und Raumfahrt) facilities in Oberpfaffenhofen near Munich. The test equipment has been mounted onboard a small, single engine turboprop aircraft. The camera model was mounted in one of the bottom ports of aircraft, all other equipment in a special rack in the cabin (Fig 7). Bottom port Fig. 7. Test equipment mounted onboard the plane Totally two test flights with each one hour duration have been made. During both flights, 9 imaging sessions have been performed. The flight altitude was approximately 2400 m with a velocity 240 km/h (67 m/s). The ground resolution of the camera model was 0.45 m per pixel. The main data flows during the tests are shown in Figure 8. Fig. 8. Flight test data flows During imaging the plane produced considerable attitude disturbances and vibrations. The distorted image from the linear sensor has been loaded to the PC. Two streams of small images from matrix sensors have been processed in real time by the optical correlator; the resulting image motion record has been loaded to PC too. Both the linear sensor image and the image motion record were reproduced on screen in real-time concurrent with the time of acquisition. After finishing of the imaging session, the distorted linear sensor images have been corrected on base of the stored image motion record. 554 Proc. of SPIE Vol. 5234

4. TESTS RESULTS 4.1 Direct results of the tests As a result of the tests, the complete end to end functionality of the SMARTSCAN imaging system has been demonstrated with real remotely sensed Earth images under airborne flight conditions. In each of 9 imaging sessions a linear sensor image (2048 x 2048 pixels) has been recorded and the corresponding image motion record has been produced by real time processing of the matrix sensor images by optical correlator onboard the airplane. The linear sensor images are distorted due to airplane attitude instability, vibrations and flight direction and velocity changes (Fig. 9). Image motion (pixels) Ideal IMR in the flight direction Real IMR in the flight direction Real IMR perpendicular to the flight direction Fig. 9. Distorted linear sensor image and image motion record (IMR) Number of the line Figure 9 shows the example of the record of the image motion with respect to one of the ends of the linear sensor (two components along and perpendicular to the flight direction). The starting point of record coincides with the image position in the moment of exposure of the first image line. In the ideal case, the image should advance in flight direction by exactly one pixel with any obtained line, so the ideal image motion record in the flight direction should be a straight line with 45 slope. The real record in the flight direction however has a different slope due to the airplane velocity deviation and local variations caused by attitude instability and vibrations. Perpendicular to the flight direction there should be no image motion in the ideal undisturbed case, so the corresponding component of image motion record should coincide with the horizontal axis. Actually there were considerably large deviations from the ideal case due to attitude instability and vibrations also in this case. 4.2 Image correction results The correction of the distorted images has been made after the flight on base of the image motion records produced inflight. The image motion record gives the position of every line of the distorted image with respect to the first line. With this information the coordinates of all pixels of the distorted image with respect to the first line have been calculated and the values of the corrected image pixels (positioned at regular intervals with respect to the first line) were determined by a standard 2D interpolation procedure. All 9 obtained pushbroom scanned images have been successfully corrected. Figure 10 represents the example of such an image correction result. Proc. of SPIE Vol. 5234 555

Distorted image Corrected image Fig. 10. Example of image correction on base of an image motion record Some residual distortions of the corrected image are caused mainly by vibration components above the Nyquist frequency. Such distortions can be eliminated by increasing of the correlation rate. A certain degree of smoothing of the corrected image is caused by the interpolation procedure itself and (on some parts of the image) by a high local velocity of image motion due to high vibration amplitudes (which are actually not expected on a satellite). 556 Proc. of SPIE Vol. 5234

4.3 Error analysis A direct determination of errors of the image motion record was not possible for the airborne tests, because no reference attitude and position data with required accuracy and bandwidth were available. To estimate the errors under these conditions, the following procedure has been applied. All matrix sensor images taken during scan of the linear sensor image (60 frames per second totally 512 fames for both matrix sensors) have been recorded in flight by high rate digital data recorder. After flight the sequence of images was played back to optical correlator, simulating the in-flight output of the matrix sensors. The playback was performed two times and the correlator produces two image motion records with the same set of matrix sensors images. For the second record, however, the optical correlator was forced to use different image fragments for image motion tracking (Fig. 11). IMR 1 IMR 2 Fig. 11. Two image motion records producing by tracking of two different fragments of the same set of images. In the ideal case the records, made by such a procedure, should coincide after subtraction of the initial shift. In our case some differences were observed (Fig. 12). δ x (pixels) σ x = 0.156 pixel Line number δ y (pixels) σ y = 0.192 pixel Line number Fig. 12. Difference between the image motion records, δ y in the flight direction, δ x perpendicular to the flight direction The errors of two records, made with by tracking of two different image fragments, can be considered generally independent (at least for high frequency component). In this case the difference between the records represents the vector sum of errors of both records. The mean square value of the difference for all 9 imaging sessions was generally within σ 0.25 pixel, what allows the conclusion that the error of image motion record is also within σ 0.25 pixel. Proc. of SPIE Vol. 5234 557

5. FUTURE ACTIVITY (SPACE VERIFICATION) The next step of the SMARTSCAN system verification should be the space testing. Possible space qualification options are governed by the following considerations. The current development status of the imaging system provides a functionality which gives a fully representative endto-end solution but with limited performances due to budget reasons. A direct one-to-one upgrade of this existing design to a space compatible version may give rather poor application performances, mainly due to the limitations in the camera optics (equivalent ground resolution only in the range of 100-150 meters). Moreover the computational speed of the optical correlator could be easily increased by using some more advanced and market ready opto-electronic components. It is therefore planned to qualify an advanced optical processor (see next chapter) in conjunction with a sufficiently performant (high-resolution) camera. The development of an advanced optical processor as a self-standing information processing component can be done rather independently from the camera development. To save development cost for the camera there are seen two possible solutions: - re-use of an existing spaceborne pushbroom camera concept - integration of Smart Scan elements into the scheme of an existing or already planned space camera, e.g. multi- /hyperspectral cameras; in this case only some changes in the design of the focal plane assembly and the interface electronics of this camera have to be covered. 6. FULL SCALE SYSTEM PERFORMANCES The image processing rate of the optical correlator model as described above is limited by 60 correlations per second due to application of standard video cameras as image sensors. With currently available optoelectronic components it is possible to improve dramatically the performance of the optical processor (Table 3) and thus performances of the whole SMARTSCAN system. Table 3. Estimated performance of the advanced optical correlator Image processing rate Errors of image shift determination Dimensions Mass Power consumption Up to 32000 correlations per second Up to 16000 correlations/s with double correlation (to improve accuracy) σ 0.15 pixel with single correlation σ 0.07 pixel with double correlation 170 x 60 x 40 mm (with housing) Within 500 g (with housing) Within 10 W with full performance (32000 correlations/s) Within 4 W with limited performance (8000 correlations/s) The power consumption depends on the actual performance of the correlator, with a limited correlation rate it can be decreased significantly. A fast optical correlator allows the recording of the image position for every line also for high resolution imaging missions, with extremely large frequency of the lines (7000 lines/s for low Earth orbit satellite with 1 m ground resolution). In principle, it allows to accept any angular velocity and acceleration of the spacecraft during image scan, making possible high quality imaging with linear sensor during deployment or moving of the spacecraft elements or even from spinning satellites. This feature is essential, if a pushbroom camera will be installed as a secondary payload on a satellite, which is not specially designed for imaging missions (for example, low orbit communication satellite). 558 Proc. of SPIE Vol. 5234

A high performance of the optical correlator also permits to produce the image motion record by tracking more than 2 image fragments. This improves the reliability and accuracy of the image motion recording. The accuracy can be improved too by performing the correlation for a pair of images two times (first correlation determines the shift between the images coarsely, then the position of the images is adjusted to minimize the shift to be measured by the second correlation, then the shift is determined finally by the second correlation). Table 4. Estimated performances of full-scale smart pushbroom imaging system Sampling rate of imaging motion record Errors of the record Dimensions and mass Power consumption 16000 samples/s for two fragment tracking 6400 samples/s for five fragments tracking 3200 samples/s for five fragment tracking and double correlations (to improve accuracy) σ 0.3 pixel with single correlation σ 0.15 pixel with double correlation The same, as for optical processor (Table 3) + cables Within 12 W with full correlations rate Within 5 W with 25% correlations rate 8. CONCLUSIONS As a result of the airborne tests the end-to-end functionality of the complete SMARTSCAN smart pushbroom imaging system has been proved with real remotely sensed Earth images under airborne flight conditions (optical, mechanical and electrical). The obtained high resolution images, which are distorted as a result of attitude instability and vibrations, have been successfully corrected on base of a real time record of the focal plane image motion, made with auxiliary focal plane image sensors and an onboard optical correlator. The errors of the image motion record (correspond to the residual image distortions after correction) were generally within σ 0.25 pixels. The application of the SMARTSCAN system allows high quality imaging with a pushbroom image sensor from satellites with only moderate attitude stability, including satellites, which are not specially designed for imaging missions (low orbit communication satellites; Space Station, etc.). ACKNOWLEDGEMENTS This research was supported by ESA/ESTEC Contract No. 14858/00/NL/PB. REFERENCES 1. K. Janschek, V. Tchernykh, S. Dyblenko, "Design concept for the secondary-payload Earth observation camera", in Sensors, Systems and Next-Generation Satellites III, Hiroyuki Fujisada, Joan B. Lurie, Editors, Proceedings of SPIE Vol. 3870, 78-86 (1999). 2. V. Tchernykh, S. Dyblenko, K. Janschek, B. Harnisch, Optical correlator-based system for the real time analysis of image motion in the focal plane of an Earth observation camera, in Algorithms and Systems for Optical Information Processing IV, Bahram Javidi; Demetri Psaltis; Editors, Proceedings of SPIE Vol. 4113, 23-31 (2000). 3. Jutamulia S. Joint transform correlators and their applications, Proceedings SPIE, 1812, pp. 233-243, 1992. 4. V. Tchernykh, K. Janschek, S. Dyblenko. Space application of a self-calibrating optical processor for harsh mechanical environment. Proceedings of 1 st IFAC Conference on Mechatronic Systems, 18-20 Sept. 2000, Darmstadt, Germany. Proc. of SPIE Vol. 5234 559