Dichoptic Fusion of Thermal and Intensified Imagery

Transcription

1 Dichoptic Fusion of Thermal and Intensified Imagery A. Toet, M.A. Hogervorst, M. van der Hoeven TNO Human Factors Kampweg DE Soesterberg, The Netherlands {lex.toet, maarten.hogervorst, marieke.vanderhoeven}@tno.nl Abstract - Subjects used the dichoptic combination of a monocular image intensifier (NVG) and a monocular uncooled microbolometer (LWIR) to detect and localise both visual targets and camouflaged thermal targets while moving through a dimly lit complex environment. The NVG imagery enabled the subjects to move freely through the environment with high accuracy, but did not mediate the detection of camouflaged thermal targets. The LWIR mode mediated the detection of camouflaged thermal targets but did not allow the detection of visual targets, and provided insufficient detail to allow accurate movement through the environment. Subjects were quite capable to dichoptically fuse the individual LWIR and NVG images, enabling them to detect all (visual and thermal) targets while moving accurately through the environment. We conclude that dichoptic fusion of NVG and LWIR imagery is quite feasible and is a simple way to provide observers with enhanced situational awareness in nighttime operations. Keywords: Image fusion, intensified imagery, NVG, LWIR, thermal imagery, dichoptic fusion. 1. Introduction Night vision devices provide the modern soldier with the ability to see, manoeuvre and shoot during periods of reduced visibility. Currently two types of night vision devices are in use: image intensifiers and thermal cameras. Image intensifiers capture ambient light (from the stars, moon or sky glow) and amplify it thousands of times by electronic means. In NVGs the image is shown via a phosphor display. The main advantage of NVGs are their small size, light weight, low power requirements and low cost. These properties have led to their widespread use. Thermal longwave infrared sensors detect the heat naturally radiated by all objects including terrain, roads, buildings, vehicles, and people and can distinguish differences in temperature between objects and their backgrounds. They provide the ability to detect targets at long range, and to look through smoke, dust, fog and other obscuring conditions. In contrast, image-intensifiers provide a short detection range and cannot see through smoke, dust, haze, and adverse weather at night. Recently small hand-held uncooled thermal cameras have become available, which operate on standard batteries. Because of their complementary nature, the combination of NVG and thermal imagery may significantly extend the range of conditions in which the dismounted soldier can operate. Compared to presently fielded NVGs, the addition of thermal imaging will provide the capability to see through battlefield obscurants (smoke screens) and to see in zero light conditions (enclosed spaces, buildings, caves). Fusion of thermal and intensified imagery can be achieved either optically or digitally. Digital image fusion has many advantages, since it enables the application of many different image enhancement techniques, the exchange of imagery with a command center and the inclusion of high level information from a battlefield management system. However, digital image processing requires significant power, reduces the NVG s pixel count, still needs significant technology advances, and the associated human factors are not yet fully understood. In contrast, optical image fusion maintains the high resolution of the NVG imagery, requires no technology breakthroughs, and provide simple to operate designs for which the human factors are fully understood. As a result, optical image fusion based on designs that use simple and proven approaches and technologies is the most promising approach to the development of short term fieldable systems. Currently a team of ITT Industries/Raytheon is developing an Enhanced NVG (ENVG) for the US Army, which is based on a simple optical overlay of the video image from a miniature LWIR thermal (microbolometer) camera on to the image from a conventional direct-view night vision monocular, with the output from both channels passing through a beam combiner [3, 5]. This approach is relatively simple and inexpensive, making it particularly suitable for infantry head-mounted systems. However, it still has some disadvantages, including a mismatch of the overlay at the edge of the field of view, the use of a complex optical path, and the need to adjust the relative brightness of both images manually. The human brain has the remarkable capability to fuse the two disparate images from both eyes in realtime into a single consistent percept. This capability has been succesfully deployed to fuse multi-modal imagery in gaze-contingent displays [8]. Here we propose to combine NVG and thermal imagery by presenting the

2 thermal image to one eye of the observer and the NVG (intensified visual) image to the other eye. Provided that there is a considerable overlap between both images, that they have a common magnification factor, and that their luminance and contrast are similar, the simultaneous dichoptic presentation of NVG and thermal imagery should result in a consistent fused percept [6, 7]. In this study we assess the perceptual effectiveness of dichoptic fusion of thermal (LWIR) and intensified (NVG) imagery by testing human target detection and navigation performance in a structured complex environment. Thereto we constructed a binocular device consisting of both a monocular image intensifier and a monocular uncooled microbolometer, with overlapping central field-of-views of equal magnification. This device allows subjects to dichoptically fuse the images from both sensing modalities. We tested the ability of subjects carrying this device to detect and localise both visual targets and camouflaged thermal targets while moving through a dimly lit complex environment. The experiments were performed for each of the sensing modalities (i.e. for NVG or LWIR) individually and for the dual mode (NVG+LWIR). The rest of this paper is as follows. In Section 2 we present the methods and equipment used in the experiments. Section 3 presents the results, and in Section 4 we will discuss the implications of our present findings. 2. Methods 2.1. Apparatus The image intensifier (Figure 1a) was a Mini N/SEAS monocular night vision goggle (NVG) from International Technologies (Lasers) Ltd (ITL). It provides a 1x magnification, and has a circular field-ofview (FOV) with a diameter of 40 deg, corresponding to about 2000 pixels. Hence, the angular resolution of the image intensifier is an order of magnitude larger than that of the thermal camera. The thermal camera (Figure 1b) was a handheld and uncooled Raytheon Thermal-Eye X100xp 7-14 µm longwave infrared camera with a focal plane array composed of 160x120 amorphous silicon microbolometers (elements that undergo a resistance change in response to absorbed radiation). The camera has a field-of-view of 17x12 deg. The image is displayed on a monochrome LCD viewfinder display consisting of 320x240 pixels, in a white-hot representation mode (hotter objects are shown brighter), and with automatic brightness and contrast gain. The viewfinder has a long (2 ) eye-relief which enables its use with glasses and masks. The NVG has a magnification factor of 1, whereas the magnification of the thermal camera is somewhat larger than 1. (a) Figure 1 (a) Mini N/SEAS monocular NVG and Raytheon Thermal-Eye X100xp 7-14 µm LWIR camera. We therefore adjusted the magnification of the NVG image to that of the thermal image by mounting a -9 diopter lens at about 4 cm in front of the NVG, which was deployed with its lens cap on. The lens cap contains a small pinhole in its center, which serves to reduce the amount of incoming light to such levels that the device can be used in broad daylight. The pinhole viewing mode provides a large depth of focus, allowing a simple optical correction of the magnification factor by means of a single lens. We determined the optical correction of the NVG image as follows. The NVG and the thermal camera were mounted close together with their optical axis parallel. Both cameras viewed a scene with targets consisting of red LEDs. These targets provided fiducial points that were clearly visible both in the NVG and in the thermal images. The image of the thermal camera was superimposed over the central part of the NVG image by a combination of a mirror and a half mirror. Finally, the image of the NVG was scaled to the size of the thermal image by placing a correction lens in front of the NVG. In practice, one would of course prefer a magnification of 1 for both devices, since this allows a dismounted soldier to correctly estimate distances and optic flow while moving through terrain. However, for the purpose of the present study the actual magnification is not important, since we merely wanted to investigate the human capability to dichoptically view and fuse NVG and thermal images Experimental conditions An artificial forest was created by arranging vertical foam sticks ( trees ) with a height of 1.5 m on a 6x8 rectangular grid with a grid size of 1 m (Figure 2a). This resulted in a course consisting of 5 parallel lanes with a length of about 7 m. The trees were clearly visible in the NVG image (Figure 2b). The thermal image showed these obstacles in much less detail (with less contrast : see Figure 2c). A large number of white paper sheets (size A4) were randomly distributed among the trees. Most of the paper sheets were blank and merely served as distractors. The thermal targets were 10 warm metal pairs of scissors placed underneath some of the blank sheet of paper (Figure 3a). The visual targets were 10

3 (a) (d) Figure 2 (a) Overview of the artificial forest with randomly distributed paper sheets. NVG and thermal image of the same scene. (d) Subject using the NVG and thermal camera to walk through the forest and search for targets. Figure 3 The search targets: (a) a real pair of metal scissors and drawn outlines of a pair of scissors. sheets of paper on which the outlines of a pair of scissors was drawn with black ink (Figure 3b). Before each run the metal pairs of scissors were briefly placed in a bucket full of warm water to raise their temperature significantly above room temperature. As a result they showed up clearly in the thermal images. However, they were invisible in the NVG image, because they were covered by opaque white sheets of paper. The drawn pairs of scissors were clearly visible in the NVG images, but could not be seen in the thermal images. Thus, we created a set of 10 targets that could only be detected by the thermal camera, and 10 targets that could only be seen with the NVG. The subject s task was to complete a course traversing the entire forest (Figure 2d), and to verbally report and indicate (by pointing with his feet or hands) all pairs of scissors that were detected on the way, as well as their their nature (visual or thermal). At the start of a run the subject was placed at the rim of the forest, facing the outer lane of trees. He was instructed to follow that lane to the end, turn at the end and take the next lane back, and so on, until all 5 lanes had been traversed. The subject was told to complete the course as quickly as was reasonably possible, without bumping into any trees. This task is similar to a situation in which a dismounted soldier has to cross a terrain covered with booby traps or buried landmines. The thermal properties of these devices (conductance, capacity) differs from those of the surrounding material (vegetation, soil), giving them a distinct thermal signature [1, 2, 4]. Each subject performed the task 3 times: once while viewing only with the NVG, once while viewing only with the thermal camera, and once while viewing with the NVG in front of one eye and with the thermal camera in front of the other eye. The subject simply used his hands to hold the cameras to his eyes (Figure 4a). In the condition in which the subject used both the NVG and the thermal camera he was first given the opportunity to adjust the position and line of sight of both cameras such that the thermal image fused with the central part of the NVG image (i.e. such that the centre of the displayed area in both image modalities showed the same room area : Figure 4d). All subjects could easily perform this adjustment. Also, the subjects had no problems fusing the thermal image with the NVG image. The position of the (visual and thermal) targets was changed randomly between each run. The order of the 3 different viewing conditions was balanced over the subjects using a Latin square design [9].

4 (a) The experiment was performed in a dimly lit room. The additional lighting was needed since we employed the NVG with the lenscap on, thus viewing through a pinhole, which considerably reduced the amount of incoming light. The subjects wore a dark hood with small slits that enabled viewing with the NVG or viewfinder of the thermal camera (Figure 4a). This prevented the subjects from using their unaided eye or the peripheral vision of their aided eye to perform their task (to detect visual targets and avoid obstacles). By using the black hood we ensured the subjects could only use the viewing device(s) at hand to perform their task. 3. Results All 6 subjects could easily fuse the NVG and thermal images. None of them reported any problems due to binocular rivalry. The subjects typically took a few minutes to traverse the entire course. Most subjects reported that they felt considerably more confident using the NVG to move around. This was partly a result of the larger FOV of the NVG (40 deg), and partly because the image of the NVG provided more detail and a better depth perception. The thermal camera provided a much smaller FOV (17x12 deg) with less detail (less resolution and contrast). When using only the thermal camera subjects have to make large scanning head movements to get an overview of the scene. Table 1 Mean (over all 6 subjects) number of detected visual and camouflaged thermal targets for each of the 3 viewing modes (intensified or NVG, thermal LWIR, and combined NVG and thermal). Target type Viewing mode Visual Thermal NVG 10 0 LWIR ± 0.3 NVG + LWIR ± 0.4 (d) Figure 4 (a) Subject using the combination of NVG and thermal camera. NVG image of part of the scene showing two visual targets. LWIR image of the scene showing a camouflaged thermal target. Superposition of on representing the dichoptically fused image seen by the subject. Table 1 shows the number of detected visual and (camouflaged) thermal targets, for each of the viewing modalities, averaged over all 6 observers. The results are unambiguous and just as expected. With NVG viewing, all visual targets are detected, but none of the thermal targets are seen. With thermal viewing, almost all camouflaged thermal targets are detected (a small fraction is missed, probably due to the small FOV of the thermal camera), but none of the visual targets is detected. And finally, with combined NVG and thermal viewing, all visual and almost all thermal targets are detected.

5 4. Discussion The NVG displayed the trees with higher contrast than the thermal camera. Also, the NVG has higher resolution and displays a larger FOV. As a result, we noted that subjects were able to move around through the forest much easier when using the NVG than with the thermal camera alone. When using only the thermal camera subjects had difficulty to orient themselves and sometimes bumped into trees. Subjects also reported that they found the NVG much more comfortable to move around with. When using the thermal camera some subjects missed 1 or 2 of the hot targets (10 in total). This is most likely due to the relatively small FOV of the thermal camera, which requires the subjects to make large scanning head movements in order to cover the entire search area. Not all subjects performed a systematic scanning behaviour, which is why they may have missed some targets. We did not encounter any problems due to binocular rivalry. Some subjects were better (required less time) at aligning the sensors and fusing the images than others. This process can of course be simplified by rigidly connecting the two sensors, such that the optical axes are aligned, and that only the distance between them needs to be adjusted to the individual inter-eye distance. A great advantage of the simple binocular fusion scheme tested in this study is the fact that the two sensors can be used individually (when one is more suitable than the other) and in combination (if required). Also, the solution is cheap, since it requires only minimal adjustments (e.g. a common head mount) to existing devices. An additional advantage is the fact that the eye that is equipped with the NVG can still use aiming devices like a holographic dot sight (which is not posible with a thermal viewing device since glass blocks longwave thermal radiation). It is probably important to have the option to adjust (equalize) the relative contrast and luminance of both images, to prevent dominance of one image modality over the other. In a follow-up experiment we plan to investigate human visual search, detection and navigation with dichoptic presentation of NVG and thermal imagery in more realistic military scenarios, like nighttime operations in rural (forest) and urban environments. Figure 5 and Figure 6 show an example of a rural scene, with a man standing between some trees. When standing in front of the trees (Figure 5) the man is clearly visible in the NVG image. However, only a few steps back he disappears in the shadows of the trees (Figure 6). In all cases he is clearly visible in the thermal image, since shadows have no effect on the image of a thermal camera (the thermal image depicts the temperature contrast between the man and his local background, which is the same in both cases). This example shows that one can avoid detection with image intensifiers simply by standing in the shadows (e.g. down a hallway or beneath a tree). However, detection by a thermal camera cannot be avoided that easily. Hence, the combination of a thermal and an NVG image allows the detection (thermal image) and the unabigous localisation (NVG image) of heat emitting targets. When moving around, the NVG image can used used for orientation and navigation (situational awareness), whereas the thermal image can be used for target detection. 5. Conclusions We find that observers have no problems with the dichoptic fusion of NVG and LWIR imagery. Monocular NVGs and thermal goggles can easily be joined together to provide observers with enhanced situational awareness in nighttime operations. References [1] Ackenhusen, J.G. (2003). Infrared/hyperspectral methods (paper II). In J. MacDonald et al. (Ed.), Alternatives for Landmine Detection (pp ). Santa Monica, CA: Rand Corporation. [2] Baertlein, B.A. (2003). Infrared/hyperspectral methods (paper I). In J. MacDonald et al. (Ed.), Alternatives for Landmine Detection (pp ). Santa Monica, CA: Rand Corporation. [3] Estrera, J.P., Ostromek, T.E., Isbell, W. & Bacarella, A.V. (2003). Modern night vision goggles for advanced infantry applications. In C.E. Rash & C.E. Reese (Ed.), Helmet- and Head- Mounted Displays VIII: Technologies and Applications (pp ). Bellingham, WA., USA: The International Society for Optical Engineering. [4] Goldberg, A.C., Fischer, T., Derzko, Z., Uppal, P.N. & Winn, M. (2002). Development of a dualband LWIR/LWIR QWIP focal plane array for detection of buried land mines. In E.L. Dereniak & R.E. Sampson (Ed.), Infrared Detectors and Focal Plane Arrays VII (pp ). Bellingham, WA: The International Society for Optical Engineering. [5] Hewish, M. (2002). Image is everything. Jane's International Defense Review, 35, [6] Kooi, F.L. (1993). Binocular configurations of a night-flight head-mounted display. Displays, 14(1), [7] Kooi, F.L. & Toet, A. (2004). Visual comfort of binocular and 3D displays. Displays, 25(2-3), [8] Nikolov, S.G., Bull, D., Canagarajah, C.N., Jones, M. & Gilchrist, I.D. (2002). Multi-modality gazecontingent displays for image fusion.proceedings of the 5 th International Conference on Information Fusion (pp ). Sunnyvale, CA: International Society of Information Fusion. [9] Wagenaar, W.A. (1969). Note on the construction of digram-balanced Latin squares. Psychological Bulletin, 72(6),

6 (a) (a) Figure 5 (a) Intensified (NVG) image, thermal image and fused image of a scene with a man standing in front of some trees. The man is clearly visible in the intensified image because he is illuminated by light from the sky (above). He is also visible in the thermal image because of his temperature contrast with the environment. Figure 6 (a) Intensified (NVG) image and thermal image of a scene with a man standing in behind some trees. The man is invisible in the intensified image because he is not illuminated by light from the sky (above). He is clearly visible in the thermal image because of his temperature contrast with the environment. The fused image shows both the man and the background in full detail.