1. INTRODUCTION ABSTRACT

Transcription

1 Long-Range Night/Day Human Identification using Active-SWIR Imaging Brian E. Lemoff, Robert B. Martin, Mikhail Sluch, Kristopher M. Kafka, William McCormick and Robert Ice WVHTC Foundation, 1000 Technology Drive, Suite 1000, Fairmont, WV, USA ABSTRACT Positive identification of personnel from a safe distance is a long-standing need for security and defense applications. Advances in computer face recognition have made this a reliable means of identification when facial imagery of sufficient resolution is available to be matched against a database of mug shots. Long-range identification at night requires that the face be actively illuminated; however, for visible and NIR illumination, the intensity required to produce high-resolution long-range imagery typically creates an eye-safety hazard. SWIR illumination makes active- SWIR imaging a promising approach to long-range night-time identification. We will describe an active-swir imaging system that is being developed to covertly detect, track, zoom in on, and positively identify a human target, night or day, at hundreds of meters range. The SWIR illuminator pans, tilts, and zooms with the imager to always just fill the imager field of view. The illuminator meets Class 1 eye-safety limits (safe even with magnifying optics) at the intended target, and meets Class 1M eye-safety limits (safe to the naked eye) at point-blank range. Close-up night-time facial imagery will be presented along with experimental face recognition performance results for matching SWIR imagery to a database of visible mug shots at distance. Keywords: Face Recognition, SWIR, Night Vision, Surveillance, Biometrics, Active Imaging 1. INTRODUCTION The capability to detect and identify individuals from a great distance, night or day, without their knowledge, could have many applications for defense, law enforcement, and private security. Installation security guards could detect known bad actors as they approach or survey a facility from a distance. Law enforcement or intelligence agents would be able to covertly monitor unlit locations 24 hours a day from a safe distance, identifying individuals and recording their actions. SWAT teams or commandos could confirm the presence of specific individuals prior to launching a targeted attack or rescue mission. Under daylight or otherwise lit conditions, it is possible today for an operator using highpower optics to manually identify people at a distance if they are familiar to him or if he can refer to a short watch list of mug shots; however, automated identification at long range is not yet available. At night, or under otherwise dark conditions, there is currently no technology that produces imagery that allows for long-range identification, either manual or automated. To address this capability gap, the West Virginia High Technology Consortium Foundation (WVHTCF), under a research contract from the Office of Naval Research (ONR), is developing the Tactical Imager for Night/Day Extended Range Surveillance (TINDERS), an active short-wave infrared (SWIR) imaging system that illuminates targets with an invisible and eye-safe SWIR laser beam. Goals for the project, from bright sunlight to total darkness, include: human detection and tracking at ranges up to 3 km; generating recognizable facial imagery at ranges up to 800 m; and identification through computer face recognition at ranges up to 400 m. When complete, the goal is for TINDERS to be a portable, easy to set-up system that can automatically detect, track, zoom in on, and image a moving person, and identify them through computer face recognition. 1.1 Background There are a number of excellent long-range imaging technologies, commercially available today for human surveillance applications, each with its own strengths and weaknesses; however, none appears to be a suitable solution for compact, long-range, covert, night/day human identification. Whether the goal is computer face recognition or simply recognition by a human operator, visible-spectrum imagery will always produce the best result if conditions allow for a quality image to be obtained. Unfortunately, under nighttime or otherwise dark conditions, there is insufficient ambient illumination of the target to produce a visible image. A spotlight could be used, but this would not be covert, and the

2 intensity required to produce a high-quality close-up facial image at long range would be damaging to the eye. Thermal or long-wave infrared (LWIR) imagery is an excellent tool for nighttime detection of personnel; however, it does not produce recognizable facial imagery. In addition, compact thermal imagers are better suited to wide-angle imagery, as narrow-angle thermal imagery (e.g. 2-mm per pixel at 150-m range) requires very large and heavy lenses. Passive SWIR imagery is another excellent technology for day/night wide-area surveillance 1. Even with no moon and slightly overcast conditions, there is enough ambient night-glow to produce wide-angle imagery such as that shown in Figure 1(a). Unfortunately, nighttime signal levels, even under a full moon are too low for narrow-angle imagery, such as that needed to recognize a person at 100-m range. Because the amount of light hitting each pixel in the target image is proportional to the total amount of light coming from the target divided by the number of pixels spanned by the target, the signal level in a passive image increases as the square of the field of view. For example, narrowing the field of view of an image by a factor of 10 reduces the signal level by a factor of 100. Active near-infrared (NIR) surveillance systems are available commercially from companies such as Vumii 2. These systems combine a long-range camera (conventional silicon CCD) with a NIR illuminator (typically around 800-nm wavelength) to produce high-quality, long-range imagery night and day. By illuminating the camera field of view with light that is invisible to the human eye, but close-enough to the visible spectrum to produce familiar-looking imagery, high-quality long-range imagery like that shown in Figure 2(b) is possible. While imagery such as this should be sufficient for human identification, the illumination power required to produce quality facial images at ranges beyond 100-m creates a possible eye-safety hazard to the target, whose face is being deliberately illuminated, and a severe eyesafety hazard (immediate and permanent damage to the retina) in close proximity to the illuminator. In addition, while virtually invisible to the human eye (at high intensity, 800-nm appears as a dull red glow), the NIR illumination is clearly visible with any night-vision goggle and most silicon-based cameras. (a) (b) Figure 1. (a) Passive SWIR image at 600-m range at night with no moon and slightly overcast conditions 1. Ambient night-glow provides sufficient illumination for wide-angle imagery, but narrow-angle imagery is not possible with this technology. (b) Active-NIR facial image at 130-m range at night, using Vumii Discoverii D With compact optics, useful image signal levels can only be achieved at such long range by creating a severe eye-safety hazard in close proximity to the illuminator. NIR Illumination is also easily seen with night vision goggles and most silicon-based cameras. 1.2 Eye-safe active-swir illumination Active-SWIR imagery at wavelengths > 1400 nm, particularly in the wavelength band most commonly used by the telecommunications industry for fiber-optic communication, overcomes the two primary limitations of Active-NIR imagery in that it is completely invisible to night-vision goggles and humans, and the eye-safe power levels are much higher. Table 1 shows a comparison of the visibility and maximum eye-safe power levels of 4 potential illumination wavelengths. As defined in the ANSI Z136 and IEC laser eye safety standards 3, Class 1M means that there is no hazard to the naked eye, but there is a potential hazard when magnifying optics (e.g. binoculars or scope) are used, while Class 1 means that there is no hazard, even when magnifying optics are used (up to 7X magnification). For the TINDERS application, the absolute minimum illumination spot diameter intentionally shined on a person s face is 1 meter, so Class 1 safety at that diameter is the goal. To keep the optics compact, the output aperture of the TINDERS

3 illuminator is limited to 5 inches. Thus, to be safe to inadvertent exposure near the illuminator, we require Class 1M safety at this beam diameter. Notice that the safe power level at >1400 nm is ~ 65 times higher than at 800 nm. Table 1. Comparison of potential illumination wavelengths. Wavelength Human visibility NVG visibility Class 1 m diameter spot Class 5-inch diameter beam 800 nm Dull red glow Visible < W < W 980 nm Invisible Visible < W < W 1064 nm Invisible Visible < W < W >1400 nm Invisible Invisible < 16.7 W < W Unlike thermal infrared, where facial features are determined by skin temperature and can vary widely depending upon the thermal conditions and metabolic state of the individual, active-swir facial imagery produces repeatable imagery, showing only features that scatter the incident light. Figure 2 shows the same individual illuminated with visible white light and illuminated with an eye-safe SWIR laser. While the skin and hair pigmentation appear quite different in the two images, the geometry of the facial features are the same. Thus, it should be possible to match a SWIR facial image against a database of visible-spectrum facial images using an appropriate computer face recognition algorithm. In addition, once a human operator becomes accustomed to the darker skin and lighter hair appearing in SWIR facial images, manual recognition of individuals based on SWIR facial images should be possible. Figure 2. (left) Facial image of an individual illuminated with visible-spectrum white light. (right) Facial image of the same individual illuminated with an eye-safe SWIR laser operating in a wavelength band commonly used for long-distance telecommunications. Note that hair appears white and skin appears dark in the SWIR image, but the same features, with the same shapes, are present in both images. Early in the TINDERS project, visible and SWIR facial imagery similar to that shown in Figure 2 was collected from 56 subjects. An experiment was performed using a commercial face recognition software package, ABIS System FaceExaminer 4, from Identix (now MorphoTrust USA), in which a single SWIR facial image from each subject was matched against a database containing 1156 visible-spectrum facial images, including 1 visible image from each of the 56 subjects and 1100 visible images from the FERET facial database 5. The commercial software, which had been designed only to match visible images to other visible images, achieved a correct match for 40 out of 56 subjects, for a Rank 1 success rate of 71%. This indicates the overall feasibility of using active-swir imaging for long-range identification. 2. SYSTEM DESCRIPTION The TINDERS project began in spring 2009, with the design of a laboratory prototype whose purpose was to prove feasibility of the system concept. Following a successful field demonstration of this proof-of-concept prototype in summer 2010, a second-generation prototype was designed with a physical form factor and software architecture more suitable for eventual field use. This second-generation prototype was first used in a field experiment in Fall 2011, and the current research efforts continue to evolve this prototype.

4 2.1 Hardware A conceptual illustration of the TINDERS hardware is shown in Figure 3. The TINDERS system consists of three physical units, an optical head, that sits on a pan-tilt (PT) stage, an electronics box that provides power, light (through and optical fiber), and communications to the optical head, and a computer that runs the user interface, low-level camera control functions, system automation, and face recognition software. Figure 3. Conceptual illustration of the TINDERS hardware. The optical head includes both the SWIR illuminator optics and the imager. In the current version of the hardware, the optical head weighs roughly 30 pounds and sits in an environmentally-controlled enclosure atop a commercial pan-tilt stage. The imager and illuminator pan, tilt, and zoom together so that the illuminator beam is always just filling the imager field of view. This serves to maximize the image signal level and avoid wasted light. The illuminator light source, located in the electronics box, delivers a maximum power of 5W to the optical head through an optical fiber in the umbilical. This light source leverages commercial technology developed for the long-distance telecommunications industry. The zoom optics in the imager are optimized for monochromatic imaging of a narrow field of view, allowing a dramatic reduction in lens complexity and weight relative to traditional zoom optics that must compensate for chromatic aberration and provide distortion-free images over the entire zoom range. Figure 4. (left) Original TINDERS proof-of-concept prototype demonstrated at a field experiment in the summer of (right) Current TINDERS prototype as of December The current prototype has a physical form factor and software architecture that are more appropriate for eventual field use.

5 Figure 5. (left, center) TINDERS prototype at a December 2011 field experiment in which a long (> 50-ft) umbilical was used to connect the optical head to the electronics box located in a powered trailer some distance away. (right) TINDERS deployed atop a 35-ft mast during a December 2011 demonstration. For this implementation, the electronics box was located at the base of the mast, with the umbilical extending the length of the mast. Figure 4 shows the original TINDERS proof-of-concept prototype, as demonstrated in a field experiment in summer 2010 and the current TINDERS hardware as of December In the current hardware configuration, the TINDERS computer is connected to the electronics box through an Ethernet cable. Cables as long as 300-ft have been used. This connection could also be made through a switched network; however, performance may suffer if the network bandwidth is too low or latency too high. The umbilical that connects the electronics box to the optical head includes power, data communications, and optical cables. When the electronics box is located near the optical head, as in Figure 4, a short umbilical (~ 15 ft) is typically used; however, longer umbilicals (> 50 ft) have been used when the TINDERS optics was deployed atop a mast or on a tripod located in a field far from any power source. Figure 5 shows examples of TINDERS deployed in such situations. 2.2 Software The TINDERS software functions include low-level hardware control, automation, enterprise messaging, face recognition, and the graphical user interface. Low-level hardware control software moves the lenses in the imager and illuminator to achieve the correct zoom and focus, controls the PT stage, controls the image sensor and receives video, and controls other system components such as the light source, GPS, laser rangefinder, and temperature controllers. Automation software can currently detect people and faces in the live video, and when completed will automatically track moving targets and automatically queue detected faces for face recognition. Messaging software allows the TINDERS system to interoperate with other systems that may need access to TINDERS status, target position, target identity, or may need to cue TINDERS to point to a particular location. The GUI allows an operator to view live video while controlling and monitoring all of the TINDERS software functions. The TINDERS face recognition software leverages the commercial ABIS System FaceExaminer software 4 from MorphoTrust USA. As part of the TINDERS research program, researchers at MorphoTrust USA developed a preprocessing filter to apply to the SWIR facial images to improve the matching performance of the SWIR images to visible-spectrum images contained in the database. In its current form, the TINDERS software allows the operator to submit video frames to the face recognition software by clicking a button on the GUI. Face recognition results are then displayed in the TINDERS GUI. Work is currently underway to automate this process, so that faces detected in the live video will automatically be submitted to the face recognition software. MorphoTrust USA is also continuing to work with WVHTCF on improving the performance of the SWIR-to-visible matching algorithms used in TINDERS. 3. RESULTS Two datasets of TINDERS facial imagery were collected. The first dataset collected using the original proof-of-concept prototype, included facial imagery of 56 subjects at distances of 50 m and 106 m, indoors in total darkness. For each subject, frontal still images were collected with both neutral and talking expressions, and images were collected with the head turned left and right by 10º and 20º while talking. The second dataset, recently collected using the secondgeneration prototype, included facial video imagery of 104 subjects at distances of 100 m, 200 m, and 350 m, all

6 collected outdoors under dark nighttime conditions. Video was collected with the subjects stationary and facing the camera as well as with the subjects rotating 360º. Figure 6 shows example imagery from the second dataset for two stationary subjects at distances of 100 m, 200 m, and 350 m. As expected, the resolution and contrast degrade as the distance increases, but sufficient resolution remains at 350 m for possible recognition. The stated goal for the TINDERS project is to achieve computer face recognition at distances as high as 400 m, which is slightly farther than the longest distance included in this dataset. Figure 6. Example TINDERS facial imagery under dark nighttime conditions for two subjects at distances of (left) 100 m, (left center) 200 m, and (right center) 350 m, along with (right) visible-spectrum image of the same subjects. The first dataset was shared with two research groups at West Virginia University (WVU), who were working independently on SWIR-to-visible face recognition algorithms 6,7. Kalka, et. al. applied a pre-processing algorithm to the SWIR images before matching them to a visible-spectrum database using FaceIt G8 software from MorphoTrust USA. They achieved a Rank 1 success rate of 90% for the 50-m TINDERS images and 80% for the 106-m TINDERS images 6. Zuo, et. al. fused the results of the FaceIt G8 software with a face recognition algorithm developed by their group 7. With a 0.1% False Acceptance Rate, they achieved a Correct Acceptance Rate of 85% for the 50-m TINDERS images and 74% for the 106-m TINDERS images. The same dataset was also used by researchers at MorphoTrust USA in their development of the face recognition software that is integrated into the TINDERS system. To evaluate their pre-processing filter, they processed 9 SWIR images for each subject at each distance, including 3 frontal neutral images, 2 frontal talking images, and 4 images with a 10º pose angle. Each image was pre-processed and matched against a database containing visible-spectrum images of all 56 subjects. For each subject, the results of the 9 searches were fused by keeping the result with the highest matching score. Figure 7 shows the receiver operating characteristics (ROC) results at 50 m and 106 m with and without the preprocessing algorithm. With a 1% False Acceptance Rate, the pre-processed results achieved a Correct Acceptance Rate of roughly 70% at both 50 m and 106 m. Surprisingly, the images with 10º pose angle accounted for more than 25% of the highest scores in the successful matches, indicating the algorithm is fairly robust for pose angles within 10º of frontal.

7 Figure 7. Receiver operating characteristic generated by MorphoTrust USA using TINDERS images of 56 test subjects at 50-m and 106-m range in total darkness. Correct Acceptance Rate of roughly 70% was achieved with False Acceptance Rate of 1% at both distances. A proper statistical analysis of face recognition performance has not yet been completed for the second dataset, recently collected with the second-generation TINDERS prototype; however, the face recognition software is certainly performing better than random chance at both 200 m and 350 m range. Figure 8 shows screen shots of successful TINDERS face recognition at 200 m and 350 m, where the correct person is chosen out of a database containing visiblespectrum images of more than 1600 individuals. For these examples, TINDERS was playing back recorded video from the second dataset as if it were live. The operator clicked a button on the TINDERS GUI, which sent 11 video frames to the face recognition software for matching. In the case of the 200-m result, 8 of the 11 frames were detected as good faces and eye positions automatically marked. For the 350-m result, 9 good faces were detected and eyes automatically marked. The detected good facial images were then searched against the visible-spectrum facial database containing over 1600 individuals, and the results fused to produce an aggregate matching score for each of the 1600 candidates. The top 20 candidates are then displayed in rank order along the bottom of the screen. In both examples, the correct person was chosen as the top match. Aside from clicking the button on the TINDERS GUI to intiate the process, the operator did not need to interact with the system to produce the results. The entire process completed in < 20 seconds. Figure 8. Example screen shots showing TINDERS face recognition from video under dark nighttime conditions at distances of (left) 200 m and (right) 350 m. In both cases, the correct individual was chosen from a database containing visible-spectrum images of more than 1600 people.

8 4. DISCUSSION In many implementations of computer face recognition, a single, high-resolution visible-spectrum facial image is matched with a very high confidence level against a large database of high-resolution visible-spectrum facial images. In the case of the TINDERS system, where the detected facial images are SWIR, rather than visible, and where longdistance resolution is often much lower than optimal, it is unlikely that a single detected SWIR facial image will ever produce a high-confidence match to a large visible database. Nevertheless, TINDERS produces repeatable, recognizable images of people under both daytime and nighttime conditions, at distances well beyond 100 m that can be matched to a visible-spectrum database using computer face recognition software, in which the statistical performance of the software is much better than random chance. Given this, we hope to be able to produce high-confidence matching through the fusion of many video frames, acquired as a single person is tracked over time. Tracking software is being developed that will allow TINDERS to follow a moving person over time. With 30 video frames per second, the strategy is to automatically select the best facial images from the video and continually submit them for face recognition. As more and more SWIR facial images of the same person are collected and compared to the visible database, the scores and or ranks of the database images can be fused to produce an identification result that continues to increase in confidence level as the process continues. Just as a noisy signal can be clarified through time averaging, a face recognition capability that has low confidence for a single captured image can be made high confidence through capturing may images of the same person at slightly different times, angles, expressions, etc. Automation software to achieve the tracking, real time face detection, quality analysis, and fusion of the face recognition results is currently being developed. In addition, Morpho Trust USA will use the most recent video dataset of 104 subjects to further improve the performance of the TINDERS face recognition algorithms. In summary, WVHTCF has developed a portable active-swir imaging system that is capable of generating recognizable facial imagery at distances of at least 350 meters under conditions ranging from bright sunlight to total darkness. Three independent research groups have used TINDERS imagery collected in total darkness from a distance of 106 m to achieve computer face recognition success rates of at least 70% in matching against a database of visiblespectrum images. The TINDERS system has integrated face recognition software developed by MorphoTrust USA, that can be used for live identification of subjects, day or night, with some success achieved at distances up to 350 meters. 5. ACKNOWLEDGMENTS This research was performed under contract N C-0064 from the Office of Naval Research, with funding from the Deployable Force Protection Science and Technology Program, and with additional support from a subcontract from West Virginia University Research Corporation through Award No DD-BX-0161 awarded by the National Institutes of Justice. The authors would like to acknowledge important technical contributions from Jason Stanley and Andrew Dolby, the MorphoTrust USA team, and the cooperation of the WVU Center for Identification Technology Research in the collection of the most recent facial dataset. REFERENCES [1] Acton, D., Counting Photons: Advances in Passive Short Wave Infrared Imaging, Technology Today, issue 2 (2010), [2] Vumii Discoverii product brochure. ftp://ftp.vumii.com/documentation/discoverii/vumii_discoverii_brochure.pdf [3] Laser Safety, Wikipedia page. [4] ABIS System FaceExaminer web page. [5] FERET Database web page. [6] Kalka, N.D., Bourlai, T., Cukic, B., Hornak, L., "Cross-spectral face recognition in heterogeneous environments: A case study on matching visible to short-wave infrared imagery," Biometrics (IJCB), 2011 International Joint Conference on, pp.1-8, Oct [7] Zuo,J., Nicolo, F., Schmid, N.A., Boothapati, S., "Encoding, matching and score normalization for cross spectral face recognition: Matching SWIR versus visible data," Biometrics: Theory, Applications and Systems (BTAS), 2012 IEEE Fifth International Conference on, pp , Sept. 2012