EFFECTS OF HELMET-MOUNTED DISPLAY IMAGE LUMINANCE IN LOW-LIGHT AUGMENTED REALITY APPLICATIONS Eleanor O Keefe 2, Logan Williams 1, James Gaska 1, Marc Winterbottom 1, Elizabeth Shoda 2, Eric Palmer 2, Alex Van Atta 2, Steven Hadley 1 1 Operational Based Vision Assessment (OBVA) Laboratory, Aeromedical Research Department, U.S. Air Force School of Aerospace Medicine, 711th Human Performance Wing, Wright-Patterson AFB, OH 2 KBRWyle, OBVA Laboratory, U.S. Air Force School of Aerospace Medicine, Wright-Patterson AFB, OH ABSTRACT Previous work with low-light augmented reality helmetmounted display (HMD) applications examined the degree to which dark state liquid crystal display (LCD) luminance bleed-through can obscure real-world objects viewed through the display. In this follow-on work, we investigate the degree to which higher levels of luminance ( 115 cd/m 2 ) within monochrome HMD images can obscure real-world objects displayed in a simulated low-light formation flight scenario. These HMD images are intended to simulate augmented-reality camera images (night vision goggles, etc.) that are present in several HMDs used in military aviation and flight simulation. Observer performance was evaluated at several different luminance levels for tasks that require locating an aircraft with active navigation lights under starlight illumination. Adaptation time between relatively high and low HMD luminance conditions was also characterized. In this paper we summarize the previous work on LCD bleed-through and extend the psychometric threshold measurements to higher luminance levels due to augmented-reality camera images. Human performance was estimated for each luminance condition using the Psi psychometric threshold estimation algorithm. These methods can be used to accurately calibrate training simulations in which highly realistic representations of lowlight see-through HMD operations are a critical requirement for effective training. INTRODUCTION Helmet-mounted displays (HMDs) are becoming increasingly popular for use in both military and industrial applications. In general, while HMDs provide considerable benefits, there are still issues present with these devices that can degrade human performance. A major problem with liquid crystal display (LCD) HMDs in particular is backlight bleed-through. Negligible during most daylight use, bleed-through can degrade visual perception of images under low-luminance (mesopic) conditions. Although progress is being made (Chen et al., 2016), in many of these HMDs it is not possible to reduce the backlight bleedthrough. Thus, it becomes necessary to train the operator under realistic lighting conditions such that the performance impact becomes well understood. Previously, it has been shown that backlight bleed-through has a significant negative impact on both static and dynamic operational task performance, specifically related to scene contrast (Williams et al., 2017); however, that work was limited in the range of backlight luminance levels tested (0 to 1.7 cd/m 2 ). In these current studies, image luminance levels were extended up to 115 cd/m 2. Human performance was measured in an aircraft detection task while using a seethrough HMD with simulated augmented reality imagery. Two different imagery conditions were measured: continuous and transient image illumination. The purpose of continuous image display was to characterize how its addition raised see-through target luminance thresholds. The purpose of transient imagery was to characterize the time course of adaptation between augmented and nonaugmented display conditions under mesopic conditions. BACKGROUND There are currently many LCD HMDs in service due to their extensive development and use. More recently, organic light-emitting diode (OLED) displays were introduced. While they do not suffer from the same backlight bleedthrough as LCD displays, OLED displays have other problems such as brightness shift due to temperature changes (Hogan & Bacarella, 2011). In addition, current LCD HMDs are still useful in many situations. Therefore, research on these systems in relation to operational tasks is highly important. Users should be aware of any performance decrements of LCD displays, including those that stem from display brightness under augmented reality conditions. OLED systems use emissive pixels and so do not have any backlight bleed-through. LCDs, on the other hand, operate by acting as a light valve to selectively block or transmit light emitted from a backlight, typically an LED or
fluorescent light source (Kawamoto, 2012). Due to the imperfect function of the LCD light valve in blocking light, the pixel is not perfectly opaque. This means that a small portion of the backlight bleeds through the LCD panel and is visible to the user, especially in the black areas of an image. This imperfect dark state, or black level, is a wellknown shortcoming of LCD displays and decreases overall image contrast (Goodman, 2012). While the backlight bleed-through is always present, it is typically only noticeable in very dim ambient conditions. However, in mesopic augmented-reality HMD applications, this high augmented reality image luminance can significantly obscure real-world objects viewed through the display, as shown in Figure 1, thus greatly impacting performance. For very low contrast features the addition of even a small amount of image luminance may render these features either partially or completely unobservable. HMD In this work, an SA-62/S HMD (Figure 2) was used to produce several image luminance levels to explore a wide range of viewing conditions. The SA-62/S is a semitransparent HMD with a 53ºx33º field of view and two 1920x1200 full color OLED displays intended for biocular augmented- or mixed-reality applications. The OLED pixels can be driven to true zero luminance, allowing a range of non-zero bleed-through levels to be emulated, as well as any image luminance levels below ~115 cd/m 2. Note that this work was conducted using the green channel only in an effort to investigate correlations with achromatic contrast sensitivity (i.e. luminance only, without color perception effects), and the green channel provided the greatest luminance range. It should be noted that, although the augmented reality HMD image contained rich scene content, the localized area of the scene, in which the stimulus discrimination is made, was used for luminance calibration purposes. All references to image brightness pertain to the local scene brightness near the stimulus location. The HMD was head tracked using a Polhemus Liberty magnetic tracking system to maintain HMD image alignment with the out-the-window stimulus. Figure 2: SA-62/S HMD. Figure 1: Simulated images of aircraft in low ambient lighting conditions without (top) and with (bottom) augmented reality HMD display. The SA-62/S permits programmatic control of the HMD brightness and video level via USB communication. Each eyepiece of the SA-62/S was calibrated using a Minolta CS200 chromameter centered in the HMD exit pupil. Specific video signal levels were generated to hold the simulated image intensity at precise, equal levels in each eyepiece. For more specifics on the calibration procedures, please refer to Williams et al., 2017.
Display A Christie Matrix StIM digital projector was used to project the simulated operational task images onto a 40-degreewide display screen. The design eye point was located 3.5 m from the StIM projection. At this distance a single pixel subtends 1.11 arcminutes. The simulated operational task under study pertained primarily to flight under starlight illumination, in which the aircraft navigation lights require accurate representation. The illumination at the eye point at various simulated distances was modeled as a point source governed by the inverse square law, based upon the Federal Aviation Administration minimum luminous intensity of navigation lights viewed from directly behind an aircraft (20 candelas). For more specifics on the calibration procedures, please refer to Williams et al., 2017. SIMULATED OPERATIONAL TASKS Each of the simulated operational tasks was implemented in X-Plane. The light point control was implemented using a custom pixel shader written in Visual Studio (C++) using the X-Plane software development kit. The control host and psychometric procedures were written in MATLAB using UDP multicast communication with X-Plane. Static Detection Task 1 The first experiment consisted of navigation lights of a KC- 135 fixation target in the center of the out-the-window projection. The KC-135 fixation target remained at a fixed brightness level (115 cd/m 2 ) that could be seen through all levels of display brightness. A stimulus aircraft appeared either to the left or right of the KC-135, as shown in Figure 3. The participant indicated the position of the stimulus aircraft as a two-alternative forced choice response. The distance to, and thus brightness of, the stimulus aircraft varied in an adaptive manner controlled by the Psi algorithm (30 trials), such that correct responses generally resulted in more difficult/dimmer stimuli, while incorrect responses resulted in brighter stimuli (Kingdom & Prins, 2010). The result is an individual luminance threshold (cd/m 2 ) at which the subject was able to make a correct response with an 81% probability of success. This experiment was performed on 20 participants with good contrast sensitivity for eight levels of image brightness spanning 0 to 115 cd/m 2. Each participant achieved a perception threshold within the out-the-window stimulus brightness range of a single pixel. Therefore, stimulus size effects remained constant across all participants. Figure 3: Appearance of the stimulus and fixation target navigation lights. Static Detection Task 2 In this experiment, the same static detection task was used, but this time to assess adaptation and recovery time as the augmented reality HMD image was switched completely on or off. This simulated the adaptation that an HMD user will encounter when initially donning or removing the HMD, or activating/ deactivating the display, in a dim environment. In this scenario, stimuli were repeated at 3-second intervals over a period of 36 seconds in which the HMD image is activated at the maximum level (115 cd/m 2 ) for 18 seconds, then deactivated (0 cd/m 2 ) for 18 seconds. Stimuli were specifically timed to occur at the moments of image activation and deactivation. This experiment was performed on nine participants who had completed the previous task. RESULTS Static Detection Task 1 The results from the first experiment illustrate a characteristic luminance threshold response in which the brighter the HMD display image, the brighter the target must be to be reliably visible through that image (Figure 4). For the first level of image luminance, target luminance threshold remains about the same as ambient light. That is, at very low image levels, almost no difference in performance is observed. Target thresholds from 0.07 to 1.7 cd/m 2 (Williams et al., 2017) have a slope of 0.4 on the log scale, which is close to the value one would expect (0.5) if performance was limited by Poisson noise of the bleedthrough (quantum fluctuation model). Target thresholds from 1.7 to 115 cd/m 2 have a slope of 1, which is expected by Weber s law model due to the more intense background luminance levels (Ganz, 1975); that is, the change in threshold is proportional to the intensity of the HMD image luminance.
Figure 4: Comparison of previous and current data, which extends the range of tested backlight and image luminance levels. Static Detection Task 2 The results from the second static detection task characterize responses to a luminance target during a continuous cycle of HMD image on (115 cd/m 2 ) and off. When the image is turned on, luminance threshold skyrockets and then quickly drops and plateaus as the photoreceptors adjust. When the image is turned off, luminance threshold drops through two of the measured intervals (6 seconds) and plateaus. As Figure 5 illustrates, these adaptation and recovery times are both on the order of 3 to 6 seconds, which is in agreement with previously published literature (Howard et al., 2001). DISCUSSION These experiments illustrate that HMDimage luminance can have a significant and quantifiable impact on operational task performance. The performance decrements are easy to predict as they closely follow standard vision models. While very low levels of image brightness barely impede viewing ability, additional image brightness quickly raises visual thresholds. Low image brightness levels mean that visual thresholds follow the quantum fluctuation model and higher levels cause thresholds to follow Weber s law. Adaptation time when switching from no image to very bright image and vice versa needs to be taken into consideration. Considerable research and development has been devoted to properly simulating night vision goggle characteristics (e.g., halos, auto-gain, visual illusions, etc.) to support training. Similar research and development may be required to ensure that aircrew are familiar with the unique capabilities, and idiosyncrasies, of HMDs as their use becomes more widespread. However, it must be emphasized that the psychometric experiments performed here simulated a specific task (point source localization) using subjects with normal contrast sensitivity. Although the concepts here are generalizable to other scenarios, particular data are not. For example, the ability to correctly identify a high-contrast point source is not the same as identifying a low-contrast extended target. In each case, the image levels that result in unacceptable performance would be necessarily different and may be further complicated by individual differences in contrast sensitivity. CONCLUSION The importance of display contrast in augmented reality applications has been demonstrated via psychometric experimentation. Specific levels of HMD image brightness have been shown to correlate with performance decrements, as predicted within the literature. Luminance threshold changes when switching from a high image level to no image and vice versa have been characterized. Figure 5: Luminance thresholds for average image brightness adaptation and recovery.
ACKNOWLEDGMENTS The views expressed are those of the authors and do not necessarily reflect the official policy or position of the Air Force, the Department of Defense, or the U.S. Government. REFERENCES Chen, S., Tsai, H., Lin, T., & Lee, H. (2016). Block-based content adaptive backlight controller VLSI design for local dimming LCDs. Proc. 23rd International Workshop on Active-Matrix Flatpanel Displays and Devices (AM-FPD), Kyoto, 63-66. Ganz, L. (1975). Vision. In B. Scharf and G. S. Reynolds (Eds.), Experimental Sensory Technology (pp. 217-258). Philippines: Scott, Foresman and Company. Goodman, T. (2012). Overview of the Photometric Characterisation of Visual Displays. In J. Chen, W. Cranton, and M. Fihn (Eds.), Handbook of Visual Display Technology (pp. 243-255). Berlin: Springer. Hogan, T., & Bacarella, T. (2011). Microdisplay contributions to system level performance. Proc. SPIE 8042, Display Technologies and Applications for Defense, Security, and Avionics V; and Enhanced and Synthetic Vision. Howard, C. M., Riegler, J. T., & Martin, J. J. (2001). Light adaptation: Night vision goggle effect on cockpit instrument reading time. Aviation, Space, and Environmental Medicine, 72(6), 529-533. Kawamoto, H. (2012). The history of liquid-crystal display and its industry. Proc. Third IEEE History of Electrotechnology Conference (HISTELCON), 1-6. Kingdom, F. A., & Prins, N. (2010). Psychophysics: A practical introduction. London: Academic Press/Elsevier. Williams, L., Gaska, J., Winterbottom, M., Bullock C., Bauer, M., Van Atta, A., & Hadley, S. (2017) Effects of HMD backlight bleed-through in low-light augmented reality applications. Proc. IMAGE 2017 Conference.