Head Mounted Displays

Transcription

1 Simon Chuptys KU Leuven Leuven, Belgium Head Mounted Displays Jeroen De Coninck KU Leuven Leuven, Belgium ABSTRACT Head Mounted Displays (HMD) are being heavily researched again, both as a new display technology and interaction medium. While they have been losing popularity after their initial hype, recent technological developments have stimulated the interest for HMD research again. In this paper we give an overview of the research that has been done related to head mounted displays. We will describe the applications of HMDs along with the major difficulties that are associated with them. TODO: should be about 150 words Author Keywords head mounted display; augmented reality; virtual reality; screen; ACM Classification Keywords H.5.m. Information Interfaces and Presentation (e.g. HCI): Miscellaneous See: for more information and the full list of ACM classifiers and descriptors. Mandatory section to be included in your final version. On the submission page only the classifiers letter-number combination will need to be entered. INTRODUCTION Head mounted displays have quite a long history of research. Initial designs, like TODO: list some early designs were built as heavy helmets strapped to the user s head, making them uncomfortable to wear for prolonged periods of time while also severely restricting head movement (let alone full body movement). Recent technological advancements have made the HMD lighter and smaller, in some cases barely different from normal glasses. Due to these advancements, HMD technology has become an exciting area of research once again. Therefore, we believe it is very important to provide an overview of current state of research, the problems typically associated with HMDs, the products that are currently available or in development and the application areas that may benefit from this technology. Paste the appropriate copyright statement here. ACM now supports three different copyright statements: ACM copyright: ACM holds the copyright on the work. This is the historical approach. License: The author(s) retain copyright, but ACM receives an exclusive publication license. Open Access: The author(s) wish to pay for the work to be open access. The additional fee must be paid to ACM. This text field is large enough to hold the appropriate release statement assuming it is single spaced. PRESENTED PAPERS TODO: An overview of the papers that are presented in this paper + short description + motivation. HEAD MOUNTED DISPLAYS Basically, head mounted displays include every kind of technology that mounts displays to the user s head. We can observe two major areas that make use of HMDs: Augmented Reality (AR) and (VR). While they both use similar devices, they each have very specific application areas and problems associated. Therefore, we think it feasible to address them separately. The idea behind (AR) is to enhance the normal vision by means of overlaying images using displays close to the eyes. This way, the wearer is assisted in everyday tasks. Possible applications include displaying relevant information about current activities, highlighting important environmental elements, or feeding the wearer information in a way that he can still carry on with normal activities. There are different possible configurations for using HMDs in AR: both a monocular (only one eye has a display) or a binocular setup can be used. Furthermore, the displays may be either transparent or opaque. Overlaying images onto the normal vision can be done in two ways: either using video see-through (VST) displays or optical see-through (OST) displays. VST-HMDs capture a video stream of the environment using cameras, which then is mixed with artificial content. The resulting images are typically projected on an opaque screen, so the user will only see images that are captured by the cameras mounted on the HMD. The advantage of this method is that conventional computer vision techniques can be used to synchronize the real-word and artificial imagery. Disadvantages include a limitation on the field of view (fov), as well as a lessened ability to focus on specific objects that are at different distances from the user. Also, latency issues may arise. OST-HMDs on the other hand make use of see-through displays, allowing the user to see real-world imagery in the conventional way, without fov, focussing or latency issues (at least, not for the real-world images). These devices typically make use of a half-silvered mirror to project the digital images onto while not obstructing the normal view (see figure 1). The disadvantages of this method are mostly due to the ever changing environmental conditions: changes in lighting 1

2 call for careful real-time adjustments in brightness and contrast. Furthermore, these AR devices typically suffer from visual interference, due to the overlaying of multiple image sources. Calibrating virtual images with what the user sees is also non-trivial. However, the OST method has been shown to offer the most natural viewing conditions [1]. Therefore, this is the most promising technique to consider regarding AR. Figure 1. Schematic overview of a typical OST device The popularity of AR has dramatically increased these past few years, mainly because of the development of the Google Glasses [2], which will allow accessibility to this technology for the general public. (VR) is where the HMD blocks out our normal vision and replaces it with a rendering of a virtual world. This is generally associated with entertainment purposes such as immersion in video games, but there are a lot of situations that can benefit from VR. Other example uses of VR are architectural visualisation, teleconferencing and visualisation of molecular structures. Replacing the entire visual field with an image on a display comes with a whole set of problems, mostly related to the differences between viewing the continuous light coming from the real world and viewing the discretized light emitted by an HMD. Like AR, VR is currently gaining popularity due to a HMD for the general public being in development (the Rift by Oculus VR). PROBLEM DESCRIPTION TODO: An overview of all problems related to HMDs, with sections for AR and VR where specific problems will be discussed. Uncontrollable conditions: Due to the ever changing lighting conditions, AR devices continuously need to make adjustments to the brightness and contrast of overlay images [3]. VST-HMDs have the specific problem of a limited dynamic range due to the use of cameras to capture real world imagery: over-bright or very dark real world footage may cause the video stream to become unrecognisable, literally blinding the user. Modern displays still can t display the dynamic range of real world luminance. OST-HDMs suffer from another problem, which is amplified by lack of proper occlusion between real and virtual images. This means that it is impossible to visualize black overlays: these regions appear transparent. This decreases readability of the overlaid footage in outdoor environments. Latency, resolution, display curvature: As mentioned before, VST-HMDs severely restrict the user s field of view. TODO: describe work of Kollenberg, Visual search in the (un)real world: how head-mounted displays affect eye movements, head movements and target detection. Furthermore, the delay due to capturing real world images and displaying them on the HMD introduces a slight latency during eye and head movement. This results in slower reaction times and reduced hand-eye coordination abilities of the wearer. Registration accuracy: When overlay footage needs to interact with the real world imagery, we need good calibration techniques in order to obtain a high registration accuracy. VST-HMDs typically suffer less from these issues, since a lot of the techniques of computer vision can be used to mix real world and virtual footage. OST-HMDs however typically can t use these techniques because of the lack of recordings from the real-world footage as seen by the user: The composition of real and virtual images happens on the user s retina instead of on the display. However, in [1] a method is explained for calibrating these OST devices. An extra video capture stream is used for this process, allowing to still use common computer vision techniques while calibration errors due to the facial structure of the wearer are minimized. Binocular rivalry: At each point in time, there is usually one eye which is more dominant than the other, meaning we mainly see images seen by that eye, while parts of the imagery captured by the other eye is discarded. Which eye is dominant and the duration of this dominance varies and is influenced by what is currently observed, making it unpredictable. This causes problems which are mainly apparent when using monocular HMDs because each eye sees a different image in this case: depending on the what the wearer is looking at, the eye with the virtual footage may not be dominant, resulting in the inability to see the virtual imagery. Depth of focus: In normal viewing, our eyes adjust themselves based on the distance to the object we are focussing at. When VST techniques are used however, all imagery resides at the same distance from the eyes, and only one focal distance is provided by the camera capturing the footage. So the perceived focal distance differs from the focal distance resulting from conventional viewing. When using monocular VST devices, the problem is even worse and could lead to one of two eyes being out of focus. For OST devices, the problems arise due to the varying focus of the real-world imagery (perceived by the wearer s eyes in the conventional manner) while focal distance of the digital images remains constant. [5] gives an overview of potential fixes to this problem, one of which makes use of a mechanism allowing dynamic ad- 2

3 justments of the focal distance (see figure 2). The method uses an adjustable liquid lens, placed between the eye and the display, which is able to translate the focal distance from infinity to the near view point of the eye. A major advantage of this approach is that it can be used in almost all existing configurations. Visual Displacement: The recorded view shown on a HMD may not match the view the user is used to. The cameras on a VST-HMD may be offset from the expected position (usually they are placed slightly more to the front and top than the users actual eyes) and the distance between the viewpoints for the left and right eye of the user may not match their actual inter pupillary distance. The former affects visuomotor performance and can be compensated for by adaptation of the displacement is not too extreme. The latter may cause eye strain.[8] frame-rates. Lowering latency thus negatively impacts the aesthetic quality and the price of the HMD.[6][11] Eyes moving relatively to the screen: The The Vestibulo- Ocular Reflex rotates the eyes in response to head movements in such a way that we keep looking in the same direction and that the image we see stays stable. Though as discussed in the previous paragraph on latency, the display doesn t update immediately in reaction to head movements. So the display moves with our head while our eyes keep looking in the same direction, and the image displayed on the screen will look as if it is constantly shifting away for a bit then snapping back into position. This same effect will also occur when tracking an object on the screen. This is explained in image 3. In the leftmost image an object is moving from left to right in the real world. The next image shows the same situation but the eye is now tracking the object. The third image is the same as the first but now it is a virtual object on a display with a limited framerate. The last image has the eye tracking the object again and we can clearly see the sawtooth causing the described effect. Depending on the type of display this will present as judder, blur, deformations and/or color fringing. The most obvious solution is again to raise the framerate, but it is currently not feasible to reach high enough frame-rates to eliminate these problems. Another solution is to use a low persistence display where instead of displaying the image for the whole duration of a frame it is only shortly flashed, and it is left up to the human visual system to do correct interpolation. In turn, this can cause problems when the human visual system expects blur, for instance when seeing objects moving at high speeds or during saccades (fast movements of the eye when changing focus to a different subject).[6][12] Figure 2. Schematic overview of an OST device allowing dynamic readjustment of focal distance. Depth of focus: As is the case with AR, the perceived distance of the screen as sensed from eye accomodation is fixed in a HMD, our eyes have to focus their lenses on this depth to see a sharp image. However, the distance information we receive from the HMD through stereo disparity does not match this (and can not match this since it varies for different points on the screen and different points in time). This mismatch of sensory information can cause eye strain and headaches. The only solution to this problem so far is to wait for our brain to get used to it.[7][6] Latency: The computer rendering the virtual world displayed on the HMD needs some time to calculate the image from the correct viewpoint every time we move our head, and the display updates at a fixed interval which can cause delays as well. Because of this, there will always be some degree of latency to updating the screen in reaction to our actions. This can be perceived as the whole world moving slightly, but the viewer can not feel this movement in their vestibular system. The reaction to mismatched vestibular inputs is motion sickness. Because of this, it is essential to keep latency as low as possible. This is done by lowering the quality of the rendering of the virtual world and by using displays with high Figure 3. Moving objects relative to the eye Limited resolution: The pixel density of displays has gone up a lot since the last wave of interest in VR, but individual pixels can still be resolved on HMDs because they are so close to our eyes. Anti-aliasing, which is also used on normal displays helps a lot but for truly immersive VR the resolution still needs to go up. The resolution issue does however seem 3

4 to be one people are willing to put up with (for now), unlike other problems that actively disrupt the VR experience such as motion sickness or headaches.[10] Visual Displacement: VR can also suffer from this problem we discussed in the section on AR, but in this should be easily solvable in the rendering software. It does require the developer to be aware of this though. EVALUATING APPLICATION DOMAINS The goal of using head mounted displays with augmented reality applications is to assist the users in their everyday tasks by improving the user s efficiency, accuracy, awareness and focus. In [3], Livingston describes some of the tasks that will most likely benefit from augmented reality. Examples include visual searching, navigation and tasks requiring situational awareness. One of the main difficulties in evaluating the use of AR in these tasks is the fact that current AR technology is fairly unoptimized. Most of the time, it consist of a single prototype model unfit for use outside of lab environments. Also, the available hardware limitations and the problems that come with it often impede human factor studies. Therefore, we need to make a strong distinction between technological and human factors when assessing the value of AR applications, which is a very difficult task. Figure 4. The warehouse testing environment In [4], the user strain as effect of augmented reality assisted tasks has been studied by analysing the heart rate variability of the test subjects. The experiment was conducted in a warehouse environment, where the test subjects were asked to retrieve a number of items identified by RFID tags (see figure 4). Each user did the experiment twice: once with a paper list of items to retrieve and replenish, and once wearing an seethrough head mounted display using an AR application to list which item to get next (along with its location). The heart rate variability showed no significant differences between the two methods (except during a short familiarization phase at the beginning of the experiment). However, the use of an HMD induced more eye strain (which can be classified as a technological problem factor). The test runs with an HMD also improved the accuracy of the given tasks. Figure 5. a) The view through the head mounted display. b)overhead view showing the correct solution. In [3], Livingston conducted a test designed to overcome the difficulties that arise due to technological factors. His solution is to conduct simple, single-task based tests that use a well designed part of the user interface. His exemplary test evaluates spatial awareness: the test involved seeing objects through walls by projecting them on a see-through HMD (see figure 5). This way, the test tried evaluating depth perception. The results measured the accuracy and variance of the perceived depth. While the depth perception behaved the same in case of real or virtual objects, the accuracy drops significantly with virtual objects, probably due to a lack of other objects at the same distance to compare dimensions with. The purpose of virtual reality is to give the user the experience of being inside of a world that does not physically exist. Providing the input to our most heavily relied on sense, vision, makes HMDs a key instrument in providing this experience. This also makes it extremely hard to perfect them, since humans are experts at detecting anything that does not look right. Some parts of the VR experience can however not (yet) be created, such as touch and acceleration. Therefore it can not really be tested how much a user believes they are in the virtual world. Some aspects that are critical to this can however be tested. R. Ruddle, E. Volkova and H. Blthoff.[9] for instance try different navigation interfaces for virtual worlds, and see how they affect travelling performance. In their tests, participants had to follow a path through an orthogonal grid of corridors. To travel well, a user needs to have an understanding of the space they are in, where they are located in it and how their actions in the real world relate to the virtual world. The paper gives a nice example of a VR-related problem that lies beyond just displaying the world convincingly. CURRENT TECHNOLOGY TODO: An overview of the products that are currently available or being developed. CONCLUSION TODO: 4

5 REFERENCES 1. S. Gilson, A. Fitzgibbon, A. Glennerster, Spatial calibration of an optical see-through head mounted display, J Neurosci Methods. 173(1): , Google, Project Glass, 3. M. Livingston, Evaluating human factors in augmented reality systems. Computer Graphics and Applications, IEEE, 25(6):6-9, J. Tumler, R. Mecke, M. Schenk, A. Huckauf, F. Doil, G. Paul, E. Pfister, I. Bockelmann, A. Roggentin, Mobile in Industrial Applications: Approaches for Solution of User-Related Issues. IEEE International Symposium on, D. Jackel, Head-mounted Displays. Proceedings RTMI, Eli Peli, Visual and Optometric Issues with Head-Mounted Displays, 1996, IS&T/OSA Optics & Imaging in the Information Age, 364, 6 pages 7. Takashi Shibata, Head mounted display, Displays, Volume 23, Issues 12, April 2002, Pages 57-64, ISSN , ( 8. Joong Ho Lee, Sei-young Kim, Hae Cheol Yoon, Bo Kyung Huh, and Ji-Hyung Park A preliminary investigation of human adaptations for various virtual eyes in video see-through HMDS. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI 13). ACM, New York, NY, USA, DOI= / Roy A. Ruddle, Ekaterina Volkova, and Heinrich H. Blthoff Learning to walk in virtual reality. ACM Trans. Appl. Percept. 10, 2, Article 11 (June 2013), 17 pages. DOI= Michael Abrash, When it comes to resolution it s all relative. November 27th 2012, Blogpost, last retrieved December 3rd 2013 from Michael Abrash, Latency the sine qua non of AR and VR. December 29th 2012, Blogpost, last retrieved December 3rd 2013 from Michael Abrash, Why virtual isnt real to your brain. May 15th 2013, Blogpost, last retrieved December 3rd 2013 from 5