EyeScope: A 3D Interaction Technique for Accurate Object Selection in Immersive Environments

Transcription

1 EyeScope: A 3D Interaction Technique for Accurate Object Selection in Immersive Environments Cleber S. Ughini 1, Fausto R. Blanco 1, Francisco M. Pinto 1, Carla M.D.S. Freitas 1, Luciana P. Nedel 1 1 Instituto de Informática Universidade Federal do Rio Grande do Sul (UFRGS) Caixa Postal Porto Alegre RS Brazil {ughini,frblanco,fmpinto,carla,nedel}@inf.ufrgs.br Abstract. This paper proposes a technique for object selection in immersive environments using a head-mounted display (HMD) and a data glove. The developed technique extends the well known gaze-directed selection technique. We propose a zoom control to focus a specific region of interest thus providing more precision in object selection. During zoom, the viewing direction obtained from the head-mounted display orientation becomes finer avoiding sudden changes in the view due to the combination of zoom and direction modification. These characteristics make the object selection easier, more accurate and more comfortable. Tests performed with 24 subjects and 9 scenarios confirmed our hypotheses. 1. Introduction Users of immersive virtual environments usually intend to manipulate objects that are part of the scene. Reaching objects may also require navigation through the virtual environment since they might not be near to the user. Manipulation requires a previous selection of the object of interest, and navigation is also frequently based on selecting a target point to indicate the path that the user wants to follow. Figure 1. Four hand positions to control zoom level. From left to right: open, almost open, almost closed, and closed. Developing simple and comfortable selection and manipulation techniques for 3D environments has been a research issue for many years, and there are several possibilities depending on the application-specific tasks, different devices and interaction metaphors. At a high level, these techniques can be classified in two categories according to the interaction metaphor used: the exocentric and the egocentric metaphors [Poupyrev et al. 1998]. In the exocentric metaphor, the proportions between the user and the objects are not maintained, assuming that the user interacts with the environment

2 from outside its reference system. This is known as god s eye viewpoint. In the egocentric metaphor, the user is part of the virtual world maintaining the dimensional coherence between him and the objects manipulated. This class is further subdivided into two metaphors: virtual hand, where the user reaches and grabs the object of interest with a virtual hand, and virtual pointer, where the user interacts with the target object by pointing at it. Although there are several techniques described in the literature [Bowman et al. 2005], some of them were designed for desktop environments and others are best suited for immersive environments. In the case of wearable devices, such as head-mounted displays and data gloves, one can take advantage of the degrees of freedom provided by these devices and extend classical and simple techniques to improve user performance. This paper focuses on object selection, more precisely, on a new approach for object selection in 3D immersive environments using a head-mounted display and a data glove, the eyescope. The use of head-mounted displays provides an egocentric immersion experience, i.e. the user is proportional to the virtual world, being immersed in it. The data glove is the natural device for implementing the virtual hand metaphor. The proposed technique extends the gaze-directed selection technique, but the user does not need to navigate through the environment. We propose a zoom control using the data glove to focus a specific region of interest, enlarging it and, at the same time, the viewing direction control given by the head-mounted display orientation becomes finer. This more accurate control is needed because the gaze-directed technique is very sensitive to the low resolution of the display, noise in the sensor s orientation and target objects distribution in the scene, especially when there are distant, small or semi-occluded objects. The extended technique makes the object selection easier, more accurate and comfortable. This paper is organized as follows. In Section 2 we discuss some existing techniques according to requirements for object selection. In Section 3 we explain the eyescope technique, detailing all issues involved in the selection operation, and how the technique handles the user s reaction. In Section 4 we show the testbed application developed to evaluate the technique and compare it with the gaze-directed selection. In Section 5 we describe the evaluation criteria and parameters, while in Section 6 we explain how the experiments were performed. Finally, in Section 7 we present and discuss the results obtained from the experiments and, in Section 8, we conclude this work with some observations. 2. Related Work Most three-dimensional interaction techniques are based on few basic metaphors. For example, navigation can be accomplished by gaze-directed steering, which implies that the user moves in the direction he/she is looking at. The same looking-at metaphor can be used to select objects: the user can point at the object of interest using 2D, 3D gaze or 3D hand techniques [Bowman et al. 2005]. A selection technique needs also a confirmation of selection by means of some implicit or explicit command and a feedback. Through observation of existing techniques described in the literature, we identify some basic requirements for object selection techniques: 1. Accuracy. The technique must provide the user with an easy and precise way

3 to select the desired object, independent of its size, location or orientation, minimizing wrong or null selection, which slows down user performance or even may cause him to abandon the task. 2. Reachability. It must be possible to reach any desired object without moving through the environment. 3. Time saving. Users, specially advanced ones, like to minimize the time to select an object. This is precision dependent because incorrect selections waste time. 4. Comfort. The user must feel comfortable during the whole procedure, and self confident concerning correct selection. 5. Simplicity. Device manipulation must be intuitive and with few controls, yielding easy memorization and short learning time. 6. Immersion sense. Traditional 2D interaction devices, like mouse and keyboard, as well as menus, lists, labels and buttons should be avoided when adopting wearable devices. These techniques can clutter scene visualization and disrupt user attention. When the visualization device has a poor resolution this can be a more serious problem. 7. Error and constraints tolerance. It must be possible to use devices with low quality interaction or restricted degrees of freedom without loosing performance. A useful selection technique must tolerate inaccurate or limited interaction devices, in order to be employed with various hardware platforms. Several authors report the implementation of selection and manipulation techniques in virtual environments like virtual hand and virtual pointers, either with the regular mouse or data gloves [Poupyrev et al. 1996, Bowman and Wingrave 2001, Nedel et al. 2003]. In the virtual hand technique, the user s hand is explicitly represented in the virtual environment. Its position and orientation in the VRE are provided to the system by specific input devices. To select an object, the user should intercept the object with the virtual hand and inform his/her intention to the application. This can be provided simply by pressing a button on the input device or performing a specific gesture with the fingers or the whole hand in the case the user is interacting by means of a data glove [Poupyrev et al. 1996, Bowman and Wingrave 2001, Nedel et al. 2003]. However, as stated by Poupyrev et al. [Poupyrev et al. 1998], selection by pointing provides better performance than virtual hand-based techniques because the latter techniques require more physical hand movement from the user. So, although fulfilling most requirements above, virtual hand-based techniques fail to comply with requirement comfort, which can be decisive for user performance. Selection by pointing techniques can be accomplished through many methods but we restrict ourselves to comment those that can be implemented with the data glove. Ray-casting is one of the most used techniques to select and manipulate objects [Poupyrev et al. 1998, Bowman et al. 1999]. This technique consists of the representation of an infinite and semi-transparent ray from the user s hand towards the point indicated by the hand orientation. To complete the selection, once the object of interest is pointed at, the user issues a specific command to the application using the input device. This technique is very powerful but it can fail the requirement accuracy when high precision selection is required, especially in a dense environment or with faraway objects

4 [Poupyrev et al. 1998, Bowman et al. 1999]. Jacob [Jacob 1990] describes a technique for selecting an object by tracking the user s eye movement, and determining to where the user is looking at. This gaze-directed technique [Bowman et al. 2005] has been used in several works [Bleser and Sibert 1990, Tanaka 1999, Yamato et al. 2000], and seems to fit well to the requirements reachability, time saving, comfort, simplicity, and immersion. However, it is not accurate, failing the first requirement. In our technique, we extend the gaze-directed technique by controlling the location of a cursor on the target object through zooming into it and enlarging its projection. We use the head-mounted display orientation to indicate the target s direction and the data glove openness to control zooming. Depending on the hand closing gesture the image is enlarged, facilitating cursor positioning on the target object. During zooming, re-orientation of the HMD is attenuated to avoid sudden changes. A closed hand corresponds to the confirmation of selection. In this way, eyescope fulfills all the requirements listed above as we will discuss in the following sections. 3. Eyescope In this section, we describe the eyescope technique in detail, showing the mapping between the input from both the head-mounted display and the data glove to operations performed in the virtual world. The essential characteristic in our extended gaze-directed selection is the control of the viewing orientation during zooming, when sudden changes of orientation could make the user to loose control of the cursor Technique design When the user is immersed in a virtual world, he/she can look at any direction as in the real world. At the center of the view there is a small target in the form of a cross, which is used to point to the desired object. Hand closing gestures control zooming level, which actually provides a way of enlarging the scene visualization. In order to have more accuracy, the user can point to a cluster of objects and activate the zoom-in command by slowly closing the hand with the data glove, causing the scaling of the scene. This allows a more precise indication of the object of interest. In the case of changing the target, it is possible to zoom out by opening the hand, and returning to the initial state to choose a new target. The actual selection of the object can be made at anytime, just by closing the hand when the cross cursor is on the target s projection Range definition In the virtual world, users are allowed to rotate 360 degrees around their own body. The up and down head movement is limited to 180 degrees. Combining these two rotations gives the user full angular navigation. A roll movement with the head (usually also with 180 degrees) to the left or to the right is also allowed, but it affects only the scene visualization. The zoom level starts at a user proportional view of the environment, i.e. in a 1:1 view. The maximum zoom is limited to 500 times the normal scene size in the application. Moreover, the zooming is executed through the enlargement of the projected scene and not with a real approximation, so passing through objects is avoided.

5 3.3. Mapping hand gestures to operations in the virtual world Hand gestures are used both to control zooming and confirm the selection of the object pointed by the cross target. A first approach of controlling the zooming level was to make it proportional to the hand closing movement, i.e., depending on how much closed the hand was the greater was the zoom level. However, that solution proved to be too inadequate due to device inaccuracy or noise (fast oscillations, and even some oscillations when the hand is static). Both cases introduced instability in objects visualization, keeping their projections changing constantly in the final image. Noise during the capture of the viewing direction makes the objects oscillate vertically and horizontally, and this inconvenient would be worse with a conventional zoom, increasing the oscillations. So, with the zooming level proportional to the closed hand degree, noises would do the size of objects change repeatedly. Thus, the alternative for zooming level control was to use a state-based control, being the closing states of the hand classified in four different situations: entirely open hand, almost open hand, almost closed hand, and entirely closed hand. This allows the definition of larger ranges for the boundary values defining hand s states, so the noise rarely is high enough to cause a sudden change of state. Also, the noise can be smoothed by applying filtering. We used a simple mean filter considering only the five or 10 last measured values to avoid delay in the device s response time while smoothing the transition between zooming levels. Figure 2 shows data glove s collected data demonstrating different hand situations (Figure 1) along a certain time: fully open, almost open, almost closed and closed hand. Even with the noise peaks in the graph of Figure 2, the data collected is kept in the same interval, thus avoiding problems related to low quality devices. Figure 2. Zooming intervals with noise When the user starts closing the data glove, hand is almost open, zooming-in is activated and the focused image starts to enlarge proportionally. An almost closed hand causes zooming to stop and the zooming level is kept steady. A closed hand means that the object pointed by the target should be marked as selected. The hand fully open means no zoom or terminating zooming-in with a smooth return to initial state. In terms of implementation, to increase performance, zooming is calculated in image space, being the objects scaled after their projection.

6 3.4. Mapping head movements to operations in virtual world As mentioned before, the head orientation controls the viewing direction in the virtual world. The observer is not allowed to move through the environment: he/she is just allowed to rotate the body thus rotating the head-mounted display. By doing this, an object is brought to the center of the field of view and the selection command can be issued. The observation direction in the virtual environment is defined by the head-mounted display angular position, its coordinates being recorded constantly and transferred to the virtual environment to set the camera parameters in the same way that the HMD is settled in the real world. Such control is simple and intuitive [Jacob 1990], so the user can just move the head like he was physically in the virtual world. Once the object of interest is at the center of his/her vision, with the cross target on it, the selection command can be sent by closing the hand wearing the data glove. With the introduction of zooming, a finer control of how HMD orientation affects viewing direction is needed. In the simple gaze-directed selection technique, the angles obtained from the HMD trackers are directly used in the virtual environment imposing a viewing direction in a 1:1 scale, as shown in Figure 3 (b). Zooming without changing orientation would be a simple matter of enlarging the image. (a) (b) (c) Figure 3. Correlating mappings: real world movement (a), simple mapping (b), eyescope mapping (c). However, when a zooming command is issued by a closing hand movement, the relation between the HMD orientation and the viewing direction must be adapted to prevent from large changes in the viewing direction. Instead of simply passing HMD orientation to the virtual world, an angular variation is computed as the difference between the new HMD orientation and the previous one. These relative values are used to change the viewing direction as a scale factor. Thus, the larger the zooming level is, the smaller will be the angular variation in the viewing direction for the same variation in the headmounted display orientation as is shown in Figure 3 (c). Of course, the use of this non-proportional mapping has a side-effect: there will be a difference between the head-mounted display angular position and the virtual world angular position caused by the scaling. During zooming out, the angular positions are adjusted accordingly, so when the fully open hand situation is reached the orientations will be the same: the zooming level reduction and the reference realignment occur simultaneously, in a gradual but fast way. In this way, we maintain the accuracy in the control

7 of the viewing direction while providing a fast way to point at the target by means of the zooming level increase. 4. Testbed Application To evaluate the technique we built a simple three-dimensional environment, containing spherical objects floating over a chessboard-like floor. This 3D environment was built as a GLUT-based OpenGL application written in the C++ programming language Apparatus We used a VFX3D head-mounted display with yaw, pitch and roll orientation sensors, maximum screen resolution of 640 by 480 pixels, and 60 frustum. For capturing hand gestures, we used a 5DT Data Glove from Fifth Dimension Technologies. This data glove has a rotation and orientation sensor as well as five flexion sensors for the fingers Scenario and experiment design The application takes as input a series of text files. Each file describes a collection of objects in the 3D virtual environment; for each object the file contains an id, its (x,y,z) coordinates, size and color. We designed 9 scenarios (Figure 4) using random distributions, regular cluster of objects and partial occluded cluster of objects. A target object is specified in each input file, being in a different situation in each scenario. In all the cases the user has the task of selecting the target object. 5. Evaluation To evaluate the eyescope technique we chose to compare it with its similar gaze-directed selection technique. The comparison took into account data logged by the application as well as data collected from questionnaires Hypotheses We considered three hypotheses, as explained below. H1. Eyescope is more efficient in the cases of far, small or partially occluded objects. With this hypothesis we verified the time spent for selecting objects in dense or faraway clusters, since the gaze-directed selection was known as not having good performance in such conditions. H2. Eyescope is always more accurate, avoiding wrong selections. With this hypothesis we verified the accuracy and error tolerance of eyescope. H3. Eyescope will be chosen as better than gaze-directed selection, and the users will feel more confident with it. This hypothesis was verified through the analysis of post-experiment questionnaires Independent and dependent variables As independent variables we used: (1) age; (2) gender; (3) previous use of virtual reality devices; (4) previous use of 3D applications, like games, CAD, etc.; (5) previous participation in experimental studies; (6) the technique being used (eyescope=with zoom/gazedirected=without zoom), and finally, (7) the scenario.

8 Figure 4. Scenarios 1 to 9 (from left to right, top to bottom) We defined as dependent variables (1) task completion time and (2) number of errors until task completion. Task completion time was measured from the time the scenario is displayed with the order until the correct selection of the object of interest. Number of errors was computed from the number of wrong selections (selection of an object different from the task specification) and number of clicks issued in empty space. 6. Experiment The experiment consisted of the selection of a target object in each scenario. Data about selection time and number of errors during selection was recorded automatically by the application, while that about the users and their satisfaction was collected through a simple questionnaire Subjects and tasks We have performed the tests with 24 subjects. Our population was heterogeneous, consisting of 9 women and 15 men aging from 19 to 37 years old. Each of them tested the two techniques with the 9 different scenarios (Figure 4), resulting in 18 tests per user, and a total of 432 tests.

9 6.2. Procedure Before the real test, the users were given a training scene (Figure 5) where they were able to practice with the techniques. The duration of this phase was defined by each user: they could skip it and start the real test at any time. The order in which a user tested each of the techniques was interleaved so that 12 series of tests were performed starting with the eyescope and 12 others starting with gaze-directed selection. Figure 5. User in training phase A pre-test form was answered by the users, which provided us with the following information about the subjects: 45% of the users have already had experiences using virtual reality equipments (like gloves, HMD, 3D mouse), all of them have access or work with computers daily, 59% have never participated in users tests and 55% were not used to highly immersive interactive applications. As soon as each scene was loaded for testing, the instruction with the goal was shown at the bottom of the screen, always notifying the user about which technique was enabled at that time. We made use of background music to improve the immersive sensation in our tests [Kyriakakis 2001] and we also play sounds for feedback in some events, for instance to indicate wrong object selection and task completion. The developed application was instrumented to collect all desired information through logging. As each user test ended, a log file was generated containing the time the user spent in each scenario and how many empty and wrong selections were made. 7. Results and discussion For the statistical analysis of the collected data, 9 variables were considered: sex, age, previous use of virtual reality equipment (yes or no), familiarity with immersive environments (yes or no), previous participation in experiments, test scenario, technique (eyescope: zoom=yes; gaze-directed: zoom=no), completion time and number of errors. Before analyzing each hypotesis, we verified a possible correlation of completion time and nuber of errors with the other 7 variables. Table 1 shows the correlation between each variable. In this table, only the absolute value is relevant. The left inferior side of the table shows the isolated correlation, whereas the right superior side shows the correlation with the influence of all the other variables. The analysis of these results shows that the

10 number of errors and the time for task completion is highly correlated with the use of the zoom (eyescope technique). Also, according to these results, it can be observed that the variables Time and Errors does not have relevant influence from the other variables. They are intimately correlated only between themselves and with Zoom. Table 1. Correlations table 7.1. Hypothesis H1 Testing the first hypothesis Eyescope is more efficient in the cases of far, small or partially occluded objects required analysis of task completion time in all scenarios, recalling that scenarios 2 and 4 corresponded to far objects; scenarios 3, 6 and 9 exemplified occluded objects; and 1, 5, 7 and 8 represented randomly distributed objects. Time data for all the 9 scenarios can be observed in Figure 6(a). Task completion time for scenarios 1, 3, 5 and 6 with the eyescope technique were greater than those for completing the task with simple gaze-directed selection technique, but in scenarios 2, 4 and 9, users performed much better with the eyescope technique. In average, taking into account all the scenarios, users were 1.72 seconds faster when using eyescope. ANOVA test applied to the data showed that the differences are significant only in scenario 2 and 6. In scenario 2, users performed better when using eyescope (F = ; p < ) while in scenario 6, task completion time was significantly lower when using the gazedirected technique (F = 6.754; p < ) When considering all the scenarios, ANOVA test showed significant difference between eyescope and gaze-directed selection techniques (F = ; p < ), with eyescope showing a mean time of seconds and standard deviation of 6.47, and gaze-directed selection, mean time of seconds and standard deviation of Hypothesis H2 The second hypothesis Eyescope is always more accurate, avoiding wrong selections required testing the number of errors during the planned tasks. The overall number of errors was smaller when the users employed the new technique. This was even more evident in the scenes where the objects were closest to each other, partially occluded or very distant from the user. Figure 6 (b) shows number of errors per scenario. ANOVA test also showed significant difference between eyescope and gaze-directed selection technique (F=38.291; p < E 09). When observing each scenario separately, all scenarios except 1 and 6 showed significant differences. In scenario 2, where the objects are distant from the user and closest to each other, the average number of errors decreased from 4.17, to 0.29, with eyescope, this difference being significant (F = 9.524; p < ). A large difference was also observed for

11 (a) (b) Figure 6. Task completion time for each scenario (a), and number of errors (wrong selections) in each scenario (b). scenario 4, where the objects are rather clustered (F = 6.872; p < ), and for scenario 9, with partially occluded objects (F = 9.632; p < ) Hypothesis H3 From the questionnaires, it was observed that the proposed selection technique showed a good acceptance rate among the users. Only 2 of the 24 users preferred the gaze-directed selection technique in the majority of the scenes. While only 29.71% said the zoom sometimes confused them, most of them (92%) said the proposed technique can be used instead of the other one in all the presented cases. 8. Conclusions We proposed eyescope, an extension of the well-known gaze-directed technique to select objects in an immersive environment using a head-mounted display and a data glove. Eyescope provides a zoom control to enlarge the area where the object of interest is located. A key characteristic is that it controls the variation of viewing orientation during zooming to avoid sudden changes of orientation that could make the user loose control of the cursor used to point at the target object. Tests were performed with 24 users selecting objects in 9 different scenarios. From the results we concluded that the envisioned acceptance and efficiency of eyescope was confirmed, both for task completion time and number of wrong selections. Taking into account all the scenes, the total time to accomplish the tasks dropped from seconds with the gaze-directed selection to seconds with eyescope. In the same way, the mean number of errors until the user had successfully completed all the tasks varied from to Therefore, the technique has shown to be effective in all the situations where a fine control was necessary. Zoom helped the users to reduce the number of erroneous selections as well as the time needed to select the correct object. Regarding future work we intend to use the other degrees of freedom of the data glove to encode different commands to allow either navigation through the environment or manipulation of the selected object.

12 9. Acknowledgments This work has been supported by CNPq (Conselho Nacional de Desenvolvimento Científico e Tecnológico) and CAPES (Comissão de Aperfeiçoamento de Pessoal de Nível Superior), both Brazilian national science funding agencies. We also thank the colleagues that participated as subjects for their help and suggestions. References Bleser, T. W. and Sibert, J. (1990). Toto: a tool for selecting interaction techniques. In UIST 90: Proceedings of the 3rd annual ACM SIGGRAPH symposium on User interface software and technology, pages , New York, NY, USA. ACM Press. Bowman, D., Kruijff, E., LaViola, J., and Poupyrev, I. (2005). 3D User Interfaces - Theory and Practice. Addison Wesley. Bowman, D. A., Johnson, D., and Hodges, L. F. (1999). Testbed evaluation of ve interaction techniques. In Proceedings of the ACM Symposium on Virtual Reality Software and Technology (VRST 99), pages Bowman, D. A. and Wingrave, C. A. (2001). Design and evaluation of menu systems for immersive virtual environments. In Proceedings of the IEEE Virtual Reality Conference 2001 (VR 01), pages Jacob, R. J. K. (1990). What you look at is what you get: eye movement-based interaction techniques. In CHI 90: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 11 18, New York, NY, USA. ACM Press. Kyriakakis, C. (2001). Fundamental and technological limitations of immersive audio systems. In Jeffay, K. and Zhang, H., editors, Readings in multimedia computing and networking, page 0. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA. Nedel, L. P., Freitas, C. M. D. S., Jacob, L. J., and Pimenta., M. (2003). Testing the use of egocentric interactive techniques in immersive virtual environments. In Interact 2003, pages IFIP. Poupyrev, I., Billinghurst, M., Weghorst, S., and Ichikawa, T. (1996). The go-go interaction technique: non-linear mapping for direct manipulation in vr. In UIST 96: Proceedings of the 9th annual ACM symposium on User interface software and technology, pages 79 80, New York, NY, USA. ACM Press. Poupyrev, I., Weghorst, S., Billinghurst, M., and Ichikawa, T. (1998). Egocentric object manipulation in virtual environments: Empirical evaluation of interaction techniques. In Computer Graphics Forum, EUROGRAPHICS 98 Issue, pages Tanaka, K. (1999). A robust selection system using real-time multi-modal user-agent interactions. In IUI 99: Proceedings of the 4th international conference on Intelligent user interfaces, pages , New York, NY, USA. ACM Press. Yamato, M., Inoue, K., Monden, A., Torii, K., and ichi Matsumoto, K. (2000). Button selection for general guis using eye and hand together. In AVI 00: Proceedings of the working conference on Advanced visual interfaces, pages , New York, NY, USA. ACM Press.