A new user interface for human-computer interaction in virtual reality environments

Original Article Proceedings of IDMME - Virtual Concept 2010 Bordeaux, France, October 20 22, 2010 HOME A new user interface for human-computer interaction in virtual reality environments Ingrassia Tommaso 1, Mancuso Antonio 1 (1) : Università degli Studi di Palermo Dipartimento di Meccanica Viale delle Scienze Edificio 8 90128, Palermo (ITALY) Phone: 0039 0916657101; Fax: 0039 091 484334 E-mail : {ingrassia, mancuso}@dima.unipa.it Abstract: This paper concerns the result of a research activity about the problems related to the human-computer interaction (both hardware and software) in virtual reality systems. In particular, a virtual reality application package, called VirDe (acronym of Virtual Design) and developed for the design review in immersive environment, has been used as a case study. After a detailed analysis of the hardware and software systems, and the examination of the main functional requirements, a new user interface has been implemented. This new interface being composed of two input devices (tracked by an optical system) permits the two-hands interaction, instead of the most commonly single-hand interaction. The measurement of the software usability level, made by users evaluation tests, shows that the new interaction system allows the users a more natural and effective software management compared to the previous solution. Key words: Virtual reality; Human-Computer Interaction; Bimanual Interaction; Virtual Prototyping; Tracking systems. 1- Introduction The problem of the user interfaces concerning computers has been always very important and, some time, it has influenced the spreading on a large scale of computer systems. Initially, in fact, the communication between computer and users happened through very complex (and low level) languages and the results were displayed through encoded cards, magnetic tape, printers, etc... The graphic interfaces allowed the wide spread of the personal computers and their usage by common users having few specialist notions. Nowadays, the more and more widespread of the virtual reality systems, also as a consequence of the continuous technological progresses and the remarkable reduction of the costs, requires the study of new human-computer interfaces able to capitalize as best as possible the potentialities of this kind of immersive systems. At the beginning, in fact, the classical interfaces based on the WIMP (Window Icon Monitor Pointer) paradigm have been largely used also in the virtual reality (VR) environments. However, these kinds of interaction systems have shown their limits due, substantially, to their intrinsic twodimensionality. This characteristic, in fact, badly adapts to 3D systems like the virtual reality ones. In this context it has been decided to analyse and develop a new interface for a VR-based software called VIRDE [IC1], aimed to allow a more natural and efficient management of the system. A new three-dimensional user interface has been developed for the manipulation of both scenes and objects. Moreover a new input device, able to replace the mouse in the virtual reality environment has been implemented. Through the new developed interface, the designers can use both left and right hands during the manipulation of the virtual scene, thus allowing a more precise and efficient interaction, also thanks to a better feeling of the selfperception. 2- User interfaces in virtual reality systems Nowadays a standard user interfaces for virtual reality applications still does not exist and many different solutions are continuously under development. The related studies, nevertheless, are quite complex because they involve multidisciplinary knowledge like computer science, psychology, cognitive sciences, ergonomics, design, communication science, and artificial intelligence. Many research activities have been developed to replace the classical 2D desktop interfaces with the 3D ones so allowing a greater freedom [LS1][RM1]. The use of 3D interfaces, nevertheless, still does not allow accurate movements, like the positioning of the cursor in a fixed point of the screen or selecting an item from a list. Most of the difficulty in carrying out such tasks, it s due to the fact that the consumer tries to place the pointer in the 3D space, without any help in stabilizing his hands [MB1] and, for this reason, often, 2D technologies are more effective than 3D [BC1] [TS1]. On the other hand, the management of a 2D input device (like the mouse) with a 3D software interface is quite difficult due to the problems of mapping a 2D interface in a 3D workspace [BW1]. Paper Number 117-1- Copyright of IDMME - Virtual Concept

The mouse is so effective because it is a 2D input device used for manipulating 2D widget on a two-dimensional screen. The mouse, nevertheless, is not the most suitable input device to drag the widgets in the three-dimensional space [OW1]. Moreover, the 2D (desktop) interfaces are not suitable for 3D applications, because they favour the use of only one hand during the interaction with the computer. That represents a limit, any user, in fact, has a natural human bent for using two hands [G1] (distinguished in dominant and not dominant) in combined and diversified way, and this kind of interaction allows a great effectiveness and high precision [HP1] [MB1] [SG1]. All that, moreover, gains a more important role in the VR systems, since it allows the user to use his body as a personal reference system. The relative distance between the hands, for instance, can represents a useful tool to evaluate objects sizes and distances in a precise way, and it can advantageously be used during the interaction with the virtual reality environment. A good three-dimensional user interface able to allow a powerful management of a VR systems, should use 3D input devices, should avoid as much as possible fluctuating objects [BC1], because these are not usually in the real world, and should use algorithms for collision detection [GR1] [KK1] to avoid interference between virtual objects. 3- About the software VIRDE The package VirDe [IC1] is a software, developed at the Università di Palermo - Italy, that integrates the classical techniques of CAE (Computer Aided Engineering) design in a virtual reality system. VirDe can be used for the design review in a VR environment and, for this purpose, it is equipped with many useful tools for CAD modelling and FEM analyses. Through VirDe it is possible to study CAE prototypes in a virtual environment. Both CAD and FEM models can be created (or imported from another software in a common exchange file format) and modified. The main advantage of VirDe is that it allows to analyses, in a more realistic way, CAD and FEM models in a immersive system. In this way designers can better undestand the models and picking out quickly any possible modelling or analyses problems. All that allows to reduce both time and costs of the product design process. Figure 2 The 3D input device : FlyStick The hardware user interface is composed of a wireless 3D input device called FlyStick (Flying Joystick). The FlyStick (fig. 2) has eight push buttons and is tracked in the 3D space through an optical tracking system, the ARTtrack [AR1]. Through the FlyStick the user can manage all the functionalities of the software by moving it in the threedimensional space like a 3D mouse. It is used for the selection of the different buttons of the menus, the sketching Figure 1 Multilevel structure of the menu Paper Number Virtual Concept_P117-2- Copyright IDMME - Virtual Concept

of geometries and, in general, all the input data. The user, at software level, interacts with the FlyStick through an avatar (a pointer), a digital alter ego representing it. This pointer (or wand) moves in the three-dimensional space in a coherent way with respect to the movements of the user during the interaction with the virtual scene. of time to complete a task because of the difficulty to manage the menus and finding the section containing the requested commands. Sometimes, as underlined before, the users were even forced to open up to three levels of menus for each task. All the tools of Virtual Design are accessible through threedimensional menus. These ones have a multilevel structure (shown in fig. 1) depending on the activated functionalities. The default position of the menus is at the screen boundaries, in order to avoid they overlap each others. Any menus can be selected, reduced to icon and moved in the 3D virtual scene through the wand. As can be noticed in figure 3, the root menu, displayed at the beginning of any working session in VirDe, represents the Main Menu through which activating the main functionalities of the system. The user can at most, starting from the Main Menu, activate three levels of different menus, dependently on the requested functionality (fig. 1). Figure 4 - Overcrowding of the screen All these menus, moreover, even if positioned in free portions of the virtual scene, because of their number, can fill the most of the screen (fig. 4). This creates some confusion in the scene, so leaving not enough space to the designer for the interaction with the system. Figure 3 - The main menu 4- Designing a new interaction paradigm One of the most important advantages offered by the virtual reality technology is the freedom in the development of techniques and metaphors of interaction. The main objective in designing new user interfaces in virtual reality environments, nevertheless, should be focused in finding the right compromise between the real world and a purely imaginary one, that should be effective but not completely unnatural to the user [LS1]. Before implementing the new user interface for Virtual Design, some preliminary studies about the system have been developed. For this purpose the Norman interaction model has been used [HH1]. Thanks to this kind of interaction framework the human-computer interaction has been analysed by simulating a working session and evaluating the system response. In particular, during some preliminary test cases submitted to the users of Virtual Design, some difficulties to contemporarily manage the interface and developing the planned tasks have been highlighted. Considering that an effective interface should not remove the attention of the user from the task she/he is developing, unfortunately, in Virtual Design that happens. The designers, in fact, usually spent a lot Figure 5 - Overlapping of menus Besides, this kind of interaction allows making only a task at each time, because if the user wants to begin a new task, other menus can overlap the yet activated ones, with very low possibility to further manage the system. It is quite difficult, in fact, to distinguish different superimposed levels of menu on the screen (fig.5). Moreover, to avoid interfering with the XYZ working planes, some menus are placed, by default, in the top parts of the screen. Unfortunately, this represents a setback because the Paper Number Virtual Concept_P117-3- Copyright IDMME - Virtual Concept

users find it difficult to select the menu buttons placed too up in the virtual scene (fig. 6). sessions, allowing more natural movements and a simpler management of the system. Figure 6 - Menu positioned too up in the screen For all these reasons, a new way of interaction has been implemented. The new developed solution overcomes the highlighted problems, particularly, improving the system interaction time, the interface organization and functionality. 4.1- The new user interface in VirDe The methodology used in developing the new interaction system has been based on the Shneiderman rules [SP1], thank to which it is possible to design, in a structured and optimized way, new user interfaces. To solve the problem related to the large number of menu levels and, consequently, the overcrowding of the screen during some phases of the working sessions, a tablet menu has been implemented. This kind of solution simplifies the current structure of the menus because groups, in a single entity, many menu levels. Only two levels of menu have been grouped, while the third one has been planned in a fixed and functional position of the screen. The choice of a tablet menu, moreover, allows avoiding the menu overlapping and, above all, reduces the interaction time. This kind of menu, in fact, does not need closing many levels when moving from a task to another, because it is possible to switch them each other very quickly. In a first attempt, the tablet menu has been planned to be in a fixed position on the screen but, since this kind of solution causes a reduction of the working space on the screen, it was decided to hide the tablet menu when not essential, and showing it only when necessary. To implement this solution, the menu structure has been linked to the FlyStick and, consequently, to the user hand movements. The choice to join the tablet menu and the FlyStick has required a new input device for all the other tasks, like e.g., objects selections, sketching and scene manipulation. For this reason a new 3D input device, called hand target, has been implemented in the system. Thanks to this kind of solution, it is possible to reduce the arm tiredness during the working Figure 7 - The hand target The hand target (fig. 7), in fact, can be easily worn by the user but, above all, it is lighter, more ergonomic and natural than the FlyStick. The use of a second input device has changed the interaction method in VirDe from a single-hand mode to a bimanual asymmetrical one [G1][HP1][RM1]. The menu interface is managed by the non-dominant hand (through the FlyStick), while the pointer is managed by the dominant hand (through the hand target ). In this way the designer can show or hide the menu only moving his not dominant harm. Obviously the menu can be moved in all the directions. In particular, by slightly moving the FlyStick along the axis perpendicular to the screen, the user can zoom in (or out) the tablet menu. 5- Test cases and results A test (user oriented) aimed to evaluate the software usability level and to compare the two methods of interaction has been made. This kind of test should allow, moreover, to understand the feeling of the users with the new (two-hands) interaction system. The software usability measurement has been performed for both the interfaces through a preliminary practical working sessions and a questionnaire, involving both skilled users and not skilled ones. The skilled users are experienced designers able to work with virtual reality CAD/FEM applications and using 3D input wireless devices. The not skilled ones, instead, are designers that only worked with CAD and FEM software in desktop systems and never used 3D input devices. 5.1 Practical test and questionnaire Before starting the practical tests, a skilled user made a demonstration about the main functionalities of the system (fig. 8), the use of the tablet menu (fig. 9), the FlyStick and the hand target. A schedule of the tasks to carry out has been also presented. Paper Number Virtual Concept_P117-4- Copyright IDMME - Virtual Concept

evaluate the easiness of use of the designed interfaces. In every question, the user had to give a value varying from 1 (low) to 5 (high). Moreover, a free suggestion section has been included to gather the users opinions and suggestions. 6- Results The experiments have been carried out involving ten skilled users and eight not skilled ones. Moreover, with the purpose to appraise the improvements in the interaction with the software and so the difficulties in the system learning, it has been required the users repeating the test two or three times. Figure 8 - Test phase: example of CAD modelling in VirDe To obtain quantifiable results, to evaluate the usability level, the efficiency and the comfort of the two compared interaction methods, all the phases of the practical tests have been timed and the numerical values of the questionnaires have been processed and analysed. The obtained results can be summarized as follows. As regards the interaction way using the fixed menus (single hand) interface, a very important characteristic to notice is the easiness of the hand coordination during the selection of the menu functionalities, both in case of the big push buttons and the little ones. Figure 9 - Test phase: presentation of the tablet menu The practical tests have been planned to explore enough in detail the software, through the navigation of the menus, the selection of objects, the scene modification, etc... In particular, during the practical tests, the users had to create an extruded solid, a loft surface and modify the scene by rotating the 3D objects, zooming and panning. The creation of the extruded solid required: - navigating inside the menus to access the extrusion modelling functionality; - selecting the sketching plane; - creating the 2D profile using straight lines; - extruding the profile with a required depth. To model the loft surface, instead, it needed to: - navigate inside the 2D modelling menu; - selecting the sketching plane; - creating the 2D profile using a NURBS curve; - creating an auxiliary planes; - sketching a NURBS curve; - selecting the two curves to create the loft surface. After the practical test, a questionnaire has been proposed to gather general information about the level of expertise of the users with the computers and the virtual reality systems, and to The navigation of the system with the fixed menus turns out to be immediate, because of its structure much similar to the WIMP interfaces. In this case, in fact, the fixed menu interface is very similar to the one of the traditional desktop systems, and that entails a greater easiness of use. As expected the users, especially the not skilled ones, have had also some problems during the tests, many of which yet identified during the analysis and design phase of the new system interface. One of the main problems of the fixed menus interface is related to the large number of displayed windows. If not closed, in fact, the different menu levels fill a wide portion of the screen. Users, moreover, have found difficult to click some menu buttons positioned in the top side of the screen. With regard to the new interface, at the first interaction, almost all the users have had some difficulties, above all, in coordinating the two devices by means of the hands. The main problem, particularly, was related to finding the exact position of the moving tablet menu to select the buttons. On the other hand, they have really appreciated the novelty of the hand target device, which allows a greater freedom than the FlyStick, especially, for the sketching and management of the virtual scene. The new implemented interface has shown a high level of learning easiness; almost all the users, yet during the second working session, have improved their performances by reducing the interaction time and coordinating the two hand devices in a more effective and natural way. After the first interaction, the time to complete the assigned tasks lowered notably, becoming comparable (sometimes lower) to that obtained with the fixed menus interface. Paper Number Virtual Concept_P117-5- Copyright IDMME - Virtual Concept

The obtained results show the single-hand interface allows a higher working speed for the completion of the tasks than the new interface but only during the first working session. It was demonstrated, nevertheless, after three/four working sessions, the two-hands interface turns out to be more effective and faster than the old interface. All that because the two-hands interface needs a period of training to learn how to coordinate the two hand-devices. Moreover, if many menu levels are open and superimposed the time and difficulty of interaction considerably increases by using the single-hand interface. A very important advantage, emerged from the analysis of the questionnaires, is that the users have not tired their hands and arms. That because the non dominant hand, wearing the FlyStick, stays along the body for a long time, while the dominant hand, even if must stay up longer, wears a very light input device (the hand target) so reducing the harm tiredness. 7- Conclusions In this paper, problems and related solutions to the human/computer interaction in virtual reality environments have been investigated. In particular, the user interface of a novel software for the design review in VR environments has been studied and improved. Two important innovations have been implemented: a tablet menu and a second input device (besides the existing one) to allows the bimanual interaction. Thanks to these solutions, very good results in terms of speed and performances requirements have been achieved. The adopted strategy has been oriented to the designer which represents the final user of the application. The new interface allows to manage the system through a new menu structure and the use of two input devices: the FlyStick and the hand target. The first, even if ergonomic, is quite complex and not suitable for the pointing and the sketching, so it has been used to manage the moving tablet menu and to activate the functionalities through its push buttons. The second, instead, very small and light, has been used as the pointer to activate and select all the system functionalities. The new implemented menu, according to the experimental tests, is very simple and easy to use. The results show the users appreciate the moving tablet menu and the possibility to interact both with left and right hands. In conclusion, the new developed interface of Virtual Design represents a good improvement of the system allowing better performances and increased usability level, making it more similar to the way of interaction the designers usually have in a real working environments. 8- References [AR1] http://www.ar-tracking.de [BC1] Bowman, A., Coquillart, S., Froehlich, B., Hirose, M., Kitamura, Y., Kiyokawa, K., Stuerzlinger, W., 3D User Interface: New Direction and New Perspective. International Journal of Virtual Reality, pp.3-14, Volume 5, Number 2, June 2006 [BW1] Bowman D. and Wingrave C., Design and Evaluation of Menu Systems for Immersive Virtual Environments, Proceedings of IEEE Virtual Reality, 2001 [G1] Guiard, Y., Asymmetric Division of Labor in Human Skilled Bimanual Action: The Kinematic Chain as a Model, The Journal of Motor Behavior, 19 (4), 1987, pp. 48-65 [GR1] Govindaraju N., Redon S., Lin M., and D. Manocha. CULLIDE: interactive collision detection between complex models in large environments using graphics hardware. SIGGRAPH Workshop on Graphics Hardware, pp. 25-32, 2003. [HH1] Hutchins E. L.; Hollan J.; Norman D.; Direct Manipulation Interfaces, Human-Computer Interaction, Volume 1, Issue 4 December 1985, pages 311 338 [HP1] Hinckley, K., Pausch, R., Proffitt, D., Patten, J., Kassell, N., Cooperative Bimanual Action, Proceedings of ACM CHI '97 Conference on Human Factors in Computing Systems, 1997 [IC1] Ingrassia T., Cappello F.; VirDe: a new virtual reality design approach; International Journal on Interactive Design and Manufacturing, pp. 1-11, Volume 3, Number 1 / February, 2009 [KK1] Kitamura Y., Kishino F., Consolidated manipulation of virtual and real objects, In Virtual Reality Software and Technology 1997, pp. 133-138 [LS1] Lindeman R., Sibert J., Hahn J., Towards usable VR: an empirical study of user interfaces for immersive virtual environments, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, pp. 64-71, 1999 [MB1] Mine, M., Brooks, F., Séquin, C., Moving Objects in Space: Exploiting Proprioception in Virtual-Environment Interaction, Proceedings of the ACM SIGGRAPH '97 Conference on Computer Graphics, (1997), pp. 19-26 [OW1] Oh J., Stuerzlinger W., Moving Objects with 2D Input Devices in CAD Systems and Desktop Virtual Environments, Proceedings of Graphics Interface 2005, pp. 195-202 [RM1] van Rhijn A., Mulder J., Spatial input device structure and bimanual object manipulation in virtual environments, Proceedings of the ACM symposium on Virtual reality software and technology, pp. 51-60, 2006 [SG1] Szalavári Z., Gervautz M., The Personal Interaction Panel a Two-Handed Interface for Augmented Reality, Computer Graphics Forum, Vol. 16 Issue 3, pp. 335 346, 2008 [SP1] Shneiderman B., Plaisant C., Designing the User Interface: Strategies for Effective Human-Computer Interaction (4th Edition), Addison Wesley, April 2004 [TS1] Teather R., Stuerzlinger W., Assessing the Effects of Orientation and Device on (Constrained) 3D Movement Techniques, IEEE Symposium on 3D User Interfaces 2008, 43-50. Paper Number Virtual Concept_P117-6- Copyright IDMME - Virtual Concept