VR-MOG: A Toolkit For Building Shared Virtual Worlds

LANCASTER UNIVERSITY Computing Department VR-MOG: A Toolkit For Building Shared Virtual Worlds Andy Colebourne, Tom Rodden and Kevin Palfreyman Cooperative Systems Engineering Group Technical Report : CSEG/11/1995 CSEG, Computing Department, Lancaster University, LANCASTER, LA1 4YR, UK. Phone: +44-524-593041; Fax: +44-524-593608; E-Mail: cseg-info@comp.lancs.ac.uk

UIST'95 Submission Category of Submission: Paper Title: VR-MOG: A toolkit for building shared virtual worlds Primary Contact: Tom Rodden Computing Department Lancaster University Lancaster LA1 4YR. U.K. E-mail: tom@comp.lancs.ac.uk Phone: (+44) 1524 593823 Fax: (+44) 1524 593608 Authors: Andy Colebourne Computing Department Lancaster University Lancaster LA1 4YR. U.K. E-mail: andy@comp.lancs.ac.uk Kevin Palfreyman Computing Department Lancaster University Lancaster LA1 4YR. U.K. E-mail: kev@comp.lancs.ac.uk Tom Rodden Computing Department Lancaster University Lancaster LA1 4YR. U.K. E-mail: tom@comp.lancs.ac.uk Abstract The paper presents a user interface toolkit to support the rapid construction of cooperative virtual environments. The toolkit is based on extending a 2D model of interaction to 3D virtual environments. The toolkit populates a virtual world with shared 3D interface objects which allow deeper semantic interaction with the virtual world. A principal focus of the toolkit is on the provision of accessible facilities to manage interface configuration and tailoring. Most existing facilities to manage cooperative environments tend to be language centred and provide only limited tailorability for users and developers. In addition, the current generation of virtual environments focus on the direct manipulation of virtual worlds. The model of interaction supported tends not to consider deeper application semantics and limited facilities are provided for developing richer applications. In this paper we present a multi-user interface toolkit that provides management facilities in a manner that allow the access to deeper application semantics. This is achieved by building upon existing 2D models of interaction and allowing virtual environments to be constructed from shared virtual interface objects. The toolkit has been used to rapidly construct a number of semantically rich cooperative virtual environments. We would also suggest that the migration of our user interface model offers the possibility of moving exiting 2D applications to virtual environments. KEYWORDS: Virtual environments, Cooperative Interfaces, Interface Toolkits.

VR-MOG: A toolkit for building shared virtual worlds Andy Colebourne, Tom Rodden. Kevin Palfreyman Computing Department Lancaster University Lancaster LA1 4YR. U.K. E-mail: {andy, tam,kev}@comp.lancs.ac.uk INTRODUCTION Over the last few years a number of systems have emerged which support the development of shared virtual environments[1, 2, 3, 4]. The systems allow a number of users to simultaneously populated shared virtual worlds. The environment is constructed from geometric objects and users are allowed to traverse the environment and manipulate the objects in them. Systems of this form offer significant possibilities for the support of cooperative work. Abstract worlds may be inhabited and shared by a number of geographically remote cooperating users who may exploit the properties of the virtual world to support their work. A number of research systems have examined the development of shared virtual environments for this purpose[1,5]. In general, the development of shared virtual environments has considered worlds with limited interactional properties. Users of worlds may manipulate objects within the world or even properties of the environment. However, limited consideration has been given to the development of more complex applications where interaction has significant semantic effects on an application. To use the terminology of Took[6], VR worlds have focused on surface interaction with limited consideration of deeper application interaction. One reason for this focus has been the lack of a model of interaction for virtual worlds. The HCI community have developed a number of different approaches to interaction for 2D interfaces. The most mature of these include event driven approaches[7] and constraint based models[8]. These models make a clear separation between interface and application semantics. This separation embodies good engineering principles for the construction of interactive systems[9]. We believe these principles are equally valid for virtual environments. To this end we have migrated the existing event driven Permission to copy without fee all or part of this material is granted provided that the copies are not made or distributed for commercial advantage, the ACM copyright notice and the title of the publication and its date appear, and notice is given that copying is by permission of the Association for Computing Machinery. To copy otherwise, or to republish, requires a fee and/or specific permission. model of interaction prominent in 2D interaction to the virtual world. In this paper we present an approach to multi-user interface construction within a toolkit called VR-MOG that allows the rapid construction of virtual environments while also providing facilities to manage the semantic aspects of the world. The approach we have adopted is to consider virtual worlds as compositions of different world objects. Each world object has its own presentation properties which determine its position and appearance in the virtual environment. In addition, objects may have interactive properties which allow access to deeper application semantics. By directly supporting the construction and editing of these world objects we can rapidly develop a range of different shared worlds. This allows us to adopt a prototyping approach to world development where properties of the world can be readily amended to meet the demands of users. We have realised the VR-MOG environment by augmenting an existing distributed multi-user VRenvironment and the extension of an existing user interface toolkit for the X-window system[11]. The migration of a callback model of interaction from X- windows allows our toolkit to support the development of rich virtual world with application functionality presented in the virtual environment. BACKGROUND AND MOTIVATION Our aim is to provide a set of facilities that allow the realisation of effective virtual worlds. The need to develop interactive systems which sit easily with the users ensures that an iterative model of development is essential. This is particularly true in the case of cooperative virtual environments where the demands of users are currently unknown. The provision of appropriate development and management facilities is central to supporting this form of development model. Support for iteration in the development process is closely associated with minimising the commitment involved in development of interfaces[10]. We seek to reduce commitment by providing facilities which allow cooperative interfaces to be: Rapidly constructed at an initial low cost to developers and users. Readily changed and reconfigured at low cost and in such a manner that this provides feedback to the prototyping cycle. The aim of our work is to provide a set of facilities which promotes user involvement in the tailoring and management of virtual worlds. Our limited understanding of cooperative virtual environments also ensures that we need to allow system behaviour to be amended to respond to demands initially unanticipated by developers. 4

We also wish to provide these facilities in a manner which does not prohibit the eventual migration of existing applications with a 2D interface to cooperative settings within virtual environments. Consequently, we wish to examine the provision of interface facilities which exist outside the application and are independent of its detailed semantics. Our intent is to provide facilities which: Allow existing applications to migrate to virtual worlds with minimum disturbance; Maintain a clear separation between the behavioural characteristics of the application and the interactive aspects of the virtual world; Provide users and developers with a easy means of configuring the behaviour of applications to meet their needs; Provide a set of simple and consistent management facilities which can be applied across a range of multi-user applications. The basis of our approach is to build upon existing and established 2D models of interaction central to many existing applications. In the same way that a 2D graphical interface can be considered as a collection of interface objects we consider shared worlds as compositions of different virtual objects. Thus the existing interface to the 2D application is replaced by a set of virtual objects placed within a shared virtual world. The virtual world is defined and supported using the DIVE system[2] the Swedish Institute of Computer Science. Each of these objects has a set of properties which determine its presentation in the virtual environment (for example, colour, position, geometry). Objects may allow these properties to be manipulated by users changing their appearance within the virtual world. We have defined and constructed a selection of these objects which mimic the properties of real world objects. These include walls, windows and light sources. This set of DIVE objects is sufficient to allow the construction of simple and relatively inactive shared worlds. responsible for updating the local world model. In effect, the world is manipulated by a set of autonomous process. Each process listens for appropriate messages to respond to and alter the world by multicasting change messages. Similar message based structures are used in other VR systems[14]. We have amended DIVE to include a general command message which is multicast to clients and may be interpreted locally. Our amendment is dependant on changing the current DIVE object definition format. Currently DIVE objects contain both rendering information and behaviour descriptions. The current set of behaviours focuses on altering the visual properties of objects in the virtual world. By attaching the command message to object behaviours object interactions can have a deeper semantic effect. Applications need only parse commands and respond to them appropriately. The DIVE description for a green box which when touched logs onto a remote machine is shown in Figure 1a with the command highlighted. The presentation of the green box in the virtual world is shown in Figure 1b. object /* Green Box */ { maj_descr "DeskBox" min_descr "object16" behaviour { arc {0 -> 1 signal INT_SIGNAL type DIVE_IA_SELECT command "system xwsh -e rlogin arb&" } } nobackface on gouraud on material "green" translation v 0.000000 0.700000 0.040000 view 0 { RBOX v 0.000000 0.000000 0.000000 v 0.480000 0.100000 0.400000 } } /* Green BOX */ Figure 1a: The dive description for a green box Callbacks in a virtual world The interactive properties of worlds that can be easily build using DIVE is restricted by the lack of a general model of interaction. Developers may construct more complex interactive worlds but will be responsible for managing the deeper interaction semantics. To allow shared worlds to access deeper semantic interaction we have amended DIVE to support a general callback model similar to those supported by 2D interface facilities. DIVE allows a number of remote users to share a virtual world by using a distributed infrastructure called SID to keep replicated models of the world consistent. World changes are multicast across machines which are 5

Figure 1b: The green box in the virtual world Logically applications are constructed in a similar manner to client programs in the X Window system with interaction objects causing particular callback actions to occur. This allows a migration path for existing 2D applications to cooperative virtual settings. Our amended version of DIVE is used to manage the virtual world and support the behaviour calls defined by DIVE objects. The following sections consider the provision of facilities to define virtual worlds populated by our amended DIVE objects. We begin by introducing a 2D interface toolkit we have amended to allow virtual worlds to be easily constructed. This is followed by a consideration of some of the example worlds we have constructed and how these may be shared by cooperative users. THE MOG TOOLKIT The departure point for developing our toolkit is a 2D user interface construction environment called MOG [11]. The MOG toolkit allows a user interface to be constructed by "drawing" it using a number of predefined interface objects. A specific goal of the MOG interface builder is to be independent of a particular set of interface widgets. The toolkit manipulates a set of defined interface objects and generates appropriate code definitions for these interfaces. The MOG toolkit has been used to produce single user 2D interfaces using the Motif widget set. It has also been configured to produced cooperative user interfaces using specialised shared interface widgets. The code produced by MOG can be used immediately within the host environment. Figure 2 shows a simple 2D interface developed using the MOG toolkit and the Motif widget set. Figure 2: A simple user interface build using MOG The interface builder allows users to construct the interface for an application from a set of predefined interface objects and provides links to underlying application callbacks. The interface for the application above was created in this way, using a canvas, a text field and two buttons. In addition to generating the application interface the MOG toolkit presents a mimic of the user interface that allows the developer to directly manipulate the presentation properties of the interface. The developer can edit the properties of interface objects and see these reflected directly in the mimic interface. Properties of the interface object are presented using a simple edit window (Figure 3). 6

To allow us to manipulate virtual world objects we map presentation attributes between 2D interface objects and 3D world objects. With the exception of position and object height a direct equivalent exists in the virtual world for most 2D presentation attributes (for example, colour and label). The position and dimension attributes of any 2D object allow us to constrain the objects in two dimensions. A number of possibilities exist for fixing the third dimension: We could allow users to specify both the object height and its position in the third axis directly. To be fully effective this requires some form of user feedback. As a default we can fix the objects to a ground plan and extrude the height of the object. Feedback in this case can rely on 2D plans of the virtual world. Figure 3: Editing the properties of an interface object The interface definition generated by the MOG interface builder is linked into the facilities provided by the host interface toolkit. The callback links generated by the interface toolkit are linked into the underlying application. The resulting interface uses the facilities provided by the host environment to present itself to users. In the case of the toolkits used to date the fully functional interface presents itself in exactly the same way as the mimic interface. The generated interface acts as a means of displaying and accessing deeper application functionality. User Interface A principal consideration in determining which of these approaches to take is the manner in which MOG mimics the user interface to display alterations to interface objects. Currently the mimic interface that reflects object alterations is a facsimile of the interface eventually produced. However, this does not need to be the case and we can allow the mimic interface to appear differently from the interface eventually generated by MOG. This separation allows us to present a 3D world as a simple 2D plan in the mimic interface with alterations to the world objects reflected in this plan. Consequently, we chose the second of our two alternatives for fixing the third dimension. We have also defined a number of world objects with specific properties for use in the virtual environment. This set of world objects is extensible and we have been adding to them as we have discovered a need for a new type of object. World objects are either primitives or composites. Primitives are 3D objects designed to be directly placed in the virtual world while composites may also constrain the position of additional objects. The set of world objects defined to date include(figure 5): Application Figure 4: The interaction model supported by MOG MIGRATING TO VR-MOG The ability of the existing MOG toolkit to support the direct manipulation of interfaces as part of their development is essential for prototyping. We wish to exploit this core facility to support the development of virtual environments. Thus rather than manipulate and present 2D interface objects we wish to configure the MOG environment to manipulate virtual world objects that exist in 3D space. 7

Wall Light Poster 3D Text3D External3D Box Deskbox Floor Room Primitives A barrier in the virtual world An illumination source in the virtual world A 3D graphic image A 3D text label A more complex externally defined 3D world object. This is defined in DIVE world format but may have been initial produced using a 3D editor Composites A simple box on top of which other objects may be placed A specialised desk object on top of which objects may rest The ground plan on top of which objects rest A container for a number of other objects World Objects World Composition Figure 5: An initial set of world objects for VR-MOG In addition to these primitive and composite objects we have discovered the need for objects which effect properties of other objects. Our principal example of this to date is windows which when placed on walls allow users of the virtual world to see through then. Other world objects in this class include doors and gateways. The VR-MOG Editor Objects making up any world make use of composite objects to define the internal structure of the world. A compositional hierarchy is constructed to define the relationship between world objects. This hierarchy is directly analogous to the use of window and widget hierarchies in 2D interfaces and serves the same purpose of defining world structure. The composition hierarchy forms the basis of world construction in VR-MOG and a simple editor is used to add world objects to the composition hierarchy. The editor is shown in figure 6 displaying the compositional hierarchy used for a model of our computer lab at Lancaster. Figure 6: The composition Hierarchy for the virtual lab A plan view of the virtual world is simultaneously presented by VR-MOG. This plan view is a specialisation of the mimic window presented by MOG and allows the presentation attributes of objects to be altered by direct manipulation. Objects within the plan are open to direct manipulation and their position in 2D may be altered by dragging in the plan view. Similarly the length and width dimensional of any object may be altered by stretching it. The plan view associated with the computer laboratory model shown in figure 6 is shown below (figure 7). 8

Figure 7: The plan view of the virtual lab 9

Virtual World Users Machine Users Login Figure 9: A meeting in the virtual world Each of the objects in the plan view has an edit window associated with them that allow users to further alter their presentational properties. For example, users can set the colour of objects. Users may also use this edit window to alter the height of objects. The command associated with an object may also be altered. The edit window for a green box which allows users to directly log into a machine in the skylab is shown in figure 8. Figure 8: The edit window for the green box Once a plan view is completed users may generate a DIVE model. This world model then forms part the DIVE environment and the application may listen for the user defined commands associated with object behaviours. DIVE allows this world model to be shared by a number of users across the network and it may be presented in either immersive or desktop mode. Figure 9 shows the virtual computer laboratory being shared by three users. Users are embodied in the world using simple geometric shapes. The MOG toolkit allows us to populate this world with a variety of interactive applications. When these worlds are run in desktop mode we also can present the user with X window interfaces to applications. When running immersively these interfaces need to be mapped into the 3D world. Researchers have already developed virtual X- Servers to support this form of presentation[12]. The model of interaction supported by the MOG toolkit is very similar to that proposed by the 2D model of interaction. However, rather than present a 2D graphic interface as a means of interaction with deeper application semantics the virtual world becomes the point of contact with the application. Users interact with the underlying application through an event driven model of interaction and results are displayed in the virtual world. This arrangement is shown in figure 10(over). 10

interaction[13]. A number of researchers have explored how these may be used in virtual worlds[14]. Application Figure 10: The model of interaction supported by VR-MOG In effect what we have done in developing VR-MOG is to migrate an existing 2D model of interaction into virtual reality. This migration has the advantage that the supporting concepts are mature and well proven. However, it suffers from the disadvantages that some of the limitations of the 2D world are inherited in our model of interaction. This is perhaps most evident in the prominence of interaction as a form of collision with objects. This is directly analogous to the point and click style of interface supported by callback based models on interaction. Representing richer and more dynamic forms of interaction is somewhat more troublesome using the VR- MOG toolkit. As in the case of 2D interface builders this form of interaction still needs to be coded by hand. Alternative models of interaction based on constraint based mechanisms offer richer forms of expressing BUILDING VR WORLDS The MOG toolkit has been used to build a number of different virtual worlds to meet the demands of user. Many of these have been build in consultation with ongoing studies of work in organisations. These include the development of a facsimile of an open plan office to allow remote users to meet and share the information kept in the office. In this section we wish to consider a second cooperative virtual world. This world provides users with a virtual "trade show" for the fashion industry. Trade shows are key aspect of work in the fashion industry. A trade show is normally made up of a number of suppliers' stalls which display a range of goods and allow people to purchase from the suppliers catalogue. In addition to allowing purchasers to meet suppliers they allow people to meet others active in the industry. Most tradeshows allow users to mill around and meet others in halls full of different stalls. With the rapid growth of the Internet commercial organisations have been placing information on the world-wide wed. A number of fashion catalogues can be directly accessed through the web and orders may be placed. Our VR world allows these world wide web catalogues to be presented to users as a virtual "trade show". VR-MOG was used to construct a room full of trade stalls that users could browse (figure 11). Each trade stall consisted of a desk with the logo of the company and a web link. 11

Figure 11: A stall in the tradeshow with the appropriate web page Each of these desks has an associated command containing a URL which was passed to the underlying application. We felt that access to the world wide web was sufficiently useful that we exploited API interfaces to existing browsers to allow web commands to be interpreted as part of the general VR environment. The www command to access the world-wide web for each trade stall was entered using the edit window in VR- MOG (figure 12). Once the URLs were available the virtual trade show took only an hour or so to construct. Amending the world was low cost with the addition or removal of involving the placement of virtual objects in VR-MOG. THE DYNAMIC NATURE OF VR WORLDS The VR-MOG toolkit allows us to place interactive objects within virtual worlds. These interactive objects can invoke deeper application semantics by sending messages via a multicast distributed infrastructure. Applications listen for these and respond by multicasting further messages. These messages may result in alterations to the state of the virtual world. This arrangement allows a collection of applications to be attached to the virtual world. Applications run simultaneously and modify the virtual world by acting upon events generated in the virtual world. This arrangement is similar to the general arrangement in windowing systems. We have populated our virtual worlds with dynamic applications that modify the state of the world. One of these focuses on making available to remote users the presence of users within our virtual computer laboratory. A simple application monitors the network in the computer laboratory to detect the activities of users on machines in the lab. We detect the presence of users on machine consoles and how long has passed since they have last been active. This information is presented to users by the application placing objects representing users in front of the virtual machines displayed in the lab (figure 13). The colour of each user object indicates how long has elapsed since the user was last active on the machine. Figure 12: An edit window with the www command 12

Figure 13: Showing presence in the virtual world Figure 14: Accessing a users home page Each user object placed by the application in the virtual world is itself an interactive object capable. When users of the virtual world touch these objects the user's world wide web page is accessed and display (figure 14). The attachment of commands with object behaviours allows the interaction outlined by VR-MOG to be augmented dynamically in this manner. Similar interactive objects have been used to created query 13

interface in more abstract worlds that act as interfaces to information repositories. CONCLUSIONS This paper has presented the development of a prototyping tool which allows the rapid construction of shared cooperative worlds. The developed toolkit results from the migration of existing 2D interaction models to the virtual world. This migration has allowed interactive objects to be defined in the virtual world which access deeper application functionality. The current toolkit exploits an initial set of 3D interface objects to construct virtual worlds. Applications currently have only a loose attachment with virtual worlds. As we develop more complex worlds that need to be more closely coupled with the underlying application we envisage the design of more complex interaction objects. The outlining and construction of these objects will result from the continued prototyping of virtual worlds. Part of this development will also inform the outlining of a model of interaction for shared virtual environments. The VR-MOG toolkit allows the development of virtual worlds to take place in cooperation with users. The toolkit allows us as researchers to investigate the properties of these worlds in close cooperation with the demands of users. The initial set of worlds built to date represent our initial attempts to construct VR environments that can be shared by geographically disparate users. ACKNOWLEDGEMENTS This work was partially funded by the COMIC project (ESPRIT project no 6225). Our thanks are due to our partners in the project, and our colleagues within the department at Lancaster. Particular thanks are due to Pete Sawyer. Without his input to the initial development of MOG this work would never have been possible. REFERENCES 1. Benford S., Bowers J., Fahlen L., Rodden T., Mariani J., Supporting Cooperative Work in Virtual Environments, Computer Journal, Vol 34 no 8 Oxford University Press. 2. Carlsson, C., Hagsand, O., DIVE- A Platform for Multi-User Virtual Environments, Computer and Graphics, Vol 17 No 6 (1993), pp 663-669. 3. C. Rich et al.,demonstration of an Interactive Multimedia Environment, IEEE Computer, Vol. 27, No. 12,Dec. 1994, pp. 15-22. 4. Greenhalgh, C. Benford, S., MASSIVE: a Collaborative Virtual Environment for Teleconferencing, to appear in ACM transactions on CHI, March 1995. 5. Codella, C., et al., Interactive Simulation in a Multi- Person Virtual World, in Proceedings of ACM CHI'92 Conference on Human Factors in Computing Systems. 1992, p. 329-334. 6. Took, R., Surface Interaction: A Paradigm and Model for Separating Application and Interface, in Proceedings of ACM CHI'90 Conference on Human Factors in Computing Systems. 1990, p. 35-42. 7. Hix, D Generations of User interface management systems, IEEE software Vol 7 no 5, pp 77-87. 8. Myers, B.A., Creating Dynamic Interaction Techniques by Demonstration, in Proceedings of ACM CHI+GI'87 Conference on Human Factors in Computing Systems and Graphics Interface. 1987, p. 271-278. 9. Taylor, R.N. and N.C. Gregory F. Johnson, Separations of Concerns in the Chiron-1 User Interface Development and Management System, in Proceedings of ACM CHI93. Conference on Human Factors in Computing Systems, 1993. 10. Hartson, H.R. and E.C. Smith, Rapid Prototyping in Human-Computer Interface Development. Interacting with Computers, 1991. 3(1): p. 51-91. 11. Colebourne A., Sawyer P., Sommerville I, MOG user interface builder: a mechanism for integrating applicaton and user interface, Interacting with Computers, Vol 5 No 3, (1993), pp 315-331. 12. Dykstra, P. X11 in virtual environments: combining computer interaction methologies, available from URL: http://www.arl.mil:80/~phil/xvr/index.html 13. Hill R.D. The Abstraction-Link View paradigm:: Using constraints to connect user interfaces to applications. Proceedings of ACM CHI'92 Conference on Human Factors in Computing Systems. 1992, pp335-342. 14 Lewis, J.B., L. Koved, and D.T. Ling, Dialogue Structures for Virtual Worlds, in Proceedings of ACM CHI'91 Conference on Human Factors in Computing Systems. 1991, p. 131-136. 14