The Visorama System: a Functional Overview of a New Virtual Reality Environment

Transcription

1 The Visorama System: a Functional Overview of a New Virtual Reality Environment André Matos 1,3 Jonas Gomes 1 André Parente 2 Heloisa Siffert 2 Luiz Velho 1 1 IMPA-Instituto de Matemática Pura e Aplicada Estrada Dona Castorina 110, Rio de Janeiro, RJ, Brasil {amatos, lvelho, jonas}@visgraf.impa.br 2 Escola de Comunicação UFRJ Av. Pasteur, 250(fundos) Rio de Janeiro CEP Departamento de Informática, PUC-Rio Rua Marquês de São Vicente 225, Rio de Janeiro, RJ, Brasil Abstract The recent developments in image-based rendering have enabled a representation of virtual environments based on a simulation of panoramas, which we call virtual panoramas. Current virtual panorama systems do not provide a natural and immersive interaction with the environment. We propose a new system that uses hardware and software components to provide a natural and immersive interaction with virtual panoramas. As part of the system we propose a specific representation for the interactions in a virtual panorama. This representation can be used as a basis for the design of a high-level language for the creation of such environments. 1. Introduction Recent developments in 3D computer graphics has led to the creation of numerous virtual reality systems. These systems use a geometry description to model a virtual environment and 3D computer graphics algorithms to render the models in real-time. The users of such systems are usually allowed to navigate through the virtual environment, often with the help of additional hardware devices that provide an interface to the navigation [1]. The real-time rendering constraint, however, imposes a limit on the complexity of the modeled environment. As real world objects tend to have an exceedingly complex geometry, this limitation has a negative impact on the photo-realistic quality of such systems. The desire to overcome this problem has motivated the development of image-based rendering systems. The image-based approach combines techniques from photogrammetry, computer vision and computer graphics to represent and display virtual environments [2]. The representation is created from a number photographs, which can be taken from the real world or from a modeled environment. This representation is then used to reconstruct the views for an observer navigating through the environment. A number of image based rendering methods have been proposed which use different representations. McMillan and Bishop [2] present a framework for analyzing such methods and describe a new aproach. Other methods have recently been proposed which are based on ideas from holography [3][4]. The representation used by some of these methods provide an interaction similar to panoramas, which have been used since the nineteenth century as a type of realistic image expression. We refer to these representations as virtual panoramas. The first system to support virtual panoramas was Apple Computer s QuicktimeVR [5]. Several other systems have recently been developped with different implementations. Despite their differences, all of them have common deficiencies. Among them we should stress the fact that they do not provide a natural and immersive interaction with panoramas. This paper proposes the development of a new system, called Visorama, that combines hardware and software components to provide a natural and immersive interaction with virtual panoramas. The system also provides an authoring environment that will allow a highlevel specification of interactions common to a panorama-

2 based virtual environment. The paper begins with an introduction to panoramas, presenting their historical evolution. It then discusses virtual panoramas and the systems that currently support them. Next it presents the Visorama system, its hardware and software components. Finally, it discusses authoring in the Visorama system. The Visorama project is a collaboration between the Visgraf project of Instituto de Matemática Pura e Aplicada (IMPA) and the Grupo de Cultura e Tecnologia da Imagem of the Escola de Comunicação (ECO) of the Universidade Federal do Rio de Janeiro (UFRJ). 2. Panoramas A panorama is a type of mural painting built in a circular space around a central platform where spectators can look around in all directions and see a scene as if they were in the middle of it. It was patented by Robert Baker in 1787 and at that time was a very popular representation of landscapes and historical events. They were usually built in two or three store buildings so spectators could walk through different scenes [8]. The drawing in Figure 1 shows a section in a three store panorama building dated from By the middle of this century, several variations of panoramas had been created. An example is the Cinerama, in which cinematographic images were projected onto a circular surface covering 180 degrees. Although three different projections were used, the image appeared to be only one. Some of these variations used additional resources to enhance the user s immersibility in the fictitious world created by the panorama. In the Hale s Tour, for example, which simulated a train ride, the spectators actually sat inside a train, and the images were displayed through the windows. In another variation, the Sensorama, effects such as stereo sound and odors were used to simulate a motorcycle ride through Brooklyn, New York City. All of these variations share a number of common characteristics. The spectators are positioned inside an environment and images are projected on surfaces around them. These images display the views that would be seen by the spectators if they were in the middle of a real scene, always trying to achieve as much realism as possible to immerse the observer in this scene. The form of interaction of panoramas is very naturally accepted by spectators since it resembles the way we are used to observing the world around us, as if we were in the center of it. Perhaps this is the psychological reason why panoramas have always been so popular. Figure 1. Section of a panorama building [8]. 3. Virtual Panoramas Recent advances in image-based rendering techniques have enabled the real-time simulation of panoramas on the computer, which we call virtual panoramas. In a virtual panorama a digital image is painted onto a panorama surface S R 3 using environment mapping techniques. A virtual camera is then used to observe the surface interactively. The user is allowed to rotate the camera around its nodal point and change its field-ofview. The image to be mapped on the surface is called the panoramic image. This is illustrated in Figure 2. The panoramic image represents the projection of the environment in the panorama surface. The virtual panorama systems provide tools for the creation of these images from photographs. These could be taken from the real world or from a modeled environment. After being mapped on the panorama surface the panorama image can be interactively observed on the screen, as if the user were at the location where the pictures were taken. The process just described involves two projections: Projection of the environment onto the panorama surface. Projection of the panorama surface onto the virtual camera screen. The panorama surface should have a simple geometric shape to facilitate these two projections. The first projection is done by generating the panoramic image with specialized cameras and lens and mapping it onto the panorama surface. The most commonly used panorama surfaces are cylinders, spheres and cubes. If the cylinder is used, the panoramic image should be obtained with a panoramic camera. The image is mapped around the cylinder, an operation which defines an isometry between the cylinder and the image domain. Therefore, the image is not deformed by this transformation. If a sphere is used,

3 the panoramic image is divided into two parts which should be obtained with fisheye lens, from the same point and at opposite directions. Each image is orthogonaly mapped onto opposite hemispheres. If a cube is used, the panoramic image is divided in six parts which should be obtained with a common lens, from the same point and in six perpendicular directions. Each image is mapped onto a different face of the cube. Figure 2. A virtual camera observing a panorama. From the previous description, a virtual panorama system has two major components: an authoring environment and a viewer. The authoring environment allows the creation of the panoramic image. If this image is to represent a projection of the real world onto the panorama surface, it should be generated with cameras and lens specialized for each type of panoramic surface, as previously mentioned. These cameras and lens can also be simulated on the computer for the generation of a panoramic image representing a projection of a modeled environment. Alternatively, some systems allow the creation of these images using a number of photographs taken with regular cameras and lens. The images obtained are digitally warped, aligned with each other and combined to create a panoramic image that is very similar to one obtained using the specialized cameras and lens. The viewer performs the projection of the panorama surface onto the virtual camera interactively, allowing the user to pan, tilt and zoom with the camera. To implement this projection in real time, it can be approximated by a warping operation of the panoramic image. The result of this operation is displayed directly on the screen [2]. Systems may also provide means for playing sound files in the virtual panoramas and for composing images, animation or 3D objects with the painted panorama image. 4. Current virtual panorama systems We will compare three virtual panorama systems commercially available today: Quicktime VR 1.0 by Apple Computer, PhotoBubbles by Omniview, RealVR by RealSpace Description QuicktimeVR supports only cylindrical panoramic surfaces, and provides a complete authoring environment for the creation of virtual panoramas. The panoramic image can be a photograph obtained with a panoramic camera or generated from a sequence of overlapping photographs taken with regular cameras and lens. It currently does not allow sound, images or 3D objects to be composed with the panorama, although it has been announced that some of these features will be included in the next version of the system. PhotoBubles only allows spheres to be used as the panorama surface. The panoramic image must be taken with a 180 degrees fisheye lens, or, alternatively, it can be generated by a modeling program. The system does not provide an authoring environment so, panoramas must be generated by Omniview. RealVR, on the other hand, supports cylinders, spheres, and cubes as the panorama surface. However, it does not currently provide an authoring environment for any of these types of virtual panoramas. It only provides a converter from QuicktimeVR panoramas to its own format. This format is an extension to VRML 2.0 that allows virtual panoramas to be included in a VRML world. As a result, it is fairly simple to compose 3D objects, pictures, videos and sound with the virtual panorama. All three systems described above provide a viewer for their panoramas with a mouse interface for panning and tilting. In QuicktimeVR and RealVR, the keyboard has to be used as zoom control. None of them use multiresolution schemes for improved image quality during zooming. Finally, both QuicktimeVR and RealVR should, in the future, have an API for controlling playback from other applications 4.2. Analysis In a real panorama, the environment image is painted on a screen wall and the painted image is directly projected on the eye of the observer. As described in section 3, virtual panorama systems try to simulate this two-projection system. Nevertheless, a virtual panorama uses three projections. In fact, besides the environment projection of the panorama image, the viewing pipeline of the system uses two projections: the projection of the virtual camera which produces the image in the monitor, and the projection from the monitor to the users eyes.

4 Moreover, the viewing direction on the virtual panorama does not change as the user turns his head to look around, which is an expected feedback in a virtual environment. The indirect means of manipulating the virtual panorama through the mouse is not natural, and requires the user to associate hand movement with changing viewing directions. As a result, current virtual panorama systems do not provide immersibility of the user in the virtual environment. Because of this lack of immersibility, they cannot be considered virtual reality systems since immersibility is one of the main characteristics of these systems. Another deficiency of these systems is that they don t provide a high-level language to specify the interaction between the user and the virtual panorama. Most of them provide an interface for specifying low-level interaction tasks involving mouse movement on the viewer s window. But in a virtual panorama system, authors should be able to specify high-level interaction tasks such as a specific sequence of operations on the virtual camera. Related to this, is the inability of these systems to support the concept of a virtual environment represented by a virtual panorama. Most of them support movement of the user within a 3D environment by jumping between panoramas. But when the user is interacting with a single panorama, they don t provide any virtual environment support for working with the panorama. One exception is the RealVR system, which has all the support of VRML to create virtual environments. This language, however, requires the specification of a world with full 3D information, which is not available on virtual panoramas. It can be cumbersome to determine the 3D positions that should be assigned to objects so that they achieve a desired interaction in the virtual panorama. On the other hand, interactions on this virtual environment are more easily represented in terms of the operations that can be applied to the virtual camera, because this information is always available. 5. The Visorama System In this section we describe briefly the Visorama system, software and hardware components, and we will show how the system provides a solution to the deficiencies of current systems described in the previous section: User immersibility; View changing interface; High-level authoring language; Support for panorama-based virtual environments; As previously described, the problem of user immersibility and view changing interface occurs in the current virtual panorama systems because the viewing pipeline uses two projections: the projection of the virtual camera and the projection of the monitor screen onto the user s eyes. Moreover, these two projections are not coupled. In order to solve these problems, we need a device that is able to perform the following functions: Couple the two projections in the user s viewing pipeline. Correlate the eye projection with the head motion of the user. This device will provide the user with an abstraction of being in the middle of the virtual environment, interacting with it through an observation device. It is implemented by a viewer that simulates binoculars but, instead of displaying the real world through lenses, it displays regions of the virtual panorama through a pair of miniature CRT screens. A sketch of the device used by the Visorama system is shown in Figure 3. Figure 3. The Visorama observation device. This binocular display provides the necessary visual immersibility of the user in the virtual panorama. It is not allowed to move, but it can rotate in a pan and tilt motion. As that is done, the panorama is updated on the screen to provide the correct feedback to the rotation. This form of direct manipulation of the viewing direction while receiving the expected feedback provides a natural view changing interface for virtual panoramas. In addition, the user is not allowed to move, which is also natural for a virtual panorama. The system has software which takes user input and generates the corresponding output according to

5 interaction tasks specified by the author of the virtual environment. These interaction tasks are specified as a sequence of user actions common to virtual environments represented by panoramas, such as a sequence of operations on the virtual camera. An authoring language is being specified which uses high-level primitives based on simple specifications of interaction tasks. The language also supports the combination of primitives to create higher-level constructs. In the next sections, we describe the Visorama system in detail. In section 6 we describe the system s main hardware components. In section 7 we describe the software components. Finally, in section 8 we discuss authoring in the Visorama system. 6. The Visorama Hardware Visorama s main hardware components, their relationships and functional groupings are shown in Figure 4. They can be classified into three hardware subsystems: input subsystem, output subsystem and control subsystem. Figure 4. Hardware components. The control subsystem is basically a desktop computer with the necessary interfaces for communicating with the other hardware components in the input and output subsystems. This computer stores all information about the virtual environment. It runs software programs that use this information and user data obtained from the input subsystem to generate feedback data for the output subsystem. This real time process imposes a minimum speed constraint on the choice of processor used, since it must be able to take user input and generate appropriate output without introducing any lagging effects. It is also important that current virtual panorama systems run on this platform, since we intend to use them as part of our system. The control subsystem generates two types of data for the output subsystem: image and sound. The first type is sent to the binocular display and the second to the stereo sound equipment. As previously mentioned, the binocular display is an immersive display device that resembles common binoculars but, instead of having a set of lenses, it has, for each eye, an eyepiece and a miniature CRT screen. The images displayed by these screens appear to the user as if they were the projection of lenses in common binoculars. Each CRT screen is connected to a video output port on the computer. If two output ports are available, each screen can be connected to one, and a different image can be displayed for each eye. Although this allows stereo panoramas to be displayed, the first version of the Visorama system does not use this functionality. The stereo sound equipment is basically a pair of headphones that are connected to a stereo sound output port in the computer. Alternatively, speakers can also be used, but these have the disadvantage that the sound of the real environment could be confused with the system s output sound, resulting in a loss of auditory immersibility. By having a stereo system, different sounds can be output to each channel in order to simulate 3D sound in panoramas where sound sources are associated with a specific viewing direction. All output generated by the control subsystem is a function of the data it receives from the input subsystem and the authoring information. The input data takes two forms: viewing direction data, which is generated by a rotating head and a set of sensors, and user control data generated by a set of additional controls. The rotating head provides a direct manipulation of the viewing direction on the panorama. Potentiometers are attached to the two rotating axes of the head to capture the binocular display s movement and send it to the control subsystem. The input subsystem has a set of additional controls: two buttons and a potentiometer. The potentiometer is used to control the zooming factor. One of the buttons is used to generate discrete actions to the system, such as selecting an object on the panorama. These two controls are easily accessible by the user, since they should be heavily used. Note that these two controls and the potentiometers on the rotating head allow the execution of position, select and quantify tasks (see [7]). The remaining button is used to take the system into a control mode for specifying settings such as volume control. The values of input devices are periodically sent to the control subsystem, which must generate the correct feedback to the user as specified by the author of the virtual environment. This is achieved by the system s software components. 7. The Visorama Software

6 The system s software program can also be divided into three main functional modules: input, output and control. These three modules and their relationships are illustrated in Figure 5. The input module takes all data from the hardware devices and sends it to the other software modules. The output module takes this data from the input module and commands from the control module and generates all image and sound output. Finally, the control module examines input data and, if appropriate, sends commands to the output module. Figure 5. Modules and their relationships. 7.1 Input and Output The input module reads the data that is sent by the input hardware. This data is translated into a format that can be understood by the remaining modules. By having this translation done by the input software, the remaining software components do not have to be modified when the input hardware is changed. The resulting data is periodically sent to the other software modules. The control module reads this data at a slower rate than it is sent by the input module. As a result, some of the input data is not processed. If the button is pressed, however, new data is only sent when the old one is read, so the exact position where the button was pressed is read by the control module. The output module only reads the position and zooming data. The input module sends these values directly to the output module so it can immediately generate the rotating and zooming feedback on the virtual panorama. In this way, delays introduced by the control module processing the data do not affect the response time of the panorama regarding movement and zooming actions. Keeping this output coordinated with the binocular display s movement is fundamental to the immersibility provided by the system, since any lagging effects introduced in this process could confuse the user. The output module has two components, the image generation component and the sound generation component. The image generation component displays the virtual panorama, static images or 3D objects, which are all combined into a single output. Any virtual panorama system can be used if it has the following functionality: displays images and 3D objects on top of panoramas and has an API that can be used to control the display of the virtual panorama. This component receives commands from the control module determining which panorama, images or 3D objects are to be displayed, and a few other commands. It then loads the appropriate files from disk and displays them. The viewing direction and zooming angle are obtained directly from the input module, and are updated each time a new set of data arrives, providing the correct feedback. The sound generation component uses system resources to play audio files on the sound output hardware. It takes commands from the control module that determine which audio files should be played and the current position in the audio files, as well as common commands such as play, pause, stop and volume control. An environment like Apple Quicktime can be used as a basis for the sound output component Control: The Visorama System at work The commands generated by the control module are based on data taken from the input module and from a file which stores authoring information about the virtual environment. This information relates input sequences to their corresponding feedback, as specified previously by the author of the environment. Its internal representation is conceptually equivalent to a state diagram. Using this representation, the system is always at a known state and a number of events are specified that cause the system to transition to another state. These events are organized in sets, where each set causes a transition to a different state. Events are defined in terms of the module s parameter space. This space is the set of all combinations of pan, tilt, and zooming angles, button states and system timers. The first three parameters can be composed into a single parameter, the viewing position, so they are treated as a point in a 3D space, the viewing space, which defines a certain viewing configuration. A basic event can be defined as the current viewing position being inside a region in viewing space, the button being pressed, or a timer reaching a certain value. A composite event can be defined as a Boolean expression involving other basic and composite events. Actions can be specified to be executed while a transition is taking place. These actions are commands

7 which the control module sends to the output module. The commands that can be sent using current virtual panorama systems are: changing the current panorama; altering the current viewing parameters; playing, pausing, stopping, or jumping to a point in an audio file; showing or hiding an image or 3D object; and starting a timer. Other interesting actions should become available as virtual panorama systems provide more functionality. The control module is driven by the state diagram representing the current virtual environment. Its execution is basically a single loop where it reads input parameters and checks if any event that causes a transition occurred. When this happens, it executes the actions specified for the transition and replaces the current state by the transition s destination state. This implementation provides a simple and efficient way of generating the output corresponding to a sequence of interaction tasks in the virtual environment. 8. Authoring in the Visorama System The main problem with the approach based on state diagrams is the tedious process of creating the diagram to specify complex interactions. The approach is powerful in the sense that it can represent any interaction possible with virtual panoramas, and allows an efficient implementation. But as interactions become complicated, it is not intuitive for the author which states and transitions should be created. To illustrate this fact, Figure 6 shows a state diagram for a simple interaction specification in a virtual environment: If the user views a region R 1 for more than t 0 seconds, play an audio S 1 with duration t 1. If the user zooms into region R 2 and if S 1 has finished playing, then play another audio S 2. If it has not finished, wait for it to finish and then play S 2. If at any time the user leaves regions R 1 or R 2, stop S 1. If at any time the user leaves region R 2, stop S 2. To relieve authors from having to specify complex state diagrams, and still be able to specify the complex interaction tasks possible with this system, we are developing an authoring environment that provides a highlevel language for the specification of interaction tasks. This authoring language has basic primitives that correspond to state diagrams representing commonly used interaction tasks. These primitives are parameterized and can be used in different situations. The parameters can be regions in viewing space, button states, or timers, in which case they define the events that cause transitions inside the primitive. They can also be audio files, panoramas, images, 3D objects, or associated information, in which case they define commands that will be issued during transitions inside the primitive. The state diagram in Figure 6, for example, could represent a primitive. In this case, R 1, R 2, S 1, S 2 and t 0 would be the parameters, while t 1 and t 2 would be derived from S 1 and S 2 respectively. The language allows the combination of primitives to create higher level constructs. Initial not ( R 1or R 2) Stop S C 1 T 1 >t 1 Play S 2 not R R 2 and T 1>t Play S notr 2 Stop S R 1 Start T not R D 0 2 A B R 1and T 0 > t 0 Play S, Start T 1 1 R and not T >t Figure 6. A simple interaction specification. Using this language, authors create the virtual environment by specifying how it should respond to user input and timing events. This event-based approach is commonly used in multimedia authoring environments because of its adequacy for controlling user interactions [9]. By creating new constructs from existing ones, authors can hierarchically structure the interactions in the virtual environment. The divide-and-conquer paradigm can be applied to the authoring process so that only a small number of constructs have to be considered at a given level in the hierarchy. In addition, a hierarchically structured interaction specification is easier to reuse when simple modifications have to be done [10]. Although the authoring process in Visorama is similar to that of common multimedia presentation systems, the events generated by the users of these systems are fundamentally different. In multimedia presentation systems and in current virtual panorama systems, events are usually associated with the mouse moving into a certain area of the screen, or the user pressing the mouse button. The user has to do some action independent of viewing the content of the panorama or presentation. In Visorama, on the other hand, events might be generated by the viewing operation itself, with no further action necessary. As a result, users can seamlessly navigate through the virtual environment, not being aware that events are being generated. The specification of events associated with a viewing region allows authors to determine the output depending

8 on how much attention the user is giving to specific regions in the virtual environment. As a result, authors can specify which output the system should generate when the viewing area is restricted to a region, or object, of the virtual environment. This type of support to panoramabased virtual environments does not exist in current virtual panorama systems. We intend to develop a visual authoring environment to support the authoring language. The language constructs and means of combining them will be specified graphically though some form of diagram. In addition, areas of the panorama will be specified by navigating through the virtual panorama. All this information will be automatically converted into a state diagram representation used by Visorama s control software. 9. Conclusions We have described the development of a system that allows natural and immersive interaction with virtual panoramas. The new system includes hardware and software components and enables powerful forms of interaction that are not possible with current systems. The system has an authoring language that provides high-level primitives for the specification of interactions in the virtual environment. The Visorama system has many applications in areas such as education, tourism, real state, entertainment and many others. As the system is developed, we intend to create an application for exploring the city of Rio de Janeiro. The users will be presented with panoramas of the city from the Corcovado hill, where most of the city is visible. They will be guided to the most important parts of the city, will be able to zoom into regions of greater interest, and eventually be taken to panoramas of other parts of the city. In addition, through the use of image processing on old photographs and modeling programs, panoramas will be created that represent parts of the city in the past. All the navigation will be guided by audio narration, with explanations and suggestions about where to go. As the technology of image-based rendering evolves, we expect that this basic system architecture can be extended to provide an even greater amount of interaction capabilities with virtual panoramas. Acknowledgments: The Visorama project is sponsored by the Conselho Nacional de Pesquisa (CNPq), Fundação de Amparo a Pesquisa do Estado do Rio de Janeiro (FAPERJ), and Fundação José Bonifácio (FUJB). August [3] S. J. Gortler, R. Grzeszczuk, R. Szeliski, M. F. Cohen. The Lumigraph. SIGGRAPH 96 Proceedings, 43-54, August [4] M. Levoy, P. Hanrahan. Light Field Rendering. SIGGRAPH 96 Proceedings, 31-42, August [5] S.E. Chen. Quicktime VR - An Image Based Approach to Virtual Environment Navigation. Computer Graphics, Annual Conference Series, 29-38, [6] N. Greene. Environment Mapping and Other Applications of World Projections. IEEE Computer Graphics and Applications, 6(11):21-29, November [7] J. D. Foley, A.V. Dam, S.K. Feiner, J.F. Hughes. Computer Graphics Principles and Practice. Addison-Wesley, Reading, MA, [8] B. Comment. Le XIX e Siecle des Panoramas. Adam Piro, Paris, [9] E. Seongbae, E. S. No, H. C. Kim, H. Yoon, S.R. Maeng. Eventor: an Authoring System for Interactive Multimedia Applications. Multimedia Systems, 1994(2): [10]L. Hardman, G. V. Rossum, D. C. A. Bulterman. Structured Multimedia Authoring. Proceedings of ACM Multimedia 93, , References [1] R. S. Kalawsky, The Science of Virtual Reality. Addison- Wesley, Wokingham, England, [2] L. McMillan, G. Bishop. Plenoptic Modeling: An Image- Based Rendering System. SIGGRAPH 95 Proceedings, 39-46,