Immersive Multi-Projector Display on Hybrid Screens with Human-Scale Haptic Interface

Transcription

1 888 IEICE TRANS. INF. & SYST., VOL.E88 D, NO.5 MAY 2005 PAPER Special Section on Cyberworlds Immersive Multi-Projector Display on Hybrid Screens with Human-Scale Haptic Interface Seungzoo JEONG a), Nonmember, Naoki HASHIMOTO, and Makoto SATO, Members SUMMARY Many immersive displays developed in previous researches are strongly influenced by the design concept of the CAVE, which is the origin of the immersive displays. In the view of human-scale interactive system for virtual environment (VE), the existing immersive systems are not enough to use the potential of a human sense further extent. The displays require more complicated structure for flexible extension, and are more restrictive to user s movement. Therefore we propose a novel multiprojector display for immersive VE with haptic interface for more flexible and dynamic interaction. The display part of our system named D-vision has a hybrid curved screen which consist of compound prototype with flat and curve screen. This renders images seamlessly in real time, and generates high-quality stereovision by PC cluster and two-pass technology. Furthermore a human-scale string-based haptic device will integrate with the D-vision for more interactive and immersive VE. In this paper, we show an overview of the D-vision and technologies used for the human-scale haptic interface. key words: immersive projection technology, human-scale virtual environment, multi-modal interaction, force feedback 1. Introduction Recently, many immersive displays have been developed for virtual reality, education, industry, entertainment, etc. These displays originated from the CAVE [3] which surrounds users with large flat screens. Despite a simple design of the CAVE, it could realize highly immersive virtual environments effectively. In other studies, they proposed systems extending the number of its screens, and created the cubic space completely surrounded with six large screens [21]. This may be an ultimate structure of CAVE-like displays because it leads to an overwhelming feeling of presence without leak of outer world. However, when users interact with VE, not only visual quality, but also natural modes of interaction and control, and perception of self-movement affect presence or immersions [5]. From this point of view, adding force feedback to an immersive VE is the most direct and natural approach which can increase user involvement. Some studies about presence experiments [14], [16] have been demonstrated that haptic feedback improves task performance and increases presence in VE. These studies showed that the users felt as if they were present to a higher degree in the virtual environment when they received haptic information. Though CAVE-like displays have been tried with the Manuscript received September 13, Manuscript revised December 1, The authors are with Precision and Intelligence Laboratory, Tokyo Institute of Technology, Yokohama-shi, Japan. a) jeongzoo@hi.pi.titech.ac.jp DOI: /ietisy/e88 d integration of an arm force feedback device, these solutions have several drawbacks. The arm can interfere in the interaction with virtual objects or hide parts of the visualization space. Furthermore, this only works within a limited space [4], [20]. For human-scale interaction between real- and virtual worlds, it is very important for us that the user can interact with virtual objects at hand or close to his hands for getting high-degree presence and realism. It means that a display system should allow the user to step into screen areas so as to gain proper control and lessen the gap between his perceived distance and physical length. Above all, for our desirable interaction system, the display and human interfaces should have flexible configuration and high visualization. In this paper, we propose a novel multi-projector display with a string-based haptic interface, for more interactive and immersive VE. We have developed a multiprojection display system named D-vision [6]. The D of D-vision originates in Divide, and Duplex-vision. The system adopts a multi-projector technology, which uses 24 PCs and projectors to provide high-resolution and wide view-angle images. Images are projected on a hybrid screen, which consists of a flat central screen and a curved peripheral screen. Adoption of flat screen on the front screen can secure the brightness and resolution per unit area. The flat surface distortion free can create high-quality stereoscopic vision. This design concept establishes effective installation spaces, and are generated more streamlined images in the hybrid screen through our rendering method. To generate force feedback effects, the D-vision combines with a human-scale haptic interface named SPIDAR- H. The haptic interface consists of some motors and flexible strings instead of electromagnetic force sensor. Because of its simplicity, it can be implemented into the D-vision flexibly. It also allows full body activity to the user, and a practicable manipulation space. Our aim is to increase the sense of togetherness through touch by allowing physical contact. Our proposed system will invoke the feelings of physical and social presence unobtainable with only hearing and seeing. Hence, when a VE provides life-size interaction and immersive virtual space with user, human metaphors will enlarge their potentials and the interfaces will promote their capability further extend. 2. Immersive Multi-Projector: D-vision In this section, we will describe how to implement our Copyright c 2005 The Institute of Electronics, Information and Communication Engineers

2 JEONG et al.: IMMERSIVE AND INTERACTIVE SYSTEM 889 Fig. 1 An overview of the multi-projector display D-vision. Projectors and a PC cluster for graphics rendering are placed around a hybrid screen. Fig. 2 A structure of the hybrid curved screen. Its central area is flat, and its peripheral area is spherical containing partial surfaces of such objects as flats, spheres, cylinders and tori. system, D-vision for immersive and interactive multiprojection system in detail. An overview of the D-vision isshowninfig Hybrid Curved Screen In many ancient CAVE-like immersive displays, a cubic screen has mostly been the dominant solution. The cubic type with 6 flat screens which completely surround users into that inside, said to an ultimate structure. It requires more complicated structure for flexible extension, and makesit more difficult to integrate othermechanical and large-scale interfaces. Other immersive displays with a spherical or arched screen have also been developed [7], [19]. They reduced distorted images and were for multi-user at the same time. These spherical display systems used only front projection to reduce any extra-spaces. However, the systems suffer from a fundamental problem that a user s shadow will surely appear on a displayed image [12]. Due to this reason, the users cannot enter freely inside a screen. When you apply rear-projection for front screens to avoid shadows, the existing displays involve crucial problems due to massive space and high cost of hardware configuration. For instance spherical screen causes ill and dark areas in the steep curved surface. In order to diminish the dark areas, the screen should subdivide into smaller parts, and be projected by using a large number of projectors according to the number of parts. This is an impractical method for its cost. Our screen design attempts to use limited space efficiently by reducing installation space. Moreover it was considered so that the system provides a virtual experience with human-scale bodily input and use force feedback at ease. Firstly, it is composed of a hybrid curved screen, which adopts flat screens for central view and curved screens for peripheral view and it is based on the structure of human s eyes. In the human s eyes, central view is used to perceive the outer world more precisely with high-resolution input, and peripheral view is used to detect movements of objects in the outer world with low resolution, but wide view angle input. Therefore, a flat fresnel-lenticular stereoscopic screen is used in the central area of the hybrid screen for high quality image projection. And, in the peripheral area, a simple curved screen made with fiberglass reinforced plastic (FRP) is used to realize a wide view angle. The curved screen is composed of some simple primitive shapes as shown in Fig. 2. The size of the hybrid screen amounts to 6.3 m (width) 4.0 m (height) 1.5 m (depth). Because large curved surfaces are difficult to build accurately, each of the screen elements in Fig. 2 is based on a simple shape and its size is relatively small, for example a radius of the spherical part is 1.5 m. These curved surfaces are connected smoothly so as not to distort projected images on the jointed part of the screens. Even if the hybrid screen covers only the users front half, no lack of efficiency of immersion can be detected. The users view direction can be controlled and kept as a central view by using a locomotion interface [11]. As shown in Fig. 1, the central flat part of the screen is for rear projection with 8 projectors with SXGA, pixels resolution. The remaining part of the screen is for front projection with 16 projectors with XGA, pixels resolution. Brightness of each projector is about 3,000 ANSI lumens. These projectors are installed at the positions indicated in Fig. 3. The projectors for the central screen, shown in Fig. 3 (a), are arranged in common style with lens-shift function to realize stereoscopic image projection. Other projectors, shown in Fig. 3 (b), adopt an original arrangement to minimize installation spaces and avoid casting a shadow on the screen. The whole size in Table 1 indicates the volume of installation spaces including screens, projectors and PCs [9], [21]. However, despite of less spacious volume, the ratio of the D-vision s screens is much higher than that of other displays. Therefore, we can confirm that the D-vision s design concept realizes an efficient configuration in limited spaces.

3 890 IEICE TRANS. INF. & SYST., VOL.E88 D, NO.5 MAY 2005 Fig. 3 Projector arrays. (a) is projectors behind the screen, and (b) is behind users. The arrangement of these projectors is original to reduce installation spaces and casting shadows. Table 1 Size of exist immersive displays. system (screen) whole (m) screen (m) ratio D-vision (4) CAVE (1) CABIN (6) COSMOS (17) size is (W) (D) (H) 2.2 Real-Time Image Correction The D-vision has a parallel rendering strategy to generate distributed and high-resolution images in real-time. The rendering processes of the D-vision are distributed to each rendering PC of the PC cluster, and work according to the role of each PC based on target areas on the whole screen. In this system, whole images of the scenes are divided into 16 areas for the distributed rendering in each PC that is connected to each projector. The 8 areas which include the central view and up and down side of that view are rendered by 16 PCs for the stereoscopic viewing using polarized glasses. As a result, the total of 24 PCs are used for image generation of the D-vision. In the compound type with flat and curve screen, however, each color map of the projectors is not balanced, and has no uniformity in the link area of two screens. It also makes render more distorted images on the change of user s viewpoint. Even if the front screen occupies a mostly flat area with free-distortion, geometry and color corrections technologies of rendered images is still needed for perceiving and generating a seamless virtual world. In our rendering method, those correction tables are measured with a digital camera and calculated accurately in advance [22]. The tables are referred in two ways as shown in Fig. 4 (a). One is that original image processing hardware is used for correcting in real-time. This method is stable and reliable with such special hardware. The other is that a programmable GPU on a PC s video card is used. Today s GPU has an enough power to process images in real-time. Each correction table is stored into a video memory as textures as shown in Fig. 4 (b), and referred in extra rendering passes to perform the image corrections. Figure 4 (b) illustrates examples of brightness correction maps as gray-scale texture images. Figures 4 (c) and 4 (d) show the result of the image corrections. Fig. 4 An image correction for the curved screen. Before the correction, images projected with various projectors overlap with each other in a state of disorder. After the correction, those images are seamlessly blended, and a virtual world is realized in the D-vision. This correction method has less computation cost and lower latency because of the high-speed image processing hardware or powerful GPUs. 3. Human-Scale Haptic Interface In this section, less invasive, more flexible human-scale haptic device, is described i.e. SPIDAR-H. And we discuss how to integrate the haptic interface sole developed with the human-scale immersive VEs. 3.1 SPIDAR-H System Adding haptic information to immersive virtual environments is especially effective for interacting and enhancing presence in those environments. The recent mainstream of haptic interfaces for the virtual reality adopts a mechanical link structure or wire-driven system. In the immersive VE, however, the invasive mechanical links prevent users from seeing surrounding images seamlessly, only work within a limited space, even if that structure is stable for highly accurate and scalable operation. On the other side, a wearable force display system for immersive projection displays [8] has been developed. The mechanism is based on the same wire-driven approach as our haptic system, i.e. SPIDAR. Its compact size and

4 JEONG et al.: IMMERSIVE AND INTERACTIVE SYSTEM 891 Fig. 5 A human-scale haptic interface. portable functions make the user walk around freely within large workspace thanks to wearable features. It has limited haptic spaces, and its structure requires larger frame for getting both-handed interaction. Its extended frames can also interfere with the user s vision. Although several type of improved haptic devices have been developed, they were still not suitable for the use in immersive virtual environments because of their size and mechanism [18]. This was a constraint or even a difficulty for integrating force feedback devices with immersive VE. The problems can be solved by an alternative interface to provide force feedback for direct manipulation within a sufficiently large space. The solution is to use less invasive and more flexible string-based haptic devices. Therefore, we adopted a human-scale interface named SPIDAR-H which is based on string-based haptic device for desktop operation [10]. Like the previous human-scale haptic system [1], the SPIDAR-H allows force feedback on one distinct point, like the middle finger, with 3 translations. As four motors are necessary for each point, 8 motors are installed for both hands. They are positioned on non-adjacent vertices of a cubic frame in the origin structure (See Fig. 5). This is a simpler configuration, for both hands force feedback and displaying haptic in every direction. The haptic rendering is implemented by the spring and damper model. Force feedback occurs when collisions or impacts happen. The simulation frame rate can vary from 1 khz to 10 khz. The haptic and visualization rendering will be synchronized stably by high speed I/O and high definition haptic controller. 3.2 Integrating SPIDAR-H into D-Vision A configuration of the haptic device may be a little laborious for satisfying both human-scale force feedback and highly immersive display. The string-based haptic devices have flexible properties, but on the other hand they are capable of having a little sensitive precision at scaleable size. The larger the SPIDAR workspace is, the less accurate the manipulations will be. However, the choice of interface de- Fig. 6 A snapshot of whole system which integrated the SPIDAR-H interface into D-vision. Each user s hand is connected to 4 strings and motors for force feedback. String-based structure is quite safe, and is less restrictive at user s full body movements. pends on whether interaction work is task performance- or human intuitive sense-oriented. If we consider the interface to focus at the potential of human s physical activities, its bandwidth can be extended at an acceptable level. If the manipulation space is +1.1m 3, the absolute maximum errors are ±1.5 cm. Despite larger space like double size, it will be within 2 3 cm range. When we performed an experiment [2] to trace both a real cube and a virtual cube with about 1 khz of control frequency, data of virtual objects was at most about 10 mm in the error difference from tracing data of real object s rim in the calibrated well working space. The results showed that interaction adding to haptic information can help the user to transfer physical information like real world, and user can interact with virtual objects accurately even if our system is a larger scale interface. We implemented the SPIDAR-H without fixed cubic frame into D-vision so as to offer a flexible hardware configuration and high visualization. Each hand manipulation space size is extended approximately 3 m 3 to visualize spaces of D-vision fully. In this system, a total of 8 motors for both hands are placed surrounding the user. Circles in top of Fig. 6 illustrate the position of the motors on the D-vision. Four motors placed in the front side of the users are fixed behind screens, and the strings are tensed through a small hole on the curved peripheral screen. The other motors are placed behind the users by using a frame for projectors. The SPIDAR-H has two rings put on the user s finger on each hand, and each ring has four strings by which motor torque is transmitted. The motors have a rotary encoder, which counts the length of the string from the ring to the motor. By using the length of the four strings, the position of the ring is calculated,

5 892 IEICE TRANS. INF. & SYST., VOL.E88 D, NO.5 MAY 2005 and the force displayed to the users is controlled as they interact with the virtual object by their own hand directly. The strings never prevent the users from immersing into virtual worlds with surrounding images. And flexibility of that strings enables the users to perform various motions freely. 4. Applications The D-vision is a novel immersive projection display, which has multi-modal interfaces as described in above sections. Like other Immersive VE, users can control using familiar input devices like game pad or joystick, and interact with wearing general interfaces such as electromagnetic tracking sensor, marks, and polarized light glass. In special users can walk around large-scale workspace for their manipulation, and use force interaction with virtual objects through their own hands. This shows that our system is a well-suited system, which gives full demonstration for direct and dynamical interactions. In this section, we will illustrate the potentials and capabilities of our system through some examples of the applications. At first, Figure 7 (a) illustrates a psychological experiment for the analysis of human s vision by using a large screen surrounding the user s field of view [17]. If the screen is extended to the user s foot, the sensation of self-motion can be effectively obtained. The feature of the hybrid-curved screen contributes to simulate proper immersive VE for human factor analysis. When an image with specific pattern moves forward and backward, the participants invokes a sense of vection, like a swing back and forth in Fig. 7 (a). At that time, by using a force plate including pressure sensors, the sway of the participant s body is measured and analyzed. Secondly, as described above, haptic feedback will be used in any aspect of virtual reality where more information can be conveyed by touch than by sight and sound alone. The typical haptic interfaces can be used to manipulate virtual objects with force-response. Figure 7 (c) shows an example of molecule visualization [15]. We can intuitively change the position of stereoscopically displayed molecules with the SPIDAR-H, and feel intermolecular force. By engaging life-like virtual human into VE, interface technologies can enrich and improve a human interaction. We can realize the force reaction with a virtual human that generate self-motions [13] (See Fig. 7 (d)). The movement of this reactive virtual human is generated by real human motion data, which is pre-captured by traditional magnetic motion capture system, and stored into our designed database. Using the SPIDAR-H device of our proposed system, the user can hold and throw the virtual ball like a real one. The haptic information originating from the user becomes key factors when generating the virtual human s reactions. In the D-vision, virtual human animate in accordance with time and physical space by user, and at the same time, transmit an appropriate force to user. As the proposed system give visual and force feedback to the user s body in an immediate fashion, it can accomplish more interactive and dynamic application as mentioned above. 5. Conclusions In this paper, we proposed a novel immersive multi- Fig. 7 Application on the D-vision.

6 JEONG et al.: IMMERSIVE AND INTERACTIVE SYSTEM 893 projector display with human-scale haptic interface. The immersive system with hybrid curved screen, rendered seamless images in real time, and generated high-quality stereoscopic images by PC cluster and two-pass technology. Furthermore a string-based haptic device is integrated with the D-vision. Because of its flexible configuration and high visualization, it enables increasingly believable and dynamic physical interaction with virtual objects or humans. Through developments of various applications at present, we confirmed the potential of extending the frontier of interaction and haptic interfaces. In future works, we will develop software which enables users to create applications easily and efficiently for the D-vision, and also try to improve immersive VE system with accuracy and richness. References [1] Y. Cai, M. Ishii, and M. Sato, A human interface device for CAVEsize virtual workspace, IEEE International Conference on Systems, Man, and Cybernetics, vol.3, no.3, pp , Oct [2] W. Choi, S. Jeong, N. Hashimoto, S. Hasegawa, Y. Koike, and M. Sato, A development and evaluation of reactive motion capture system with haptic feedback, Proc. 6th IEEE International Conference on Automatic Face and Gesture Recognition, pp , [3] C. Cruz-Neira, D.J. Sandin, and T.A. DeFanti, Surround-screen projection-based virtual reality: The design and implementation of the CAVE, Proc. SIGGRAPH 93, pp , [4] A. Fischer and J.M. Vance, PHANToM haptic device implemented in a projection screen virtual environment, Proc. workshop on Virtual environments 2003, pp , [5] B.G. Witmer and M.J. Singer, Measuring presence in virtual environments: A presence questionaire, Presence, vol.7, no.3, pp , [6] N. Hashimoto, J. Ryu, M. Yamasaki, T. Minakawa, H. Takeda, S. Hasegawa, and M. Sato, D-vision: Immersive multi-projection display with a curved hybrid screen, Proc. 8th International Conference on Virtual System and MultiMedia (VSMM2002), pp , [7] W. Hashimoto and H. Iwata, Ensphered vision: Spherical immersive display using convex mirror, Transactions of the Virtual Reality Society of Japan, vol.4, no.3, pp , [8] M. Hirose, K. Hirota, T. Ogi, H. Yano, N. Kakehi, M. Saito, and M. Nakashige, HapticGEAR: The development of a wearable force display system for immersive projection displays, Proc. Virtual Reality 2001 Conference (VR 01), pp , [9] M. Hirose, T. Ogi, S. Ishiwata, and T. Yamada, Development and evaluation of immersive multiscreen display: CABIN, Systems and Computers in Japan, Scripta Technica, vol.30, no.1, pp.13 22, [10] M. Ishii and M. Sato, A 3D spatial interface device using tensed strings, Presence, vol.3, no.1, pp.81 86, [11] M. Iwashita, A. Toyama, N. Hashimoto, S. Hasegawa, and M. Sato, Development of locomotion interface based on step-in-place movement, IEICE Trans. Fundamentals (Japanese Edition), vol.j87-a, no.1, pp.87 95, Jan [12] C. Jaynes, S. Webb, R.M. Steele, M. Brown, and W.B. Seales, Dynamic shadow removal from front projection displays, Proc. ITCVG 01, pp , [13] S. Jeong, W. Choi, N. Hashimoto, S. Hasegawa, and M. Sato, A design of reactive virtual human with force interaction, IEICE Technical Report, HIP , [14] E. lotta Sallnas, K. Rassmus-Grohn, and C. Sjostrom, Supporting presence in collaborative environments by haptic force feedback, Proc. ACM Tran. Computer-human interaction, pp , [15] J. Murayama, L. Bougrila, Y. Luo, K. Akahane, S. Hasegawa, B. Hirsbrunner, and M. Sato, SPIDAR G&G: A two-handed haptic interface for bimanual VR interaction, Proc. EuroHaptics 2004, pp , [16] B. Petzold, M.F. Zaeh, B. Faerber, B. Demi, H. Egermeier, J. Schilp, and S. Clarke, A study on visual, auditory, and haptic feedback for assembly tasks, Presence, vol.13, no.1, pp.16 21, [17] J. Ryu, N. Hashimoto, and M. Sato, Analysis of vection using body sway in immersive virtual environment, Correspondences on Human Interface, vol.5, no.3, pp , [18] K. Salisbury, F. Conti, and F. Barbagli, Haptic rendering: Introductory concepts, IEEE Comput. Graph. Appl., vol.24, no.2, pp.25 32, [19] N. Shibano, T. Hatanaka, H. Nakanishi, H. Hoshino, R. Nagahama, K. Sawada, and J. Nomura, Development of VR presentation system with spherical screen for urban environment human media, Transactions of the Virtual Reality Society of Japan, vol.4, no.3, pp , [20] N. Tarrin, S. Coquillart, S. Hasegawa, L. Bouguila, and M. Sato, The stringed haptic workbench: A new haptic workbench solution, Proc. Eurographics, FP15-2, Sept [21] T. Yamada, M. Hirose, and Y. Iida, Development of complete immersive display: COSMOS, Proc. 4th International Conference on Virtual Systems and Multimedia (VSMM 98), pp , [22] M.Yamasaki,T.Minakawa,H.Takeda,S.Hasegawa,andM.Sato, Technology for seamless multi-projection onto a hybrid screen composed of differently shaped surface elements, Proc. 7th Annual Immersive Projection Technology Symposium (IPT), VR , March Seungzoo Jeong received a master s degree in the department of computational Intelligence and Systems Science, the Graduate School of Science and Engineering, Tokyo Institute of Technology in Currently she is on the doctor s course in the precision and intelligence laboratory, Tokyo Institute of Technology. She is interested in human-scale interaction, virtual human and human interface. Naoki Hashimoto received B.S., M.S. and Dr. Eng. degrees from Tokyo Institute of Technology in 1997, 1999 and He has been a research assistant at Precision and Intelligence Laboratory, Tokyo Institute of Technology since His research interests include immersive projection technology, human-scale virtual environment, computer graphics and its applications. He is a member of VRSJ. Makoto Sato graduated in 1973 from Department of Physical Electronics, Faculty of Engineering, Tokyo Institute of Technology, where he obtained the Doctor of Engineering in He became an assistant in the same faculty, and is now a Professor of Precision and Intelligence Laboratory, Tokyo Institute of Technology. He is engaged in researches on pattern recognition and virtual reality.