CIS Honours Minor Thesis. Research Proposal Hybrid User Interfaces in Visuo-Haptic Augmented Reality

Transcription

1 CIS Honours Minor Thesis Research Proposal Hybrid User Interfaces in Visuo-Haptic Augmented Reality Student: Degree: Supervisor: Ulrich Eck LHIS Dr. Christian Sandor

2 Abstract In 1965, Ivan Sutherland envisioned the ultimate display [Sutherland, 1965], a multimodal human-computer interface, which provides haptic feedback. Inspired by this vision, computer haptics [Srinivasan and Basdogan, 1997] has gained increasing attention in the research community during the last two decades. Some haptic environments incorporate augmented reality technology (Visuo-haptic Augmented Reality, VHAR) to create a higher degree of immersion. A VHAR system should enable users to sense and interact freely in real and virtual worlds, but available haptic devices are limited. This means, that no single device is able to provide accurate feedback for VHAR, hence a combination of devices might be desirable, similar to the Hybrid User Interface [Feiner and Shamash, 1991] in AR. Hybrid haptic systems have the potential to improve the users s haptic sensation, but this area of research has not received a lot of attention. The proposed research aims at building a hybrid haptic platform which aids further work in this field with integration, calibration, and registration tasks and to evaluate combinations of haptic devices in respect to their suitability for VHAR. Based on the results of the evaluation, a VHAR application with hybrid haptic feedback will be delivered. ii

3 Contents 1 Introduction Motivation Research Question Approach Thesis Plan Literature Review Haptics Introduction to Haptics Haptic Devices Haptic Rendering Augmented Reality Introduction to Augmented Reality Visuo-Haptic Augmented Reality Hybrid User Interfaces (HUI) Summary Method Available Hardware Evaluate Existing Software Frameworks Setup a Visuo-Haptic Environment Develop a Hybrid Haptic Platform Evaluate Hybrid Haptic Feedback in VHAR Expected Outcomes Research Limitations Future Work Timeline 14 iii

4 1 Introduction Human perception is multi-modal: the senses of touch and vision do not operate in isolation, but rather closely coupled. This observation has inspired systems that allow users to see and touch virtual objects at the same location in space: Visuo-Haptic Augmented Reality (VHAR). The basic concept of VHAR is to integrate haptic devices into an Augmented Reality (AR) environment, to provide haptic feedback and advanced interaction capabilities to the user. Haptic devices were first used in flight simulators and remote teleoperation systems and evolved to human computer interaction devices since the early 1990 s [Srinivasan and Basdogan, 1997]. Today, various haptic devices exist, but they are not used in many AR environments, as it is difficult to register and calibrate the devices within the environment. At the same time, AR has received increasing attention [Azuma, 1997]. Most recently, commercial products started to appear, because of technological advances that allow better graphical output, miniaturisation of components like Head-Mounted Displays (HMD), and improved algorithms for computer vision and tracking. But, research in AR has mainly focused on enhancing the visual sensation rather than improving user interfaces and developing new interaction techniques [Azuma et al., 2001]. Examples of commonly used interaction techniques are: hand and marker tracking, tangible interfaces, and the use of heterogenous devices to improve user interaction. Some proof of concepts exist for integrating haptic devices into AR setups; but, exact alignment, lag reduction and error minimisation are seldomly addressed [Harders et al., 2009]. Furthermore, most of the vendors of haptic devices ship their own framework for haptic rendering, which cannot easily be integrated or combined with another. Therefore, many VHAR applications work only with one type of device. The next section explains the motivation for this research in more detail (Section 1.1) and states the research question in Section 1.2. Then, the approach is given in Section 1.3 and a plan for the thesis is presented in Section Motivation In 1965, Ivan Sutherland envisioned the ultimate display [Sutherland, 1965], a multimodal computer display which provides audio-visual output and haptic feedback and interaction. This vision is similar to what is known as Holodeck 1 and has inspired research in the fields of AR and computer haptics. Current technology is far from being 1 1

5 1.2 Research Question able to create such an ultimate display, but recent research has made progress in providing haptic feedback and interaction in mixed and AR systems. Typically there is one haptic device per user, which limits the possibilities of interaction. These haptic devices itself are limited in the way they can be used, for example: kinetic feedback devices are either good for tool-like interaction or they provide haptic feedback for grasp-like tasks and tactile devices can usually produce only limited variety of tactile sensation, are complicated to build, and integrate. This means, that no single device is able to provide accurate feedback for VHAR, hence a combination of devices might be desirable. Hybrid haptic systems have the potential to improve the sensation, but this area of research has not received a lot of attention. The research will investigate possible combinations of haptic devices in an AR environment as stated in the next section. 1.2 Research Question My research question is: which combination of haptic devices is effective in a VHAR environment? Similar to hybrid user interfaces for AR [Feiner and Shamash, 1991], this research will explore a combination of multiple and possibly different haptic devices with the goal to improve the immersiveness of VHAR applications. The first hybrid VHAR application Haptic Devices n Scope of this work 2 VHAR Hybrid VHAR [Sandor et al., 2007] 1 Hybrid UI in AR [Feiner and Shamash, 1991] n Visual Devices Figure 1.1: Hybrid VHAR has two dimensions, this work wil focus on hybrid haptics. 2

6 1.3 Approach was published by Sandor and colleagues [Sandor et al., 2007]. The proposed research aims at systematically exploring the design space of hybrid haptic user interfaces in VHAR (see Figure 1.1). 1.3 Approach There exist various ways to provide kinetic feedback with haptic devices. This research will focus on two types of devices: pen-shaped devices, e.g. Sensable PHANTOM [Salisbury and Srinivasan, 1997], Novint Falcon 2 and string-based systems, e.g. SPIDAR-8 [Hasegawa et al., 2001]. Each type of device has it s specific strengths and limitations, therefore there is no perfect device. It is cumbersome to combine devices from different vendors, since they have different requirements of platforms and software frameworks and need to be calibrated and registered. Furthermore it is necessary, that there is also accurate calibration and registration between AR and the haptic systems [Harders et al., 2009]. To found the basis for research on hybrid user interfaces in VHAR a platform is needed, which helps creating hybrid-haptic AR environments and eases calibration and registration tasks (see Figure 1.2). The hybrid haptic platform will be built on top of MR-Platform [Uchiyama et al., 2002] with one HMD and a selected haptic rendering framework with multiple device support. The PHANTOM and Falcon haptic device are also available for the project, but it is 2 VHAR Application with Hybrid Haptic User Interface, e.g. Extended Haptic 3D Painting Application with bi-manual feedback Framework for Hybrid Haptic User Interfaces in Visuo-Haptic Augmented Reality Software Haptic Rendering Framework with multiple Device Support e.g. H3D Sensable OpenHaptics Devicespecific Framework Devicespecific Framework Canon MR Platform Hardware Phantom Device 1 Device... HMD... n Haptic Graphics / MR Figure 1.2: Overview of a VHAR application with hybrid userinterface. 3

7 1.4 Thesis Plan not yet known if other devices like SPIDAR or tactile actuators can be used. So, it is not yet clear, which combinations of haptic devices can be evaluated and the examples given are subject to change. 1.4 Thesis Plan The thesis will be undertaken over a period of approximately ten months. The first month will be spent by evaluating software and hardware. Then one month for specifying requirements and designing the framework is planned. The development of the hybrid haptic platform will take approximately three months, and the application development and evaluation of hybrid haptics will probably need another four months. In parallel, it is expected to work on thesis writing during the last two months. 4

8 2 Literature Review This chapter provides a review of the research on the topics of haptics (Section 2.1), AR (Section 2.2) and hybrid user interfaces (Section 2.3). 2.1 Haptics This section covers the significant work on haptics. In Section 2.1.1, the human haptic recognition is explained. Then, an overview on haptic device technology is given in Section Finally, the haptic rendering pipeline is discussed in Section Introduction to Haptics The term haptic (from the Greek haptesthai, meaning "to touch") describes something related to or based on the sense of touch, like visual for seeing. The human sense of touch works on several levels: the cutaneous, kinesthetic and haptic system. The cutaneous system consists of receptors in the skin, while the kinesthetic receptors are located in the muscles, tendons and joints. The haptic sensory system uses the cutaneous and the kinesthetic senses within an active procedure like controlled body motion. For example, the cutaneous system becomes active when the user is examining a surface, while the kinesthetic system is active when objects are modified [Otaduy and Lin, 2005]. The work of Newell et al. (2001) has shown, that the human ability to recognise objects with the haptic senses differs from the way, objects are recognised visually. They found out, that similar to the viewpoint dependence in visual recognition where objects are easier identified from the front side, the haptic system is better in identifying objects from the back [Newell et al., 2001]. In the early 1960 s, the first haptic devices where used in flight simulators and masterslave robotic teleoperation systems. These devices worked by mechanically coupling a master and a slave system. Later, an electrical servomechanism was used to decouple the systems and the basis for modern haptic devices was founded. Based on this work, researchers started to replace the slave robot with a computer system which simulated the slave. This can be seen as the beginning of computer haptics [Otaduy and Lin, 2005]. The next section discusses haptic devices for human computer interaction. 5

9 2.1 Haptics Haptic Devices Human computer interaction today happens mostly with devices known from desktop computers, like a keyboard or mouse. While these devices are good for text input and simple graphical tasks, they lack higher dimensionality and feedback. To improve the users sensation, various types of haptic devices have been created. They can be categorised into: kinetic and tactile devices. Kinetic devices are usually based on one of the two concepts: robot-like devices for tool interaction and string-based systems for grasp-like tasks. Both concepts use sensors for multi dimensional user input and actuators to apply the virtual feedback forces. In 1994, Massie and Salisbury designed the first, widely used kinetic device: the PHANTOM haptic device, a 6-DOF input device with 3-DOF haptic feedback [Salisbury and Srinivasan, 1997]. It is built like a robot-arm with a probe for the user to interact with. A well known implementation of the string-based concept is the SPIDAR device [Sato, 2002] (see Figure 2.1). (a) Fingertips attachement (b) Manipulation using SPIDAR-8 Figure 2.1: The SPIDAR haptic device [Hasegawa et al., 2001]. Tactile devices simulate the surface properties of virtual objects: Benali-Khoudia et al. (2004) give a detailed overview on existing tactile interfaces. These devices are usually built using electro-mechanic or electro-chemical actuators [Benali-Khoudja et al., 2004]. Recent research has found, that electro-active polymers are good candidates for tactile actuators [Mazzone et al., 2003]. When haptic devices are attached to a computer, they need to be controlled by a program. This process is called haptic rendering and is discussed in the next section Haptic Rendering The haptic rendering pipeline is bidirectional as shown in Figure 2.2. While in visual rendering, there is only output to the human operator, she can also provide input to 6

10 2.2 Augmented Reality Human operator Video Haptic device Visual rendering Haptic rendering Simulation engine Figure 2.2: The haptic rendering pipeline the haptic subsystem. This feedback-loop introduces strong requirements on latency and stability of the haptic rendering pipeline. The haptic rendering component consists of a renderer for force response calculation, device drivers and a physics simulation with collision detection [Salisbury et al., 2004]. It usually shares the scene graph with the visual rendering engine. To render force responses correctly, good collision detection is a crucial precondition. Lots of research has been done to improve efficiency and accuracy of collision detection [Jimenez et al., 2001], [Lin and Otaduy, 2005] and data structures [Otaduy and Lin, 2003]. When a collision is detected, the forces need to be calculated based on the penetration depth and the physical properties of the virtual object. A well known algorithm to compute feedback forces is the god-proxy object [Zilles and Salisbury, 1995] and it has been improved by simplifying the algorithms for proxy objects, force shading and friction [Ruspini et al., 1997], [Hover et al., 2009]. Weller and Zachmann (2009) are working on a unified approach for physically based simulation and haptic rendering [Weller and Zachmann, 2009]. 2.2 Augmented Reality This section covers the work on AR, why it is a useful way of displaying information and why haptic interaction and feedback will improve the users sensation Introduction to Augmented Reality AR is a technology that adds visual information to the users view of the real world. The augmentation is created from a model of a virtual world around the user, which is registered with the real world. It is overlayed in realtime and the user can interact in both worlds [Azuma, 1997]. There are various ways to add the visual information, like wearing a HMD, using projectors to add images onto real world objects or using a mobile display device which acts as a looking glas into the virtual world [Azuma et al., 2001]. 7

11 2.2 Augmented Reality There are many potential applications that could benefit from using AR technology. Typical examples include medical training and assistance, maintenance and repair, entertainment and aircraft navigation/targeting [Azuma et al., 2001]. Augmenting relevant information to the field of view usually improves the users performance or reduces errors in complex tasks. There exist several ways to interact with AR systems. Traditional input devices like mouse, joystick, touchpad or buttons can be used to select actions and to interact within the virtual environment, however these devices are usually not within the users field of view or lack 6 DOF input capability. Therefore it can be tricky to achieve a task like manipulating an object in the 3D space. Tracking of the users hand or markers can be used for tasks like selecting options or object placement. This is an improvement over the traditional input devices, because the interaction is done within the users field of view and registered correctly in the virtual world. There is also ongoing research in using tangible interfaces to integrate the interaction into the physical world [Azuma et al., 2001]. However, the mentioned interaction methods do not create any feedback to the user and do not let the user feel the virtual object she interacts with. The next section discusses methods to provide haptic feedback in AR Visuo-Haptic Augmented Reality Haptic devices are normally used in combination with some visual display, where they expand the user s sensation by providing feedback and allowing advanced interaction techniques. Srinivasan (1997) gives a detailed overview on the taxonomy, research status and challenges of haptics in virtual environments [Srinivasan and Basdogan, 1997]. Typical environments combine haptic devices with a monitor [Wei et al., 2008], [Hasegawa et al., 2001] or projectors [Richard et al., 2006]. With this approach the virtual world is displayed only in 2D and is not aligned with the real world or the haptic device. Furthermore depth perception is poor and occlusion cannot be displayed correctly. This can be improved by using AR technology, for example the user can wear a stereo HMD which allows registered display of virtual objects added to the user s view. However, using AR introduces new problems: the haptic system needs to be registered correctly within the real and the virtual world, otherwise the perceived haptic feedback will not match the user s expectation. This gets more complicated, as haptic devices are mechanically not precise enough to work out of the box and need to be calibrated [Bianchi et al., 2006], [Harders et al., 2009]. Furthermore, the haptic devices can obstruct the scene and should be hidden from scene. This can be done by using optical camouflage [Inami et al., 2000] or by virtually replacing parts of the scene with computer vision techniques [Cosco et al., 2009]. 8

12 2.3 Hybrid User Interfaces (HUI) 2.3 Hybrid User Interfaces (HUI) Hybrid user interfaces are composed from multiple input or output devices, to enhance the user s sensation or to overcome device limitations. As shown in Figure 1.1 there exist two dimensions for hybrid user interfaces in VHAR. Multiple display devices form a HUI for AR while multiple haptic devices form a HUI for haptics. When multiple displays are combined with multiple haptic devices, they could form the ultimate HUI for VHAR. Feiner et al. (1997) combined a low resolution HMD with a high resolution flat panel screen to enlarge the overall space usable for organising windows on a desktop, while allowing the user to interact with the focused content on the high resolution display. The desktop is placed on a virtual sphere around the users head. Their system adapts well to the way people usually organise pieces of papers on a desk [Feiner and Shamash, 1991]. To improve the haptic feedback, Richard et al. (2006) combined a SPIDAR haptic device with vibro-tactile gloves in a virtual environment, to provide tactile cues for the operator [Richard et al., 2006]. Passive haptic devices can also be used to enhance the haptic sensation as shown in an experiment, where the researchers from Postech, Korea used a haptic device to modulate the virtual stiffness of a real object [Jeon and Choi, 2009]. 2.4 Summary As VHAR is still in it s infancy, there has not been much research in this area. The main focus until now has been on exploring possibilities of haptic feedback in AR and solving problems like registration and calibration. As shown by Feiner (1991), hybrid user interfaces can enhance the user s sensation by eliminating device limitations. Inspired by this work, this research will evaluate possible combinations of haptic devices and their advantages. Furthermore, a platform for hybrid haptics in VHAR will be implemented, to aid future research on this topic. 9

13 3 Method This chapter describes the method to answer the research question. First the available hardware is discussed (Section 3.1). Second, the existing software frameworks for haptic rendering need to be evaluated (Section 3.2). Third, a visuo-haptic environment needs to be set up (Section 3.3) and a platform for hybrid haptic feedback created (Section 3.4). Then, possible combinations of haptic devices will be evaluated and an application created, to demonstrate the hybrid haptic user interface in VHAR (Section 3.5). Finally, the research limitations will be stated (Section 3.7) and the future work will be sketched in Section Available Hardware In order to implement hybrid haptic user interfaces for VHAR, multiple haptic devices are needed. The Magicvision Lab at the University of South Australia will provide a PHANTOM [Salisbury and Srinivasan, 1997] and two Novint Falcon 1 devices for the project. Furthermore it is expected, that a SPIDAR-10 device [Sato, 2002] will be available end of August The Lab is also equipped with a Canon ST-HMD and a license for the MR-Platform framework [Uchiyama et al., 2002], which will be used to create the augmentations. Due to the uncertainty whether the SPIDAR-10 will be available in time, several options to create hybrid haptic feedback are proposed, but the work will focus on creating a single application which delivers the greatest improvement for the user s sensation with the available devices. To register the virtual and real world with the AR and haptic system, some kind of tracking is necessary. At this time, there are no external tracking systems available for the project. There are two options to overcome this problem: using fiducial markers which is supported by MR-Platform [Kato and Billinghurst, 1999] and marker-less tracking as proposed in PTAM [Klein and Murray, 2007]. 3.2 Evaluate Existing Software Frameworks There exist several frameworks for haptic rendering. To evaluate these frameworks, the requirements need to be specified, similar to the common criterions for evaluating haptic rendering algorithms [Ruffaldi et al., 2006]. Existing Frameworks for haptic rendering

14 3.3 Setup a Visuo-Haptic Environment include: Chai3D 2, H3D 3, OpenHaptics 4 and Springhead 5. Some of these frameworks are closely coupled to the vendors hardware, some support devices from multiple vendors, others build on vendor specific frameworks to add support for multiple devices. A framework with multiple device support which is easily extensible and well designed needs to be chosen as basis for the hybrid haptic platform. 3.3 Setup a Visuo-Haptic Environment To aid the initial development of the hybrid haptic platform and as basis for the evaluation of hybrid haptics in VHAR, a proper setup of a visuo-haptic environment needs to be done first. This includes: preparing a desktop space with fiducial points, installing all required softwares with their dependencies and attach multiple haptic devices to the computer system. 3.4 Develop a Hybrid Haptic Platform Once the visuo-haptic environment is set up and the haptic rendering framework to build on is selected, the hybrid haptic platform can be developed. First, the requirements need to be specified. Second, an overall system design needs to be created and a prototype will be implemented. Based on the insights of the prototype, the final platform will be developed, including unit tests and documentation. This platform will integrate MR-Platform with a haptic rendering framework with support for multiple devices. It will ease the registration and calibration of newly added devices and support future development of hybrid user interfaces in VHAR. 3.5 Evaluate Hybrid Haptic Feedback in VHAR Based on the platform for hybrid haptics in VHAR, several combinations of haptic devices will be evaluated. Depending on the availability of the hardware, some or all of the proposed hybrid interfaces can be tested: Create an enhanced 3D painting application using a SPIDAR-10 and a PHAN- TOM similar to the VHAR painting application [Sandor et al., 2007] (see Figure 3.1)

15 3.6 Expected Outcomes Provide tactile cues for haptic interaction using a PHANTOM with vibro-tactile actuator attached to the probe A combination of PHANTOM and Novint Falcon to provide bi-manual haptic feedback (a) Outside view (b) User s view Figure 3.1: VHAR with passive and active haptics. A user paints with a virtual brush on a virtual teacup [Sandor et al., 2007]. 3.6 Expected Outcomes The research will answer the research questions that were specified in section 1.2. The research will determine which combination of haptic devices will form a good candidate for a hybrid user interface in VHAR and provide an application for further investigation. It is expected, that combining multiple haptic devices will enhance the users sensation by providing bi-manual haptic feedback. To support future research on hybrid haptics in VHAR a platform will be developed, which eases the tasks of integrating, registering and calibrating haptic devices into a VHAR environment. 3.7 Research Limitations As was mentioned in section 3.1, it is not yet clear which haptic devices will be available for this project, so the stated examples for evaluating hybrid haptics in section 3.5 are subject to change. Furthermore there is no precise, external tracking available for the project, which will limit the quality of registration and the possible applications. It is not intended to create high quality graphics output, but simple virtual worlds with sufficient detail to test hybrid haptics in VHAR. Finally, only one display device will 12

16 3.8 Future Work be used for developing the platform and the application, but the resulting framework should be extensible to support multiple displays as well. 3.8 Future Work As more haptic devices will be available at the lab, they can be integrated into the platform and new combinations can be evaluated. Furthermore it is expected to create a user study to verify the improvement of accomplishing tasks with hybrid haptic feedback based on the delivered application. Other components for tracking, sensor input or better graphics should be integrated into the platform later. 13

17 4 Timeline Figure 4.1: Timeline Minor Thesis 14

18 Bibliography R Azuma. A survey of augmented reality. Presence: Teleoperators and Virtual Environments, 6(4): , , 7 R Azuma, Y Baillot, R Behringer, S Feiner, S Julier, and B MacIntyre. Recent advances in augmented reality. IEEE Computer Graphics and Applications, 21(6):34 47, , 7, 8 M Benali-Khoudja, M Hafez, and J Alexandre. Tactile interfaces: a state-of-the-art survey. In Proceedings of the 35th International Symposium on Robotics, pages , Paris, France, G Bianchi, B Knoerlein, G Szekely, and M Harders. High precision augmented reality haptics. In Proceedings of the EuroHaptics 2006 Conference, pages 1 9, Paris, France, F Cosco, C Garre, F Bruno, M Muzzupappa, and M Otaduy. Augmented touch without visual obtrusion. In Proceedings of the 8th IEEE International Symposium on Mixed and Augmented Reality, pages , Orlando, FL, USA, S Feiner and A Shamash. Hybrid user interfaces: breeding virtually bigger interfaces for physically smaller computers. In Proceedings of the 4th ACM Symposium on User Interface Software and Technology, Hilton Head, South Carolina, USA, ii, 2, 9 M Harders, G Bianchi, B Knoerlein, and G Szekely. Calibration, registration, and synchronization for high precision augmented reality haptics. IEEE Transactions on Visualization and Computer Graphics, 15(1): , , 3, 8 S Hasegawa, Y Koike, and M Sato. Evaluation of two-handed multi-finger haptic device spidar-8. In Proceedings of the 11th VRSJ International Conference on Artificial Reality and Teleexistence, Tokyo, Japan, , 6, 8 R Hover, M Di Luca, G Szekely, and M Harders. Computationally efficient techniques for data-driven haptic rendering. In Proceedings of the IEEE Third Joint EuroHaptics conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, pages 39 44, Salt Lake City, UT, USA, M Inami, N Kawakami, D Sekiguchi, Y Yanagida, T Maeda, and S Tachi. Visuo-haptic display using head-mounted projector. In Proceedings of the IEEE Annual International Symposium on Virtual Reality, pages , New Brunswick, NJ, USA,

19 Bibliography S Jeon and S Choi. Haptic augmented reality: Modulation of real object stiffness. In Proceedings of the EuroHaptics 2009 Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, pages , Salt Lake City, UT, USA, P Jimenez, F Thomas, and C Torras. 3d collision detection: a survey. Computers and Graphics, 25: , H Kato and M Billinghurst. Marker tracking and hmd calibration for a video-based augmented reality conferencing system. In Proceedings of the 2nd IEEE and ACM International Workshop on Augmented Reality, pages 85 94, G Klein and D Murray. Parallel tracking and mapping for small ar workspaces. In Proceedings of the 6th IEEE and ACM International Symposium on Mixed and Augmented Reality, pages , M Lin and M Otaduy. Sensation-preserving haptic rendering. IEEE Computer Graphics and Applications, 25(4):8 11, A Mazzone, R Zhang, and A Kunz. Novel actuators for haptic displays based on electroactive polymers. In Proceedings of the 10th ACM Symposium on Virtual Reality Software and Technology, pages , Osaka, Japan, F Newell, M Ernst, B Tjan, and H Bülthoff. Viewpoint dependence in visual and haptic object recognition. Psychological Science, 12(1):37 42, M Otaduy and M Lin. Clods: Dual hierarchies for multiresolution collision detection. In Proceedings of the Eurographics Symposium on Geometry Processing, pages , Dublin, Ireland, M Otaduy and M Lin. Introduction to haptic rendering. In Proceedings of the 32nd ACM Annual Conference on Computer Graphics and Interactive Techniques, Los Angeles, CA, USA, P Richard, D Chamaret, and F Inglese. Human-scale virtual environment for product design: Effect of sensory substitution. The International Journal of Virtual Reality, 5 (2):37 44, , 9 E Ruffaldi, D Morris, and T Edmunds. Standardized evaluation of haptic rendering systems. In Proceedings of the 11th IEEE Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, Los Angeles, CA, USA, D Ruspini, K Kolarov, and O Khatib. The haptic display of complex graphical environments. In Proceedings of the 24th ACM Conference on Computer Graphics and Interactive Techniques, pages , Los Angeles, CA, USA, K Salisbury and M Srinivasan. Phantom-based haptic interaction with virtual objects. IEEE Computer Graphics and Applications, 17(5):6 10, , 6, 10 16

20 Bibliography K Salisbury, F Conti, and F Barbagli. Haptic rendering: introductory concepts. IEEE Computer Graphics and Applications, 24(2):24 32, C Sandor, T Kuroki, S Uchiyama, and H Yamamoto. Exploring visuo-haptic mixed reality. Technical Report of IEICE, PRMU 106(470):31 36, , 11, 12 M Sato. Development of string-based force display: Spidar. In Proceedings of the 8th VSMM International Conference on Virtual Systems and Multimedia, Gyeongju, Korea, , 10 M Srinivasan and C Basdogan. Haptics in virtual environments: Taxonomy, research status, and challenges. Computers and Graphics, 21(4): , ii, 1, 8 I Sutherland. The ultimate display. In Proceedings of the IFIP Congress, page , New York, USA, ii, 1 S Uchiyama, K Takemoto, and K Satoh. Mr platform: A basic body on which mixed reality applications are built. In Proceedings of the 1st IEEE and ACM International Symposium on Mixed and Augmented Reality, pages , , 10 L Wei, A Sourin, and O Sourina. Function-based visualization and haptic rendering in shared virtual spaces. The Visual Computer, 24(10): , R Weller and G Zachmann. A unified approach for physically-based simulations and haptic rendering. In Proceedings of the 36th ACM Conference on Computer Graphics and Interactive Techniques, New Orleans, Lousiana, USA, C Zilles and K Salisbury. A constraint-based god-object method for haptic display. In Proceedings of the IEEE International Conference on Intelligent Robots and Systems: human robot interaction and cooperative robots, volume 3, pages , Pittsburgh, Pennsylvania, USA,