HAPTEX. System Requirements and Architectural Design

Transcription

1 Project no.: IST-6549 HAPTEX HAPtic sensing of virtual TEXtiles Instrument type: Thematic Priority: STREP Priority 2: Information Society Technologies Deliverable D1.1 System Requirements and Architectural Design Due date of deliverable: July 15, 2005 Actual submission date: July 11, 2005 Deliverable resubmitted on: January 14, 2006 Start date of project: 01/12/04 Duration: 36 months Organisation name of lead contractor for this deliverable: MIRALab - University of Geneva Nature of deliverable: Report Dissemination Level: Public Project co-funded by the European Commission within the Sixth Framework Programme ( ) HAPTEX Project Consortium Revision: 2.2 The HAPTEX Project Consortium groups the following Organizations: ORGANIZATION SHORTNAME ROLE STATE MIRALab - University of Geneva UNIGE Coordinator Switzerland University of Exeter UNEXE Partner United Kingdom Scuola Superiore Sant Anna PERCRO Partner Italy University of Hanover UHAN Partner Germany Tampere University of Technology SWL Partner Finland

2 - This page has been intentionally left blank - 2

3 Table of Contents 1. INTRODUCTION USER REQUIREMENTS ANALYSIS Motivation State-of-the-art in textile evaluation Fabric hand Projects and Research Overview Definition of the final objectives st Scenario: Force Feedback nd Scenario: Tactile Feedback rd Scenario: Fabric Identification REQUIREMENTS ANALYSIS Hardware requirements Overview of the expected hardware requirements Software requirements Real-time animation Features Coding conventions Developing platform Document versioning system Documentation system FABRIC SELECTION AND MEASUREMENTS The Fabric Selection Process Fabric Selection Process Selected Fabrics Creation of the fabric properties database The Process of measuring fabrics: Description of KES-F Description of digitized output ARCHITECTURAL DESIGN Related work Haptics Virtual Reality Engine Grab device API Eligibility Ongoing work on Multimodal interaction Suitability and need of a dedicated solution Haptex System Overview Overall View of the System VTV - Virtual Textile Viewer Use cases Data Structure

4 5.3.3 VTV Architecture Large-Scale Model The Haptic Renderer Overview of the Haptic Renderer Small-Scale Model Expected data flow with 3D visualisation and hardware interface Fingertip model for haptic force computations The Tactile Drive System Background Proposed design Tactile renderer The Force-Feedback Device Input/output of the Force Feedback Device The Grab Device Data Exchange Overview Visualisation and Haptic Rendering (MIRALab-UHAN) Tactile Display and Tactile Rendering (UNEXE-UHAN) Force-Feedback Device and Haptic Rendering (PERCRO-UHAN) REFERENCES

5 1. Introduction HAPTEX ("HAPtic sensing of virtual TEXtiles") is a research on multimodal perception of textiles in virtual environments. The goal of this project is to integrate a visual representation of virtual textiles with a haptic/tactile interface, thus allowing the user to have the sensation of feeling the virtual garment. Both the visual simulation and the haptic rendering are based on the actual physical properties of the simulated textile, taken from specific measurements on real textiles. This makes it possible for the user to identify different kind of fabrics. The achievement of the final goal of creating full haptic interaction with real-time animated textiles will facilitate the future realisation of such multimodal frameworks, featuring haptic interaction with any kind of virtual objects. This document describes the overall architecture of the HAPTEX multimodal system. First, we present the user requirements analysis undertaken for the HAPTEX System. We define the long-term goal of the HAPTEX project and present our motivating ideas. Then we focus on the state-of-the-art in textile evaluation. Here we describe the common methods for evaluating fabrics and we give an overview of projects and research institutes linked to this field. We conclude the second chapter defining our final objectives, giving a description of our expectations regarding the final demonstrations. In the third chapter, we describe the expected hardware and software requirements as well as the coding conventions to be respected during the implementation phase. The fourth chapter deals with what we consider the input of the HAPTEX multimodal system: the selection of fabrics and the related measurements. Finally, in the fifth chapter we discuss the resulting global architecture. First, we describe existing platforms for multimodal VR systems and we analyse their eligibility in respect to our requirements. This last chapter ends with a detailed description of the main components of the HAPTEX system and an overview of their cross relationships. 5

6 2. User requirements analysis In this chapter we provide an overview of the long-term goal of the HAPTEX project; we present our motivating ideas, the need for our work and the state-of-the-art in the area of textile evaluation. We conclude the chapter defining our final objectives, giving a description of our expectations regarding the final demonstrations. 2.1 Motivation HAPTEX -HAPtic sensing of virtual TEXtiles- is a research on multimodal perception of textiles in virtual environments. The goal of this project is to integrate a visual representation of virtual textiles with a haptic/tactile interface, thus allowing the user not only to have the sensation of feeling the virtual garment, but also to identify different kinds of fabrics. Our research is of a long-term nature and involves particularly high risks, as there are still many issues to be solved, such as the encoding of tactile perception of fabrics, the integration of force feedback with multi-channel tactile feedback, and the synchronization of complete haptic feedback with physical-based simulation of fabrics featuring real-time animation. However, these risks are compensated by the expected advancement of multimodal interaction tools, techniques and know-how resulting out of this project. The ideas proposed for HAPTEX have so far not been realized by other research groups. Previous research either addressed these fields separately; or the combination of vibratory stimulation with force feedback has been limited to stimulators with very few channels [Murray et al., 2003]; or the visual feedback did not take into account physicalbased simulation featuring real-time animation [Govindaraj et al., 2003] [Huang, 2002]. The focus of HAPTEX lies on the integration of haptic interaction (both force feedback and tactile perception) with real-time animation of virtual textiles. We expect HAPTEX to provide a system which allows a person to perceive, touch and manipulate virtual textiles. However, the results of the HAPTEX project will not be limited to the application in the field of haptic interaction with virtual textiles. They will provide advancements which will facilitate the future realisation of multimodal frameworks featuring haptic interaction with any kind of deformable virtual objects. 2.2 State-of-the-art in textile evaluation Both for the physical-based simulation of textiles and for the haptic rendering it is essential to gain detailed information not only about the physical properties of textiles, but also about the way people touch textiles. The textile industry has been working on the analysis of fabric evaluation methods since decades. Early studies on this topic date back to 1930 [Pierce, 1930], and many other followed in the fifties and sixties [Abbot, 1951; Cooper, 1960;Dahlberg, 1961; Lindberg et al., 1961; Grosberg et al., 1966]. In the seventies, a specific committee, organised by Professor Kawabata, was established in Japan to develop an evaluation system for the handling of fabrics [Kawabata, 1980]. The resulting KES-F system is still one of the main standards in this field. 6

7 In the field of haptics, there have been previous attempts to combine these studies on physical properties of fabrics with garment simulation [Govindaraj et al., 2003] [Huang, 2002]. However, these attempts did not take into account real-time animation. In the following, we first describe the common methods for evaluating fabrics; afterwards, we give an overview of projects and research institutes linked to this field Fabric hand Fabric hand is a generic term for the tactile sensations associated with fabrics that influence consumer preferences. It is basically a reflection of overall quality, consisting of a number of individual physical properties, and is the human response to touching, squeezing, rubbing, or otherwise handling a fabric. [Hui et al., 2004] In general, fabric hand can be evaluated by subjective or objective methods Subjective methods Subjective assessments treat fabric hand as a psychological reaction obtained by the sense of touch. It is a primary descriptive method based on the experience and sensitivity of human beings. [Hui et al., 2004] It is the traditional method of assessing fabric handle. Professional handle experts sort out the fabric qualities by touching, bending and stretching each fabric to classify them according to their sensory descriptions. The main drawbacks are [Hui et al., 2004; Dent, 2000]: Assessment process very time consuming and costly Different fabric sensory perceptions of individuals Different background and experiences of individuals Problems of communication between experts and consumers (for instance the same adjective can be used with a different meaning by different individuals) In the apparel industry there is still the need to define a standard, effective and efficient method of fabric hand assessment [Dent, 2000] Objective methods Objective assessments attempt to predict fabric hand using instrumental data and sensory-instrumental relationship [Behery, 2005]. Several methods have been developed, among others: Kawabata FAST Hand testers (Ring Test, Slot Test) Such testing methods are still costly, cumbersome and time consuming. Some new methods for fabric hand assessment are being developed [Strazdiene et al., 2005] and some researchers are conducting comparative analysis between objective and subjective evaluation [Grineviciute et al., 2005] Projects and Research Overview Haptic Simulation of Fabric Hand NTC Project: S00-PH08 (formerly I00-P08) 7

8 Partners: Philadelphia University, California State University, University of Pennsylvania, Rutgers University, The State University of New Jersey Leader: Prof. Muthu Govindaray (School of Textiles and Materials Technology, Philadelphia University) Research Goal: to develop a virtual fabric handling experience using a haptic display. Some commercial systems are already on-line where the drape of a garment can be visualized, but someone purchasing has no sense of the hand of the fabric. The work aims to simulate fabric hand so that a consumer would not only see a virtual garment drape but also would be able to feel the fabric [Govindaraj et al., 2003]. First step: On the basis of the Kawabata KES-F system they have generated a 3-D profile of the fabric surface profile. Second step: They have developed a new device, the Philau Haptic Device, as a combination of force feedback and tactile display. The device consists of a feeler pad at the end of an articulated arm. The magnetic brakes get their input voltage proportional to the surface-friction of the fabric, while the tactors pins follow the contour. Work in progress: o Subjective evaluation of the device; o Improvement of the device to include the feeling of compressional compliance of the fabric; o Designing of an opposing thumb configuration to feel the fabric between index and thumb; Web Site Address: National Textile Center (NTC) Partners: The National Textile Center (NTC) is a research consortium of eight universities: Auburn University (Consumer Affairs, Engineering), University of California at Davis, Clemson University, Cornell University, Georgia Institute of Technology, University of Massachusetts at Dartmouth, North Carolina State University and Philadelphia University. Their mission is to enhance the knowledge base for the continuing viability of the U.S. Fiber/Textile/ Fiber Products/Retail complex. Web Site Address: Texas Tech University s International Textile Center (ITC) Dr. Seshadri S. Ramkumar, Ph. D., Research Scientist Dr. Ramkumar has developed an artificial polymeric finger, a device which not only simulates the way fabric is felt with the human hand, but also assigns a standard measurement to each fabric. The system was created in hopes of providing a faster and more accurate method of evaluating aesthetics than human touch or other established industry methods [Ramkumar, 2002; Ramkumar et al., 2003a; Ramkumar et al., 2003b]. Web Site Address: American Association of Textile Chemists and Colorists (AATCC) 8

9 The AATCC is the world's largest technical and scientific society devoted to the advancement of textile chemistry. In 1990, the AATCC Committee RA89 developed Guidelines for the Subjective Evaluation of Fabric Hand. This procedure describes guidelines for the presentation of fabrics for the evaluation of hand. Its purpose is to standardize the conditions under which a fabric is evaluated for one or more of the constituent elements of hand. Such guidelines may be used in the following circumstances: When different people at different times wish to follow the same protocol for examining fabrics. In training evaluators to detect and distinguish among different constituent elements or components of hand. When an individual wishes to duplicate the conditions under which a fabric has been previously evaluated. With a panel of individuals evaluating the same fabric(s). Web Site Address: Kaunas University of Technology (Kaunas, Lithuania) (Faculty of design and Technologies) Among the main research areas, they are also working on the evaluation of fabric hand. They developed a new method for the evaluation of fabric hand [Strazdiene et al., 2005], basing on comparative analysis between subjective hand evaluation (results obtained by judges survey) and objective one (parameters determined experimentally by using the methods of mathematical statistics) [Grineviciute et al., 2005]. Liverpool John Moores University (Liverpool, U.K.) They are working on a project whose aim is to develop an intuitive visual and haptic communication system using the standards and expectations of the textile and related industries with particular regard to professional aesthetic and psychological perspectives, and the working methods of the industry [Dillon et al., 2001]. 2.3 Definition of the final objectives The final demonstrations will present the features of the HAPTEX multimodal system. We propose three scenarios, featuring different chances of success. The scenarios have been designed for the purpose of experiencing the features of the system and present them to final user. The first two scenarios have a different focus, and serve to experiment with different properties of the final system. The third scenario integrates the first two, and will be adapted to suit the requirements and take advantage of the previous experiences with the first two scenarios. The numbering order of the scenarios reflects the order in which they will be realised st Scenario: Force Feedback The first scenario is depicted by Figure 1. Its focus lies on the evaluation of the fabric s real-time animation and on the forcefeedback interaction. 9

10 Placement of textile The fabric is hanging in virtual space; its upper border is fixed. Apart from the upper border, the rest of the fabric is animatable and deformable Possible actions The user is able to draw up his fingers to the fabric, pinch and rub it using two fingers and stretch/pull it, the upper part remaining fixed Limitations This scenario is used to experiment with the fabric s deformations and the haptic interaction with the haptic interface hardware limited to force-feedback. The tactile feedback is not integrated in this scenario. Figure 1 Force feedback scenario Rate of success For this scenario we foresee a high rate of success nd Scenario: Tactile Feedback The focus of this scenario lies on the tactile feedback of the haptic device. Figure 2 shows this scenario Placement of textile The fabric is placed on a rigid support. Its boundaries are fixed. Figure 2 Tactile feedback scenario Possible actions The user can press his finger on the fabric and rub it, feeling its surface and seeing the resulting deformation of the textile Limitations Fabric deformation and animation is very limited due to the fact that the fabric s borders are fixed. Also, force-feedback is not available. The focus of this scenario lies in the tactile evaluation of the fabric Rate of success This scenario is a bigger challenge than the first one, because of the complexities of presenting textural information through a tactile display rd Scenario: Fabric Identification The third scenario will optimise the first two, giving a demonstration of the overall system and integrating real-time fabric deformation with complete haptic interaction. This means that this scenario will feature both force-feedback and tactile-feedback simultaneously. The goal of this scenario is to provide a demonstrator which will allow the user to sense a haptic feedback of virtual textiles according to their visual representation and identify different kinds of fabrics. The evaluated fabrics will be chosen from the HAPTEX fabric database (see 4.1.3). 10

11 Rate of success The complete integration of the first two scenarios represents a very challenging task. 11

12 3. Requirements analysis This chapter presents the requirements analysis undertaken for the HAPTEX System. It describes the expected hardware and software requirements as well as the coding conventions to be respected during the implementation phase and agreed by the HAPTEX Consortium. The users of the final HAPTEX multimodal system will be confronted with a visual representation of an animatable piece of fabric in combination with its haptic perception. The system will be consisting of the haptic interface connected to a high-end PC, which will display on its screen a viewer providing the visual feedback for the haptic interaction. 3.1 Hardware requirements The hardware requirements of the whole haptic interface will be described in a separate document, provided with deliverable D4.1. The haptic interface will be connected to a high-end PC, which will comply with state-of-the-art technology in terms of graphics adapter, CPU speed and RAM Overview of the expected hardware requirements The configuration of our final demonstrator will have high-end hardware requirements. In particular, we will require the latest technology in the following domains: Processor : multi-processor High-end graphics card with 3D accelerator RAM 3.2 Software requirements The monitor connected to the HAPTEX PC will display the software implemented in the context of the HAPTEX project. The implemented software will consist of a viewer, which will provide the visual feedback for the haptic interaction, showing both the simulated textile and the finger model. The viewer will be the visible part of the HAPTEX software integrating several components such as visual rendering, physical-based simulation, haptic rendering and the control of the tactile device and the force-feedback device. The developed application will be Windows-based and optimized for the latest release of this operating system (XP/Longhorn) Real-time animation The animation refresh rate is expected to reach 30 fps. The application will simulate a given textile of about 20cm x 20cm, represented by a textured, low resolution polygonal mesh Features Next to basic interaction methods which will allow to change the viewing camera position (rotation, translation, zoom), as well as rendering options (point, wire frame or solid mode, force indication), the user will be able to select the simulated textile from a given database of virtual textiles. 12

13 The HAPTEX software will have the aspect of a typical WinXP application. 3.3 Coding conventions Coding conventions are programming guidelines used to standardize the structure and coding style of an application, and are one of the most valuable tools for producing consistent code. This facilitates the collaboration between partners, making the code easier to read, understand and maintain. Coding conventions focus on the physical structure and appearance of the program, including naming conventions for objects, variables, and procedures; standardized formats for labelling and commenting code; guidelines for spacing, formatting, and indenting. For the HAPTEX coding conventions please refer to Annex the HAPTEX website Developing platform Microsoft Development Environment 2003 with Visual C++.NET v Document versioning system A versioning system, which allows distribution of a team of developers over a largescale network as well as parallel asynchronous collaboration, will be used to coordinate and facilitate the development of the HAPTEX software modules. A versioning system facilitates the display of differences between several subsequent versions of a document (time-based versioning), and the tracking of modifications between several different variants of the same version (author-based versioning). Furthermore, by allowing automatic merging, it makes it possible to build a single final state of the document as the result of several parallel contributions, making the collaboration converge to a single state. The versioning system used within the HAPTEX project will be "Subversion", a centralised, open-source version control system Documentation system Doxygen is a cross-platform documentation system for C/C++ and several other programming languages. It can generate an on-line documentation browser (in HTML) and/or an off-line reference manual in HtmlHelp (CHM), RTF, PostScript, PDF, HTML and Unix man page formats. The documentation is extracted directly from the sources, thus forcing it to be consistent with the source code. Doxygen is freely distributed under the terms of the GNU General Public License SubVersion has been developed by CollabNet ( 13

14 4. Fabric selection and measurements This chapter deals with the creation of the fabric properties database and the related measurements of fabric samples. We describe the process which leads to the creation of the input of the HAPTEX multimodal system. First, we describe what kinds of measurements have been chosen and how these will be performed. Then, we define the system s input parameters: the fabric samples, where we first focus on the fabric selection process. Afterwards, we list the chosen fabrics to be evaluated; we describe how the measurements take place and what kind of parameters we can obtain as a result. The second part of this chapter deals with the configuration of the final demonstrator and presents the choice of the visual and haptic evaluations to be performed with it. In order to reach our final goal of evoking the sensation of touching textiles through our multimodal haptic perception system, we selected a range of very different fabrics in terms of material (fibre), binding and physical properties such as thickness or weight. The class of sample materials chosen for evaluating the HAPTEX system offers prototype problems that are typical when implementing realistic haptic/tactile perception in the general context. Moreover, the selection is made of a wide range of textiles with a rich variety of structures and delicate palpable, periodic patterns of different resolution scales. The choice is the result of a fabric selection process. Samples of these fabrics will be subject to KES-F measurements, out of which a database describing physical properties of textiles will be created. 4.1 The Fabric Selection Process In order to reach our final goal of evoking the sensation of touching textiles through our multimodal haptic perception system, we selected a range of very different fabrics in terms of fibre material, fabric structure and physical properties such as yarn/loop density, thickness and mass per unit area. The class of sample materials chosen for evaluating the HAPTEX system offers prototype problems that are typical when implementing realistic haptic/tactile perception in the general context. Moreover, the selection is made of a wide range of textiles with a rich variety of structures and delicate palpable, periodic patterns of different resolution scales. The choice is the result of a fabric selection process. Samples of these fabrics will be subject to KES-F measurements, out of which a database describing physical properties of textiles will be created Fabric Selection Process For selecting the different kind of fabrics to be contained in the fabric properties database, we identified three criteria, depicted in Figure The first criterion is the fibre material of the fabrics. We can distinct between natural and man-made fibres. Natural fibres contain plant fibres (e.g. cotton or linen) and animal fibres (e.g. wool or silk). Man-made fibres are either based on natural polymers (e.g. viscose, cupro or acetate) or synthetic polymers (e.g. polyester, elastane or polyacryl). 2. The second criterion is the fabric structure, i.e. the type of weaving or knitting. There exist different types of weaves in woven structures. Plain weave, twill and 14

15 satin/sateen are the basic weaves, from which derivative weaves are developed. It is also possible to combine different weaves in one fabric. Knitted fabrics are divided into two groups, weft knitted and warp knitted. Basic structures in weft knitted fabrics are plain jersey (single jersey), rib knit, purl fabric and interlock. Derived weft knitted structures are e.g. tuck knits, held stitch knits, fleecy fabrics and terry fabrics. Most common warp knitted structures, e.g. charmeuse, full tricot or velour structure, are made from two systems of warp yarns, i.e. two systems of stitches. Velvets, mesh and terry fabrics are also produced by warp knitting. Nonwoven fabrics differ from knitted or woven fabrics, because they are not based on yarns. They are based on webs of individual fibres, which can be bonded to each other by several means.. 3. The third criterion of selection will be based on the physical aspect and dimension of the fabrics, like thickness, density and weight. Figure 3 - The 1st fabric selection process To ensure a realistic simulation of textiles, it is important to study the characteristics of a broad range of fabrics. Therefore, each fabric group is represented in the fabric database. One third of the samples are made of 100% natural fibres, one third of 100% man-made fibres and one third of the samples are blended fabrics. All the main fabric 15

16 types are represented and also one leather sample was chosen. Fabric samples have different structures, densities, thicknesses and weights. The described criteria for the selection of fabrics represent as most variety as possible in terms of physical parameters and expected haptic characteristics Selected Fabrics Fabrics of natural fibres Description Fibre content Structure Weight g/m2 1. Denim 100% CO twill Shirt cotton 100% CO combined 120 twill 3. Cord 100% CO velveteen Linen 100% LI plain weave Gabardine 100% WO twill Crepe 100% WO plain weave Silk 100% SE plain weave Natural silk (bourette) 100% SE plain weave Wild silk (dupion) 100% SE plain weave Jute 100% JU plain weave Blended fabrics Description Fibre content Structure Weight g/m2 11. Flannel 80% WO twill % PES 12. Denim 62% PES twill % CO 3% EL 13. Plaid 35% PES twill % AF, AO 30% WO 14. Tweed 66% AF, AO combined % WO 10% PES 10% CMD twill 15. Velvet 92% CO 8% CMD velvet Lurex knit 70% PES held stitch % PA knit 17. Crepe-jersey 85% PES 15% EL rib knit Woven motorcyclist wear fabric, coated 72% PA 28% PU plain weave Woven easy care 65% PES twill 180 fabric 20. Warp knitted velour fabric 35% CO 90% PA 10% EL warp velour knit

17 21. Weft knitted plain fabric 98% CLY 2% EL single jersey Fabrics of man-made fibres Description Fibre content Structure Weight g/m2 22. Taffeta 100% CA plain weave Crepe 100% PES plain weave Satin 100% PES satin Felt 100% PES nonwoven Organza 100% PES plain weave Fleece 100% PES weft knit Woven upholstery 100% PES woven Jacquard Woven outdoor 100% PES plain weave 90 leisure wear fabric 30. Tulle 100% PA warp knitted Warp knitted tricotsatin tulle 100% PA warp knitted tricot-satin Leathers Description Material Structure Weight g/m2 32. leather 100% Leather Creation of the fabric properties database The Fabric Properties Database will be the result of a double process of selection, measurement and evaluation of fabrics. The first phase will start with the selection of 30 fabric samples, chosen after the aforementioned criteria and representing a very broad range of fabrics. Afterwards, several fabric measurements will take place, and the samples will be thoroughly evaluated after these measurements. The evaluation phase will consist of measurement analysis and experimental simulation activity. The aim of the evaluation will be the identification of those fabrics giving the most satisfying results in terms of believability and simulation behaviour. From these suitable fabrics, other 20 samples will be chosen, in order to be first measured and then evaluated. The collected information will be presented in the fabric properties database, which will serve as the source of textile selection for the HAPTEX software. 17

18 Figure 4 - Creation of the Fabric Properties Database The Process of measuring fabrics: Description of KES-F The KES-F system (Kawabata s hand evaluation system for fabrics) was developed in Japan by the Hand Evaluation and Standardization Committee (HESC) organized by Professor Kawabata in 1972 [Kawabata, 1980]. In this fabric objective measurement method, scientific principles are applied to the instrumental measurement and interpretation of fabric low stress mechanical and surface properties such as fabric extension, shear, bending, compression, surface friction and roughness. The fabric handle is calculated from measurements of these properties. According to the standard type of measurements some characteristic values (Table 1) are calculated from recorded curves obtained by each tester both warp and weft direction. Tensile properties (force-strain curve) and shear properties (force-angle curve) are measured by same machine. Bending properties (torque-angle curve) are measured bending first reverse sides against each other and after that the face sides against each other. Pressure-thickness curves are obtained by compression tester. The measurements of surface friction (friction coefficient variation curve) and surface roughness (thickness variation curve) are made with the same apparatus using different detectors. Tensile Shearing Bending LT WT RT G 2HG 2HG5 B Linearity of load-extension curve Tensile energy Tensile resilience Shear rigidity Hysteresis of shear force at 0.5º shear angle Hysteresis of shear force at 5º shear angle Bending rigidity 18

19 Compression Surface 2HB LC WC RC MIU MMD SMD Hysteresis of bending moment Linearity of pressure-thickness curve Compressional energy Compressional resilience Coefficient of friction Mean deviation of MIU, frictional roughness Geometrical roughness Table 1: Characteristic values in KES-F system Description of digitized output Standard Kawabata measuring equipement provides analog output signals that have to be digitized in order to be stored in a electronic database. The principle of digitizing the output of Kawabata measurements is that similar curves as in analogue form can be plotted using the measured values. The output parameters are the same as in analogue form in the x- and y-axis. The sampling rates in various measurements can be fixed differently and according to requirements. E.g. in bending the sample rate is 20 Hz and in surface profiles and friction 1 khz. The program calculates the characteristic values in KES-F system (Table 1) printing them on the report page. A more detailed description of the Kawabata and other physical fabric measurements will be given in report D

20 5. Architectural design This chapter presents the architectural design of the HAPTEX system, firstly giving a general overview of the system, and then describing each single component more into detail. Since multi-sensory channel simulation is still quite new, there is no well established method for creating a multi-sensory environment within an application. The development of the HAPTEX software modules will consist of a continuous optimization process of component integration, which aims at avoiding and solving common problems regarding synchronization, latency and efficiency. Some of these problems may have been encountered by previous researchers involved in similar projects. To avoid unnecessary development and narrow down the choices to be taken for the design of the HAPTEX architecture, we will undertake a review of the state-of-the-art in haptics-related projects and software platforms. We will first review the existing work, then compare the resulting systems and finally evaluate to which extent it is suitable to build the HAPTEX System on top of an existing framework for multimodal applications. 5.1 Related work The reproduction of the human senses by a computer is of great interest for the huge amount of applications it could provide. Approaches to simulate multisensorial experiences are especially taken by multimodal Virtual Reality (VR) applications. Due to the rather young nature of this technology, however, we will see that only a very limited amount of research has been done with similar challenging objectives as HAPTEX. For this reason, more than by the specific features of the presented platforms, we will be interested in their architectures and their underlying concepts. To focus on the purpose of HAPTEX, the main components of a haptic platform are first highlighted in the following figure. Figure 5 Haptic platform components Three main blocks are in the center of such a platform The Simulation module computes the Virtual Environment (VE) under the law of the virtual world. The Haptic Renderer module computes the force that are sent to the device from the VE and the present and past device position. The Graphic Renderer module displays the VE. The previous modules make use of two more blocks: The Device driver, which is able to read and send data to the device (for a 6 degree-of-freedom device, it is able to send a force and a torque). The collision detection module, which is able to detect collisions. In the context of the HAPTEX project, the Simulation module has to compute the behavior of a physically based, deformable textile, displayed by the Graphic Renderer. At the same time, the Haptic Renderer computes the data for the force and tactile feedback. The visuo-haptic simulation has to be perfectly synchronized, allowing for 20

21 realistic interaction and manipulation. Numerous challenges arise from the strict realtime requirements of this highly physically based simulation of a deformable object. In the following we will explore what would be the solutions offered by an available platform. We begin with the tools for haptic rendering, describing different open-source and commercial APIs and toolkits. Then, we have a closer look at existing frameworks proposed in the literature. Afterwards, we highlight different Virtual Reality engines offering integrated haptics solutions. Finally, we analyze the eligibility of these alternatives and the standards being currently raised in multimodal applications. This section ends with a conclusion summarizing the choices resulting from our analysis Haptics Haptics Toolkits The most popular haptic device nowadays is probably the Phantom Device, constructed by Sensable Technologies [sensable]. The first toolkits reviewed are uniquely dedicated to this device. To develop applications with the Phantom series, Sensable has proposed two toolkits: Ghost and OpenHaptic. Both toolkits enable developers to program the haptic rendering obtained through the Phantom by specifying the geometry and the physical properties of the materials. The graphic rendering is done using opengl. Both toolkits offer to handle two loops a haptic one loop called servo loop running at 1 khz and a graphic one running at 30 Hz. Ghost uses a scene graph structure to manage the Virtual World thus making it efficient to manage large scenes. It is possible to load a static scene by reading data from a VRML file. An optional library GhostGL can be used for the graphic rendering. The scene graph is passed to GhostGL that uses opengl for the rendering. OpenHaptic comes with two APIs: a low level one (HDAPI) and a high level one (HLAPI) building on top of the other. With HLAPI, the geometry of the object is specified to the haptic renderer with calls patterned after opengl, thus facilitating the integration of haptics into opengl based application. It uses three loops, the graphic and the servo loop (similar to Ghost) and a collision loop that runs at 100 Hz and which is responsible for detecting the collision between the haptic end cursor and the virtual object. HDAPI allows to communicate with the haptic device by reading positions and sending forces and torques. It also takes care of threading issues and safety cut-offs such as maximum force exceeded temperatur. The low level API allows the developer to specify all the particularity of the force feedback. It offers facilitates the definition of threads through a specific scheduler. The sources are not publicly available. At the lowest level comes the Phantom Device Drivers (PDD), which allows uniquely to set the data (position / force) of the device. SenseGraphics, author of H3D [h3d], uses the open standards X3D and opengl to build a scene graph-based API which uses the OpenHaptic toolkit to add the management facility of a scene graph application. Although being open source, it is build on the top of HLAPI, it only supports the Sensable haptic devices and all the sources are not available. Boeing propose the Voxmap PointShell Software that specifies the haptic rendering using voxel [MacNeely et al., 1999]. It is organized into three modules: a voxelisation engine that also computes basic collision detection a swept volume module that extracted the exact touched volume a force generation module that computes the forces to be sent This software uses ghost to communicate with device so it is also Sensable dependant. 21

22 Another key player in the haptics community is Reachin [reachin]. They propose Reachin API 4.0 to develop application with haptic support. This API supports different constructor devices such as the phantom form Sensable or the delta and Omega form ForceDimension or the laparoscopic surgical workstation from immersion. Also in this case, the sources are not publicly available. An open source solution is proposed by researchers from the Stanford University: CHAI 3D. It can be used as both high level and low level tool, to develop an application or by specifying our own opengl based graphic rendering. One of the limitations is that it only supports 3 degree of freedom [Conti et al., 2005]. Finally, a new open source solution was presented recently by researchers from the sirslab - University of Siena: the Haptik library [haptik]. The Haptik library is a component-based open-source library which provides a hardware abstraction layer to access haptic devices; it is built using loadable plugins that allow a extensible layout. Haptik is only for haptic device access and does not have support for graphical display Haptics frameworks Two interesting frameworks to develop haptics applications are RTPM and ITOUCH. Proposed by Pava and MacLean, RTPM (RealTime Platform Middleware) is an architecture for prototyping realtime multimodal I/O projects running within a network [Pava et al., 04]. RTPM uses Common Object Request Broker Architecture (CORBA) and a custom Virtual Device abstraction. Its architecture is shown in the following figure. Figure 6 Real time platform basic architecture Within the European project TOUCH HAPSYS, Pocheville et al proposed a multimodal haptic framework [Pocheville et al., 2004] for virtual prototyping: ITOUCH. It is designed to be modular and it proposes a haptic device abstraction to not be tied to a particular haptic device. As illustrated in the following figure, ITOUCH is divided into three modules: the core system is responsible for handling the OS and the configuration. Moreover, it manages the scene graph describing the objects in the scene. the input/output system handles the haptic device and the graphical rendering which is done based on opengl the simulation system is responsible of the behavioural model of the scene and of the collision detection. It is not clear whether I TOUCH will be relased as an opensource framework or not, but it is not available for now. 22

23 Figure 7 - I-touch framework architecture Virtual Reality Engine Although there is a large choice of VR engines, no specific standard is available yet which is able to meet all the different needs. HAPTEX sets very specific requirements in terms of physics simulation, visualization and haptic rendering. For our particular purpose, we will restrict our review at some representative engines and see what would be the solutions to get support for haptic device. VHD++ [Ponder et al., 2003] is a vertical real-time framework supporting component based development of interactive audio-visual simulation applications in the domains of Virtual and Augmented Reality. VHD++ is mainly developed in MIRALab (University of Geneva) and VRLab (EPFL), it is extensible through the use of component based development and it offers of large set of services. Its architecture can be seen on the following figure. Figure 8 VHD++ architecture overview by example One way VHD++ could support haptics could be through VRPN [Taylor et al., 2001]. VRPN permits to use different kind of VR hardware from CAVE to force feedback, it is a set of classes within a library that provides a level of abstraction for VR devices and allows them to run within a network transparent interface. The integration of VRPN into VHD++ would be quite straight-forward. Another tool that supports VRPN is VRJuggler [Bierbaum et al., 2001]. VRJuggler is a free and open source development framework for Virtual Reality engines started in Iowa 23

24 State University. But the solution of using VRPN was not chosen by Fisher and Vance [Fischer and Vance, 2003] that show an integration of a Sensable haptic device into VRJuggler using the Ghost. The last tool, OpenMask [openmask], is an open source platform developed by the siames team of irisa. It proposes a distributed system that handles the data flow and allows external modules such as collision detection or mechanical solver to be plugged and it also offer visualisation modules. An example of haptic integration has been realised by the i3d team of inria with the CONTACT collision detection library and a Spidar haptic device Grab device API The Grab is a force-feedback device with 2 points of interaction developed by PERCRO, a member of the HAPTEX Consortium. The Grab Device is controlled by a specific API. The EHAP library provides a complete framework for the development of haptic enabled application with particular features related to the GRAB device. The EHAP library provides services for the construction of virtual environments physically based where the user can interact by haptic devices. The EHAP system is structured following a stacked approach in a manner similar to the OSI stack of Networking. The lowest level is the interface to the hardware provided by the specific device driver, above that the DeviceLibrary provides an abstraction that allows the software definition of Device and Point of Contact. The Haptic layer provides the touching capabilities whereas the Physical layer provides dynamics based interaction with the objects. At the top level there is the Application layer. The modules that constitute the EHAP library are the following as depicted by the diagram. Dynamic Simulation: through external physics engines; Graphics Collision Detection Device Library Haptic Rendering Figure 9 - EHAP vs. OSI Stack Figure 10 - Modules of the EHAP library The EHAP system is a multi-rate multi-threaded system characterized by different components running in a concurrent way as presented in Figure 11. The main activity is provided by the Device Library that asks to the application the feedback force. The components of this interaction are: 24

25 Device Thread, that should compute the haptic rendering; Dynamics Thread, that computes the dynamic simulation of the environment; Graphics Thread for the visualization Figure 11 - Threads of the multi-rate EHAP System The EHAP library is made of different modules that provide different aspect of the simulation that can be performed through this library. A detailed description of the modules can be found on the HAPTEX website 3. Additional information about the Device Library, the part of the EHAP library dealing with the haptic hardware, is given in section Eligibility For sake of clarity, the features available of the different choice previously seen are settled on the following table. Simulation Graphic Rendering Collision detection Force rendering Tactile rendering Distributed System Device Driver Ghost Ogl * 3dof * OpenHaptic Ogl * 3dof * H3D Ogl * 3dof * CHAI 3D Ogl * 3dof * * Haptik * * VPS * * RTPM * * ITOUCH * Ogl * * * * VHDPP * Shaders * * * VrJuggler * Shaders * * * OpenMask * Shaders * * * EHAP * Ogl * * * * * Figure 12 Features available Open source 3 (Menu Item Dissemination-Papers and Documents) 25

26 Several comments should be drawn from this table. First the graphical rendering between VR engine and the other solutions are really different since those engines provide the support a lot of features, for example some of the latest shaders, and the other only use opengl so the user has to specify everything by hand. Another comment is that the haptic rendering needs to be a dedicated solution for HAPTEX since no other system provides support for tactile rendering. Moreover, the force rendering algorithm are restricted to 3 degrees of freedom (DOF), while HAPTEX targets 6 DOF. The first point that should be raised is the difficulties to integrate force feedback with a VR engine. Those Strong difficulties arise form the particularity of haptic that asks for an update rate of 1kHz. Indeed most of the VR Engines are build around a renderer that run at most at 200 Hz, because highest update rate are no use. And so the integration of haptic into a VR engine would ask a redesign of the core of the engine. Another disadvantage of VR Engine is that it has a lot of feature useless for our purpose, such as support for tracker, big scenes or collision detection which is provided by the consortium. And because of this wide range of features, it asks usually a leaning time not negligible to manage the behaviour of the engine. Furthermore RTPM doesn t meet our requirement because its interest is to run the application through a distributed system. On the other hand, none of the presented haptic toolkits could be a perfect solution for our challenging objectives. First, because the source code of most of them are not publicly available. Then, because of the uniqueness of our requirement in terms of haptic feedback. And finally for the Haptik toolkit, its main interest is its ability to use different kind of device and this ability is not interesting in our case because a dedicated device will be built. Concerning the EHAP library, it is clear that in the context of the HAPTEX project we will surely reuse some components (e.g. the device drivers and some architectural concepts). The adoption of the complete EHAP framework, however, is very questionable due to the unnecessary complexity arising from this choice, which would have no significant benefit. Moreover, the specific nature of the visualized graphics and the underlying physical models of the HAPTEX Project would still require to rewrite most components (e.g. graphics, physical simulation, tactile rendering) Ongoing work on Multimodal interaction Quite close to our interest, the field of multimodal interaction is really active, with the proposition of a standard by W3C and the development of a open source multimodal platform by the Network of Excellence Similar. The World Wide Web Consortium (W3C) is currently defining a standard Multimodal Interaction Framework [w3c]. As it can be seen on the following schema, the multimodal interaction framework aims at defining a level of abstraction above an architecture. But haptics is not explicitly considered by them. Figure 13 The W3C Multimodal Interaction Framework 26

27 An ongoing work to define a multimodal platform is carried on within the European Network of Excellence Similar [similar]. Within this network, OpenInterface [openinterface] is designed. It would be an open source platform that combines two areas: signal processing and human computer interaction. A support for the demanding need of haptic rendering is specifically mentioned in their specification. The different layered of this platform can be seen on the following figure. Figure 14 - Functional units interaction of OpenInterface platform Suitability and need of a dedicated solution As we have seen, the presented solutions for building multimodal VR applications are either too broad and complex, trying to provide interfaces for building very generic applications, or they are too specific and hardware-oriented. Due to the very new and very challenging issues tackled by HAPTEX, it is evident that for the development and implementation of the separate components we cannot rely on existing software. Both the graphics to be visualized and the new haptic hardware to be realized in the context of the HAPTEX Project are very specific and require ad-hoc implementation. In terms of software architecture, however, HAPTEX does not set very high requirements, being the computing performance the main constrain. On the graphical side, we deal with the problem of rendering highly realistic, physicalbased deformation of textiles in real-time. Since at this stage of the development the available computing ressources are too important for beeing used in rendering additional objects in VR, we decided to limit the graphical visualization of the whole HAPTEX System to the deformable cloth and a very simple hand model representing the user s interaction. There is no need, at this stage, to support other geometrical objects in virtual space. On the side of the haptics, we are going to develop new hardware, which will be supported by specific drivers which again will be implemented by the HAPTEX Consortium for this purpose. Furthermore, we have to deal not only with the specific hardware device, but also with the specific haptic rendering of textiles, both on a macrolayer (force-feedback) and micro-layer (tactile feedback), which is also not supported by any available platform. Facing these high challenging requirements for the single components, the fastest and easiest way to develop the software to be run on the HAPTEX demonstrator is to directly link them from scratch. This solution promises the best performance, since the interconnection of the single components is straightforward: to handle the information workflow between haptics and visualisation (large-scale- and small-scale model) we introduce a linking element, which defines the proximity region around the contact area. Through this channel we update both models. The visualisation is then handled by the Large Scale Model and rendered via opengl, while the haptic rendering is driven by the 27

28 Small Scale Model, which also communicates directly with the drivers of the HAPTEX haptic hardware. We surely can take advantage of previous work in the area of frameworks for multimodal system. The use of an existing framework, however, could eventually imply a potential decrease of computation speed and at the same time increase the complexity of the system, without bringing significant advantages. For this reason, we decided to develop a specific application. The HAPTEX software can be seen as a potential basis for developing future multimodal applications for haptic interaction with deformable objects. Thanks to the software documentation, developers from the community of multimodal VR applications will be able to build on this important experience and take advantage of the know-how acquired over the duration of HAPTEX. 5.2 Haptex System Overview The HAPTEX system uses complementary rendering techniques simultaneously in order to improve the global realism of a virtual environment. The multimodal computation and rendering software is composed by different components which are responsible for several key tasks such as visual rendering, physical-based simulation, haptic rendering and the control of the tactile device and the force-feedback device Overall View of the System For what concerns the system architecture of our multimodal haptic interaction software, our effort to support haptic/tactile perception of material requires the integration of the following components into one single application: A Large Scale Model able to evaluate the overall mechanical behaviour (deformations and reaction forces) of the whole fabric, to extract a local geometry close to the interacting fingertips and to evaluate the equivalent mechanical impedance of the remaing part of the fabric reduced to the boundary of the local geometry; A Small Scale Model able to evaluate the local deformations of the material interacting with the deformable fingertips, the actual contact area of the fingertips with the fabric, the force distribution on the local geometry and the relative velocity between fingertips and fabric; A database storing the parameters of both the macroscopic and the microscopic properties of different selected fabrics, serving as input for the Large Scale Model and the Small Scale Model respectively; A Tactile Renderer able to produce suitable commands for the tactile actuators according to the force distribution, to the relative velocity and to the small scale properties of the fabric; 28

29 Figure 15 Component view of the integrated HAPTEX System

30 Figure 16 - Intuitive view of the HAPTEX System 30

31 HAPTEX - HAPtic sensing of virtual TEXtiles D1.1: System requirements and architectural design 5.3 VTV - Virtual Textile Viewer The VTV Virtual Textile Viewer is in charge of the visualisation of the HAPTEX system. It provides basic interaction methods for changing the viewing camera position, as well as rendering options. The viewer s main task is to output the computed animation and provide the visual feedback for the haptic interaction, displaying both the simulated textile and the finger model. The Virtual Textile Viewer is based on the Microsoft Foundation Class Library (MFC). The use of MFC simplifies the development of a graphic application through the MFC templates, and allows easily designing and handling typical windows-style dialog boxes. Also, the document/view architecture implemented with MFC supports multiple views of the same document particularly well. However, there is also the need to keep the dependencies to MFC as low as possible, in order to avoid limitations deriving from its complex structure. Also, the fact that different releases of the MFC library may not be fully compatible among each other is another important reason for avoiding a strong coupling to MFC. In the following, we first give an overview of the use of the VTV, defining possible use cases. Then we briefly describe its architecture with the help of UML diagrams Use cases The user of the HAPTEX multimodal system can interact with it using the haptic interface and/or the mouse. Using the mouse, the user is able to change the view of the scene. Possible view manipulations are rotation, translation and zoom. It is possible to swap between the different camera manipulation settings using a combination of keyboard and mouse input, i.e. dragging the mouse while pressing the keyboard (e.g. the ALT- or CTRL-key). The rendering options can be selected using the respective key combination or using the menu. Possible options are rendering in point-, wire frame- or solid mode. It is also possible to display the forces exerted on the fabric toggling the respective setting. Using the haptic interface, the user is able to have a haptic perception of the virtual textile. The pointer of the haptic interface represents the user's fingertip. When the pointer collides with the virtual textile, the surface information at the contact point is sent to the haptic interface, and the user feels the virtual fabric. The mechanism of collision detection and -response, as well as the physical based computation of the forces acting on the textile, are handled by the internal processing modules of the HAPTEX application. The dynamics involved in this process serve as input to the real-time animation, which displays the resulting deformations of the textile in the viewer. Next to the simple collision, a possible interaction with the virtual fabric using the haptic device would imply the user holding the fabric, and being able to perform manipulations such as stretch, displace and shake. Figure 17 depicts the described use case: the user interaction with the VTV. All connections between the possible activities can be interpreted as <<extends>> relations.

32 HAPTEX - HAPtic sensing of virtual TEXtiles D1.1: System requirements and architectural design Figure 17 - User Interaction with VTV Data Structure The virtual textile will be represented by 3D polygonal mesh surfaces. The animation of this polygonal mesh will be performed computing the deformation of the virtual textile with time (mechanical simulation). The need for real-time performance will limit the mesh to less than thousand elements. Also for optimization reasons we will make use of a regular mesh. One of the objectives to be achieved is to simulate the full nonlinear anisotropic behaviour of cloth. For this purpose, the simulation scheme which will be developed within the HAPTEX project will base on first-order finite elements and share techniques from particle systems VTV Architecture The implementation of the Viewer is based on modules which interact with the MFC core of the application. The interaction with the Haptic renderer is represented by its interfacing module Haptic Manager and is only sketched for matters of completeness. The viewer is designed to integrate the several components and be independent from MFC and OS-related libraries, keeping in mind the goal of portability. 32

33 HAPTEX - HAPtic sensing of virtual TEXtiles D1.1: System requirements and architectural design Static view The following figure displays a static view of the Virtual Textile Viewer and shows its dependencies to the MFC document/view architecture. Figure 18 - Static UML view of the VTV architecture The separation between the VTV modules and the MFC core are clearly outlined in Figure 18. The Composition relationship, signified by a black diamond, is a strong form of containment. The arrowhead on the other end of the relationship denotes that the relationship is navigable in only one direction. Document is composed of Interface. In this way, Interface acts, as the name says, as interface: most of the interactions are handled by this module and not by Document. The same principle is used by the View module, which is composed of Camera Manipulation. Thanks to this scheme, the application is less MFC-dependent Dynamic View The sequence diagram reported in Figure 19 depicts three different actions: 1. Start of the application and object initialization At its start, the application instantiates all main objects forming the typical MFC application, as well as their interfaces as described in the previous part. 2. Virtual scene draw and update When the view has to be redrawn, the View receives an update request which is passed first to the Camera Manipulation module and subsequently to Interface. Here, the camera position update request is dispatched to the Haptic Manager, which communicates with the Haptic Renderer to update the haptic workspace. Interface also calls the virtual textile object to be displayed and both haptic scene and haptic pointer are updated. 3. Change of camera view through mouse interaction The third dynamic change shown represents a change of view using the mouse. A message is sent to the View, which forwards it to the Camera Manipulation. 33

34 HAPTEX - HAPtic sensing of virtual TEXtiles D1.1: System requirements and architectural design Here, the coordinates are updated and sent to Interface, which takes care of the rest. We see that the tasks managed by the MFC classes are kept at a minimum. In this context, the Interface takes a central position, as it receives most of the messages and dispatches them. 34

35 Figure 19 - Dynamic UML view of the VTV architecture

36 HAPTEX - HAPtic sensing of virtual TEXtiles D1.1: System requirements and architectural design Modules description MFC Core Document: Document class objects, created by document-template objects, manage the application's data in a typical MFC application. They are responsible for the internal representation of the data manipulated in the view, and may be associated with a data file. In our case most of these tasks are covered by the Interface module, in order to keep a weak dependency from MFC. View: In a MFC application, there might exists one or more view objects, each responsible for different views. Each view is a window that is attached to a document and associated with a frame window. Views display and manipulate the data contained in a document class object. These tasks are done in VTV by the Camera Manipulation module. VTV Interface: This module is at the centre of the application, as it handles most of the operations. Virtual Textile: The textile object is described by an array of points. This module describes the geometry of the textile object it gives its properties. Camera Manipulation: This module is in charge of the view; it handles the camera position, the projection and the mouse interaction. It is the base module defining the view, handling graphic initialization and the changes of view parameters. Other Components Haptic Manager: This module synchronizes the haptic rendering with the displayed object. 5.4 Large-Scale Model The Large-Scale model reproduces in-plane deformations using a representation that shares techniques from particle systems and finite elements. Its main features are the following: Geometry Can use any arbitrary triangle mesh as a description of the large-scale cloth geometry. For the specific needs of the HAPTEX project however, we will use a regular mesh for optimizing computation time. Real-time requirements will limit the mesh to less than thousand elements. Mechanical input The mechanical input of the model bases on strain-stress curves modelled as polynomial splines, for weft, warp and shear tensile deformation modes. Numerical resolution requires these curves to be C2-continuous. Performance considerations require these curves to be as low degree as possible (quadratic or cubic), and as few segments as possible. These curves are in practice approximated from the load and unload curves of KES- F tensile tests; finding a more precise way to approximate the curves will be

37 HAPTEX - HAPtic sensing of virtual TEXtiles D1.1: System requirements and architectural design one of the areas of investigation until month 24 of the project. The accurate modelling is limited to the expected range of tension for our application depending also on the limitations on the side of the haptic hardware. Bending stiffness Bending stiffness will be neglected in the large-scale model, since its effects are too weak and the elements too large to make any significant and accurate change. This will considerably enhance the computation speed. Deformation viscosity The system also supports deformation viscosity (force / deformation speed) which is the only really mechanical dissipative effect that can be modelled by the system, and would be required for modelling accurately the dissipative effect of cloth motion (issue: how to measure viscosity of fabrics?). Additional parameters Additional parameters can be: fabric surface mass gravity anisotropic viscous aerodynamic damping Resolution The resolution is carried out using a particle system that considers three-way interactions on the mesh triangles effectively modelling the accurate strainstress behaviour of the cloth surface. This offers much more accuracy than the traditional spring-mass models currently used for cloth simulation. Numerical integration The numerical integration is carried out using variations of Implicit Euler and Implicit Midpoint schemes, adaptively for optimal accuracy and stability. The Jacobian of the mechanical forces is dynamically recomputed at each iteration, for ensuring a good accuracy and performance when dealing with nonlinear models. This is done through a particular implementation of the Conjugate Gradient method used during the resolution. This system is numerically equivalent to a nonlinear dynamic first-order finite element system (degrees of freedom = position and velocity of the mesh vertices). Output to Haptic Renderer The Large Scale Model sends to the haptic renderer the Local Geometry of the fabric that is a finer interpolation of its surface in proximity of the user fingertips. Furthermore it sends the mechanical impedance (Force Jacobian) of the remaining part of the fabric reduced to the boundary of the local geometry. 5.5 The Haptic Renderer The haptic renderer is responsible for controlling the haptic devices. As there are two different kinds of haptic devices, namely force feedback and tactile devices, the haptic renderer is divided into two main packages. 37

38 HAPTEX - HAPtic sensing of virtual TEXtiles D1.1: System requirements and architectural design Overview of the Haptic Renderer The following figure depicts the communication flow between the different components of the haptic renderer. The components involved in the force computations are in the upper part, whereas the tactile components are in the lower part of the image. Figure 20 - Communication flow between force- and tactile renderer Package: ForceRenderer The ForceRenderer package is the link between the physical simulation and the force feedback devices. As the force feedback devices require forces to be sent at a higher rate than the physical simulation can compute, it has to apply calculations on an approximated local model of the simulation to satisfy this condition. Furthermore, it has to take the finger 38

39 HAPTEX - HAPtic sensing of virtual TEXtiles D1.1: System requirements and architectural design deformation under consideration to obtain the contact area on the textile which is needed along with its movement by the tactile renderer (see Figure 21 for an illustration of the interaction with the physical simulation). This package contains the following components: ForceRenderer The ForceRenderer class defines the interface to the physical simulation and manages the contact handlers for multi-contact interaction. DeviceManager It takes care of the attached devices. In detail it is responsible for initialisation, calibration and integration of force feedback devices different kind in the virtual space. FFDevice This component encapsulates any force feedback hardware. Therefore it is defined taking into account the common properties of force feedback devices. These are the number of contact points the user can interact with in a virtual environment. For example, a force feedback driven data glove can have five contact points, a stylus device only one. HPoint HPoint represents a mediator for forces and position changes of contact points. One can attach listeners to be notified of any change in the state of a contact point. The mediation is bi-directional which means that also forces can be directly applied to the contact point. ContactHandler As the name already implies, it handles the contact between a haptic point (HPoint) and objects. The component is defined as abstract in order to be independent of the underlying contact model that is used for the haptic point interaction (e.g. fingertip deformation or Rigid Sphere/Virtual Proxy). Local Geometry Physical Simulation Force distribution on local geometry ForceRenderer with Fingertip deformations Figure 21 - Interaction between physical simulation and force renderer 39

40 HAPTEX - HAPtic sensing of virtual TEXtiles D1.1: System requirements and architectural design Package: TactileRenderer The tactile renderer controls the tactile devices consisting of an array of contactors. Every single contactor is driven by superposed oscillations of different frequencies. The amplitude of the signals is adjusted by the tactile renderer with an update rate of 40 Hz using the information about the fingertips movement over the surface. Different approaches will be tested for the tactile rendering. Experiments based on [Gescheider et al., 2002; Johnson et al., 2001; Hollins et al., 2002] will be carried out. This package contains the following components: TactileDevice This component encapsulates a single actuator array. Information about the geometrical alignment of the actuators is expressed in a two-dimensional device coordinate system. The generated signals for the actuators can be controlled by appropriate methods. TactileModel Representation of a tactile texture. For every two-dimensional texture coordinate an appropriate SurfaceProperty object is returned. TactileRenderer Renderer that uses the contact information of the fingertip and the corresponding TactileModels to generate appropriate signals for the corresponding TactileDevices. SurfaceProperty A structure that represents properties of a single point of the TactileModel (e.g. height and friction). ContactArea Representation of the contact between a single fingertip and the garment in texture coordinates. For a contact information about the corresponding TactileModel and the orientation of the fingertip is also provided. This component can also represent no contact between the fingertip and the garment. The area of contact is also available. ContactMovement In this component the ContactAreas are managed. These areas are computed by the ContactHandler of the ForceRenderer package at a high frequency Small-Scale Model The goal of the preprocessing of weave textures is the generation of an appropriate small scale surface model of given fabrics. This model needs to fulfil the requirements of the tactile renderer. The small-scale surface model describes the height profile of the fabric, and is not to confuse with the output to the haptic renderer stemming from the large-scale model, which gives the geometrical shape interpolation of the contact point. The preprocessing is not a real-time operation and may even be manually supported. There are currently two different sources of input: 1. Kawabata measurements (in detail, surface friction and geometrical roughness) 2. Optical surface scan (e.g. photograph) 40

41 HAPTEX - HAPtic sensing of virtual TEXtiles D1.1: System requirements and architectural design Kawabata measurements The Kawabata measurements are described in [Kawabata, 1980]. Significant for the small scale surface are the measurements of roughness and friction (KES-FB4). For a description of the Kawabata measurements, please refer to In a first step the measured data has to be reduced to its basic essentials. This is achieved by a combination of noise reduction (e.g. by wavelet thresholding), Fourier transform and filtering of certain frequencies. As the Kawabata measurements are one-dimensional (in warp and weft direction) a twodimensional surface has to be reconstructed. A very similar method for the generation of a small scale surface model is described in [Huang et al., 2003] and in more detail in [Huang, 2002] Optical Surface Scan An optical surface scan of a fabric sample is easy to obtain and may be used to reconstruct its height profile. This method was implemented at the University of Hanover and is described in detail in [Schulze, 2005]. First, the surface periodicity has to be analysed to reduce the amount of data and of noise. Therefore, the smallest tile has to be found by the recognition of the isometry group with the aid of stochastic methods. To obtain a height profile we use a shape-from-shading algorithm [Ping-Sing et al., 1994] More Information about the Weaving Woven Jacquard fabrics involve at least two different weaves (see Figure 22). A texture segmentation algorithm can detect the borders between these different weaves resulting in an appropriate model of this specific kind of fabrics [Chaudhuri et al., 1995]. 41

42 HAPTEX - HAPtic sensing of virtual TEXtiles D1.1: System requirements and architectural design Figure 22 - Woven-in motifs in Jacquard fabric by using two opposing weaves Expected data flow with 3D visualisation and hardware interface The physical simulation within the 3D Visualisation package is unable to run at an update rate of 1 khz required for the haptic sensation. Therefore, the simulation has to transmit the relevant information of the model to the force renderer in order to calculate the steps between the simulation cycles. The region most likely being a candidate for these intermediate haptic force calculations is defined by a bounding sphere considering the predicted position and the catchment area of the contact model (e.g. fingertip). The simulation requests this bound from the ForceRenderer and sends the local information back containing the following: local geometry (from the global model) forces and their derivatives The local geometry allows the collision detection between fingertip and the (local) model to take place inside the haptic loop. Complementary to the geometry, the simulation has to deliver information of the underlying physical model (e.g.: force vectors and their gradients), such that in case of a contact between fabric and fingertip the force renderer can compute deformations of the fingertip and the resultant forces on the fabric. Afterwards, the forces are then transferred back to the simulation to update the global model. The following figures summarize the dataflow of the force renderer and the tactile renderer in chronological order. 42

43 HAPTEX - HAPtic sensing of virtual TEXtiles D1.1: System requirements and architectural design Physical Simulation Force Renderer Contact Handler FFDevice ProximityRegion ProximityRegion HPoint LocalGeometry LocalGeometry Forces Force Distributions ProximityRegion LocalGeometry Force Distribution ProximityRegion LocalGeometry Figure 23 - Force Renderer Figure 24 - Tactile Renderer Fingertip model for haptic force computations Although the virtual proxy (point contact) model is sufficient for many applications and has also some advantages like the fast computation, it is neglecting the tangential torque occurring in real-world contact situations [Kinoshita et al., 1997]. Tangential 43

44 HAPTEX - HAPtic sensing of virtual TEXtiles D1.1: System requirements and architectural design torques can arise because the normal force is distributed across the contact area rather than focused at one point. This circumstance necessitates the integration of a fingertip deformation model [Serina et al., 1998] into the haptic rendering. As the system has to compute the contact area within the fast haptic loop we cannot use a very precise model. The computational load would consume the whole processing power and leaves no time for other calculations. Hence it is not only desirable to have a rather simple model [Barbagli et al., 2004] but also a model which can increase or decrease in precision having regard to the remaining processing power within the haptic loop. Therefore, we define an interface that allows the fingertip deformation model to change the precision at runtime. This is achieved by sending the local geometry from the contact handler to the deformation model. The model modifies then the geometry and the respective forces at the points which are in contact with the fingertip. Because of the restriction to a local geometry these modifications can be calculated within the small timeframe of the haptic loop. 5.6 The Tactile Drive System Background The tactile stimulator for the HAPTEX project will have approximately 48 contactors distributed over the tips of the thumb and the index finger. The drive signal for each contactor will be independently specified, using dedicated electronic hardware under computer control. This external hardware (and associated control software) will be specifically developed for the HAPTEX project. The design will incorporate waveform generation in the external hardware, with digital control signals to specify the frequency content of the individual drive waveforms. This design derives from two systems previously developed by UNEXE: 1. a system which is computationally more complex, in which the drive waveforms are completely specified in software (i.e., the software generates a multiplexed sequence of sample values for the drive waveforms to each contactor) and the external hardware is simply a set of digital-to-analogue converters; 2. a system which is computationally simpler, in which the frequency content of the drive waveforms is set globally in the external hardware, and the software generates only on/off control signals for each contactor Proposed design Figure 25 shows the proposed design for the drive electronics. The drive signal in each channel is a mixture of signals from two (or more) sinewave generators with different frequencies. The mixture is specified by digitally controlled attenuators (2-bit attenuators are shown, giving three levels and off, but 3-bit or 4-bit systems are possible alternatives) Tactile renderer Figure 26 gives an outline of links to the software module (the tactile renderer). This provides control signals to the drive electronics in the form of data packets which specify 44

45 HAPTEX - HAPtic sensing of virtual TEXtiles D1.1: System requirements and architectural design the drive signals delivered to each of the contactors (as shown in Figure 25). Control data for the drive electronics require updating at around 40 times per second. (There may be a limited advantage of using a higher update rate of around 80 Hz). The output from the tactile renderer derives in the first instance from surface profiles and measured frictional variations which are represented in the small-scale surface model. The output is also dependent on the position, orientation, velocity and contact pressure of the finger and thumb. An important aspect of the tactile output will be the representation of contact area on the skin. In fact, for a realistic simulation, the spatial distribution of tactile stimulation may be more important than the precise nature of the drive waveforms. Figure 25 - Schematic diagram of the drive electronics for the tactile stimulator 45

46 HAPTEX - HAPtic sensing of virtual TEXtiles D1.1: System requirements and architectural design Figure 26 - Schematic diagram of the tactile renderer and associated components 5.7 The Force-Feedback Device Input/output of the Force Feedback Device Definitions The Force Feedback Device (FFD) is composed by two main components: The Electro- Mechanical Hardware (EMH) and the Low Level Controller (LLC), i.e. the electronic hardware plus the control software (see Figure 27). The EMH of the FFD has two main functionalities: to exert forces on the user fingers in response to proper electric currents supplied by the LLC to its actuators. to provide position signals to the LLC relating to its current kinematic configuration, that is correlated with the user s posture. 46

47 HAPTEX - HAPtic sensing of virtual TEXtiles D1.1: System requirements and architectural design Figure 27 - Scheme of the FFD components The Low Level Controller has two main functionalities: to supply the proper currents to the actuators of the EMH on the basis of the forces on the users supplied as input by A.P.I. to translate the signals from the EMH in its kinematic configuration and transmit to A.P.I. the position and velocity of the defined bodies of the EMH in contact with the user. The LLC operates at relatively high refresh rate (typically 1kHz). Within the HAPTEX project it is foreseen to realize two different FFD: the first one is named Two Point Interaction Device and the second Whole Hand Interaction Device Two Point Interaction Device Functionality The Two Point Interaction Device will be able to exert 2 independent forces on the fingertips of the user s thumb and index. Each force can have arbitrary orientation and intensity (within predefined limits) in 3D space. Moreover the FFD will give the position and the orientation of the fingertips respect to a fixed frame. The force data, provided by the A.P.I., are composed by two 3-component vectors, representing the two forces on the user fingertips. The position provided by the device is composed by two 6-component vectors, each representing the position of a point belonging to the fingertip and the orientation of the fingertip. There are various forms for representing the orientation; the choice of the most suitable will be made considering the requirements of the dedicated A.P.I. 47

48 HAPTEX - HAPtic sensing of virtual TEXtiles D1.1: System requirements and architectural design Figure 28 - Scheme of the forces that can be exerted with the Two Point Interaction Device Interface with the user The two points FFD will act on the index and thumb fingertips of the same hand (see Figure 28). The interaction components of the EMH will be permanently in contact with these fingertips. Through these components it will be possible to exert on the user s fingertips the required forces and it will be also possible to track the movements of the user s fingertips, estimating their absolute position and velocity. Interface with A.P.I. Input from A.P.I The input from the A.P.I. consists of 2 independent 3-component vectors each representing the force vector to be exerted on the user s fingertip. Two force vectors of three components can be specified, one for the index fingertip and one for the thumb fingertip. The two vectors are expressed in a fixed, absolute, Cartesian reference frame. Output to A.P.I The output to the A.P.I. will consist of 2 independent 6-component vectors, each representing the position of a point of the user fingertip (the definition of such point will be given after the detailed design of the device, see Figure 29) and three angles (for example Euler angles) that give the orientation of the fingertip. Moreover, since the knowledge of velocity of the finger is fundamental for a realistic tactile feedback, the FFD will provide 2 independent 6-component velocity vectors, one for each finger. The first three components of such a vector represent the linear velocity of the defined point of the fingertip and the last three components will represent the angular velocity of the fingertip. Figure 29 - Reference Point for the finger position (dimensions to be defined) 48

49 HAPTEX - HAPtic sensing of virtual TEXtiles D1.1: System requirements and architectural design Whole Hand Interaction Device Functionality The Whole Hand Interaction Device will be able to exert 11 independent 1-component forces, one for each phalanx of the user fingers (excluding the little finger). Each of such forces will lay in the sagittal plane of the finger and it is normal to the longitudinal axis of the corresponding phalanx (see Figure 30). The FFD will provide also the kinematic configuration (posture) of the user s hand. Figure 30 - Scheme of the force that can be exerted by the Whole Hand Interaction Device Interface with the User The Whole Hand Interaction Device will act on the user s hand through 11 contact points. The forces will be transmitted through the interaction components of the EMH that will be permanently in contact with the palmar region of each phalanx of the user s fingers. The kinematic configuration of the user s hand will be estimated by measuring the relative position of these components. Interface with the A.P.I. Input from A.P.I The input from the A.P.I. consists of: 3 vectors of 3 components, each representing the 3 forces to be applied on the phalanxes of the finger (for the index, middle and ring fingers) 1 vector of 2 components representing the two forces to be applied on the phalanxes of the thumb. Output to A.P.I The output to the A.P.I. will consists in: 4 vectors of 4 components, each representing the relative angles between the adjacent phalanxes of the finger. 4 vectors of 4 components, each representing the time derivative of the above finger relative angles. 1 vector of 6 components, representing the position and the orientation of the user palm respect to a fixed frame. In order to give the palm position, a point on the user palm has to be defined (the complete definition of such point will be given after the detailed design of the device) 49

50 HAPTEX - HAPtic sensing of virtual TEXtiles D1.1: System requirements and architectural design The Grab Device The Grab is a force-feedback device with 2 points of interaction developed by PERCRO, a member of the HAPTEX Consortium. The GRAB haptic device is controlled by the user application through a driver running over the Windows platform. This driver communicates with the controller machine of the device through parallel port or network link. The developer can interface to this driver by a C++ library called DeviceLibrary. While the EHAP library the general framework for developing haptics applications with the Grab has been already presented in section 5.1.3, in the following we will give a closer look at the the whole system configuration and at the Device Library, which represents the interface of the software with the haptic hardware The complete Grab System The entities of the GRAB system are: GRAB device; Control PC connected to the device through power and control cables; Application PC that is connected to the Control PC with a network cable; Grab Driver, which is a software that controls the status of the device; DeviceLibrary, a C++ software library that abstracts the interaction with the Driver; EHAP, a complete library for the development of Haptic Application; User Application; The following diagram shows the relation between the above entities of the system. The upper part of the diagram presents the connection between the three HW entities whereas the lower part details the internal structure of the software inside the Application PC. Control PC ethernet Application PC Ethernet Grab Driver DeviceLibrary User Application Figure 31 - Relation between entities of the GRAB system 50

51 HAPTEX - HAPtic sensing of virtual TEXtiles D1.1: System requirements and architectural design The Device Library The DeviceLibrary is a library that is part of the broader EHAP library of PERCRO that provides the interface to different devices, like commercial devices and the GRAB itself. This library can be used inside the EHAP or separately. The core of this library is a class named BaseHI that provides an abstract interface to different devices with the following structure: Figure 32 - Device library class diagram The device description provided by the BaseHI contains a number of points of contact for the haptic interaction and a number of buttons for the user selection. The BaseHI requires the user to provide a call-back interface of class DevAppInterface that is invoked inside the haptic loop for providing a force response to the device. The device attached to the BaseHI can be initialized by the Activate method and then the haptic loop is controlled by the Start and Stop methods. 51

52 HAPTEX - HAPtic sensing of virtual TEXtiles D1.1: System requirements and architectural design 5.8 Data Exchange Overview Visualisation and Haptic Rendering (MIRALab-UHAN) 52