Design guidelines for generating force feedback on fingertips using haptic interfaces

Transcription

1 Design guidelines for generating force feedback on fingertips using haptic interfaces Carlo Alberto Avizzano Antonio Frisoli Massimo Bergamasco PERCRO Laboratory Scuola Superiore Sant Anna Pisa, ITALY Editor: Martin Grunwald Manipulation and grasping have key importance in most types of interactions between humans and the world surrounding them (Douglas and Kirkpatrick, 2002; Klatzky and al., 1990). Even if almost all existing haptic interfaces provide a user interaction based on a single contact point, an increased number of contact points, not only allows to display a more natural haptic interaction (Jansson and Monaci, 2003; Frisoli et al., 2004), but also improves the quality of interaction that users can perform in the environment. Haptic exploration is highly dependent on the number of fingers used for exploration of common objects (Jansson, 2000), the largest difference appearing between the One finger and the Two fingers conditions (Jansson et al., 2003), and as proven by Jansson et al. (Jansson and Monaci, 2006) by the ability to discriminate a precise tactile pattern during the exploration. In (Frisoli et al., 2005), we found an experimental confirmation of this hypothesis: the haptic exploration do not improve with the increase of contact points, from one to two fingers. This suggests that the restriction imposed on the fingerpad contact region can blunt the haptic perception of shape and so indicates that local haptic cues play an important role in haptic perception of shape. Factors that can account for the observed performance in these experiments are lack of physical location of the contact on the fingerpad and lack of geometrical information on the orientation of the contact area, that constitute interesting insights and suggestions for the design of haptic displays. Multipoint haptics (Barbagli et al., 2004a,b) are devices that can simultaneously interact with the user through more than one contact point. These systems allow both force and torque feedback during the simulation of dexterous manipulation and complex manoeuvring of virtual objects and can improve the interaction in several applications, e.g. assembly and disassembly in virtual prototyping (McNeely et al., 1999; Wan and McNeely, 2003), medical palpation during simulated physical examination of patients (Kuroda et al., 2005) and many other ones. In this chapter we present different approaches to improve the quality of haptic feedback during virtual manipulation of objects Four different aspects of modelling perception and manipulation are proposed and investigated through conducted experimental studies. c 2007 PERCRO, Scuola Superiore Sant Anna

2 Initially we investigate the capabilities of using a haptic system for grasping and manipulating virtual objects, by means of a two contact points haptic device. The work also investigates the relationship between human prehension and features of the physical model of the grasped object, finding out how grasping in virtual conditions present higher forces and safety margins than in real conditions. A possible motivation of this observed difference is due to the limitations of kinesthetic devices in stimulating local mechanoreceptors. A possibile improvement in this sense to kinesthetic haptic devices is presented in section 2, where we wonder whether local haptic cues provided at the fingertip can improve haptic perception of shape. We adopt a prototype of a new encountered haptics, allowing haptic exploration of three dimensional shapes, and show how discrimination threshold for curvature perception can be significantly improved by providing both kinesthetic and local haptic cues at the contact point. In section 3 we introduce an alternative approach to enhance haptic perception, by using haptic illusions to elicit haptic sensations, and discuss about potential applications and ways to simplify the design of future haptic devices. Finally in section 4, we evaluate the usage of high frequency vibrotactile tactors to render the feeling of contact during the manipulation. Such a device consists in an active digital glove integrated with an array of vibrotactile actuators placed at level of fingers phalanxes. The device is equipped with embedded electronics that allows to control the motors directly from the Virtual Environment. 1. An investigation of manipulation capabilities in Virtual Environments Grasping an object allows us to identify some of its properties (geometry, material, surface textures, Klatzky et al. (1999)), change its physical state (position in space, internal structure) and use it for mediated interaction with other objects. The possibility of interacting with more than one point of contact is fundamental for the manipulation of objects in virtual environments. Humans un-consciously use sub optimal (Westling and Johansson, 1984) algorithms for the prehension of objects when performing tasks with their hands. For instance, during a peg-in-hole task, they precisely adjust the relative position and interaction force between the peg and hole. Johansson and Westling conducted a series of experiments, relating tactile information to grip force when performing a lifting task (Johansson and Westling, 1984). The ability to adjust grip force appears to be independent of the surface friction characteristics, but further studies from the same authors confirmed that this is not true for the case of objects with differet curvature, and propose an active role of rotational friction for the stabilization of the grip (A.W. Goodwin and Johansson, 1998). The nature of contact during slip provides important tactile cues regarding features on the surface as well as the nature of movement of the object, and can explain how humans take advantage of slip sensitivity when perceiving objects. In this study, we used a GRAB haptic system (Avizzano et al., 2003), composed of two identical robotic arms, to provide the force-feedback for the two fingers during simulated grasping operations. The user can operate the device by inserting his fingers in two thimbles placed on the end-effectors of both the arms, as show in figure 1, so that both single hand (thumb and index) and two hands (right and left indexes of two hands) interaction are possible. 2

3 Figure 1: The device during a typical grasping procedure (left) and the associated force rendering integration scheme A set of rubber thimbles of different sizes allow any finger size to properly fit in the device. Each arm has 6 degrees of freedom (dofs), of which the first three ones, required to track the position of the fingertip in the space, are actuated, while the last three ones, required to track its orientation, are passive. While the user is grasping and manipulating objects in the virtual environment, the device is controlled by multiple concurrent threads, from the fastest internal 1KHz haptic loop to slower external collision detection loops, physical modelling, up to the slower 20Hz graphical loop (see figure 1). The Collision Detection is actually implemented using an external module based on bounding volume hierarchy and running at about 100Hz. The simulation of the dynamics of the objects is achieved through a Dynamic Simulator, that is carried out through a separate thread running at about 200 Hz. Whenever an object is grasped in virtual environment by the user s fingers, it is virtually tied to the contact points through a couple of springs representing the virtual stiffness of the environment. The application allow the experiment to change the working parameters in terms of size, weight and surface stiffness. While the size was kept fixed, in our experiments we interchanged weights from 0.1 to 0.5 Kg, and surface stiffness from 500 to 2000 N/m. The control loop also implemented a linear friction model who generates the force information for determining the object motion. The position of each contact point is measured directly by the haptic interface (x h ) and the relative feedback force (F) is computed through a constraint-based proxy method with friction, based on (Melder and Harwin, 2004; Barbagli et al., 2003). The haptic rendering algorithm computes the position of an additional proxy point x p, lying on the object geometry, and on the basis of current xh position of the haptic interface and contact geometry (C) the force is generated through a direct rendering method F = G(x h, x p,c), using the elastic coupling between the proxy x p and the interface position x h. The quality of object grasping was investigated by means of several numerical experiments. Firstly the force sequences during grasping, such as the ones shown in figure 2, were investigated. Three phases can be identified: the first is the lifting phase when the user starts to grasp the object, the second is the holding phase and finally there is the releasing 3

4 Figure 2: Force and position during grasping, lifting, holding and release of a virtual object phase when the object starts to slide. In the diagrams the green and red lines represent the position of the contact point and the object along the Y axis (aligned along the gravity vector). During the lifting and holding phases the two positions have a constant difference that depends on the grasping point on the object, then in the releasing phase they diverge because the object is falling. The black line shows the status of the friction that is zero when there is no contact, one during the non-slip state and two in the slip state: it is clear that during the lifting and holding phases the proxy is in non-slip state because it is firm between the fingers and, when the object is released, initially it changes to slip mode and then the contact is lost. Finally the blue line represents the grasping force that has an increasing and varying behaviour during the lifting phase, but it is almost constant during the holding phase. In order to assess the pick and place operations, grasping information were compared to available data on human grasping of real objects (Westling and Johansson, 1984; Johannes and Green, 1973). The influence of weight on static grip has been experimentally studied by (Westling and Johansson, 1984), where safety margins for grasping for prevention of slipping are analyzed. The safety margin is defined as the difference between the grip force 4

5 and the slip force, that is the minimum grip force required for preventing slipping. Three healthy right-handed men, aged between 27 and 35 yrs, served as subjects for the study. The subjects sat on a height-adjustable chair. In this position the subject might held with his right hand the two thimbles connected to the haptic interfaces, and respectively wear them on the thumb and right index of his hand. A wide visualization screen was placed in front of the screen and a desktop, where during the experiment the subject was invited to place its elbow. A sequence of 27 objects was presented twice to each subject, for a total of 54 runs performed by each subject. All the objects in the randomized sequence were cubes with the same geometry, with pseudorandom changes in the weight m (0.1,0.2,0.4 Kg), in the friction coefficient, both static μ s (0.4, 0.8, 1.2) and dynamic μ d (0.3, 0.6, 1.1), and in the stiffness k(0.5,1, 2 N/mm). In each randomized sequence all the possible combinations of weight, friction and stiffness, without repetition, were presented to the subject. The values of μ d were associated to μ s. The experiments were conducted with only one grasping condition, with the object hold between index and thumb tip of the same hand. Values for friction coefficients were assumed by (Kinoshita et al., 1997) where experimental values of linear friction are reported between index tip and different materials, equal respectively to 0.42, 0.61 and 1.67 for rayon, suede and sandpaper. Each subject was asked to grasp and lift the object using index and thumb and then to slowly release the object letting it falling down. A significant correlation was found between the values of the gripping force F n and stiffness, weight and friction values. The value of grip force F n was found to be significantly positively correlated with mass and stiffness, while negatively with friction value (p < 0.001). Greater gripper forces are required for holding heavier weights and stiffer objects, while lower gripper forces are required for higher friction values. In (Westling and Johansson, 1984) it was found that the relative safety margin, defined as the safety margin in percent of the grip force, was about constant during lifting with increase of weights, was almost constant with change of weight. The calculation of the safety margin in the case of virtual manipulation allows to make an interesting comparison. As it is shown in the logarithmic plot in Figure 3 below in the case of virtual manipulation, the safety margin tends to be reduced with increasing weight of the lifted mass. This can be explained by the larger dispersion of grip forces observed for lower mass values. In fact, due to the absence of local sensation of slip, it was more difficult to discriminate the weight of lighter objects. Moreover lighter objects required a smaller resolution in the control of force (ΔF), that is limited by the position resolution of the device ΔX, according to the law ΔF = kδx, where k is the simulated contact stiffness. This is confirmed by the finding that better safety margins are obtained for lower values of the stiffness, as it is evident from the plot above, where grip forces are plotted vs. slip forces. While the minimum required grip force is represented by the diagonal line, experimental data can be clustered in three main groups according to the value of contact stiffness during the simulation. The evaluation of the performance during grasping and weight lifting, has shown that the simulation produces outcomes that are similar to experimental findings on real objects. Overall from this study it results that kinesthetic haptic devices when used to simulate operations of grasping blunt the haptic perception in such a way that safety margins adopted by human subjects are greater than in reality. 5

6 Figure 3: Safety margins in manipulation of a virtual object 2. Enhancing haptic perception by directional and geometrical local cues A way to improve the local perception of shape and grip forces at the contact points is to elicit a direct stimulation of the mechanoreceptors at the fingertip, enriching the kinesthetic force feedback usually provided with traditional haptic devices. Different solutions have been proposed to validate the effects of an encountered haptics. In (Kuchenbecker et al., 2004) the shape recognition is due either to the perception of slipping of the fingerpad over the object surface or to the displacement of the contact area over the fingerpad. In (Salada et al., 2002) preliminary tests reveal that relative motion can be used to render haptic sensation. In (Kuchenbecker et al., 2004), a new haptic device is presented which integrates grounded point-force display with the presentation of contact location over the fingerpad area. The second approach considers that recognition of shape is linked to the perception of the orientation of the object surface at the contact points. Hayward et al. (Dostmohamed and Hayward, 2005) demonstrated how curvature discrimination can be carried out through a device providing only directional cues at the level of the fingerpad, without any kinesthetic information and moreover with a planar motion of the finger. This concept is also exploited to build robotic systems that can orient mobile surfaces on the tangent planes to the virtual object that is simulated, only at the contact points with the finger (Yokokohji et al., 2005). In the solution presented therein two haptic devices were differently used to support and track user finger and to present force feedback to the finger tip. In this section we describe a new type of encountered haptic system, composed of a fingertip haptic interface in the shape of a plate (Cini et al., 2005) integrated with a tracking device. The interaction among the user and the device goes by means of an active control of the haptic plate which is moved in correspondence of the virtual objects to be touched, while the tracking system plays the role of accurately monitoring the finger position. Suppose the user is interacting with a virtual object: when the finger is out of the surface of the object, the plate is kept far apart from the fingertip. When the finger touches the virtual surface, the plate comes into contact with the fingertip with an orientation determined by the geometric normal of the explored surface, as shown in figure 4. 6

7 Figure 4: Conceptual scheme of the device Figure 5: PERCRO Encountered haptic device, details of the fingertip plate and tracker 2.1 Fingertip haptic interface The supporting haptic interface is a pure translational parallel manipulator with three degrees of freedom (DoF). A classic impedance haptic control scheme was adopted, generating a force proportional to the penetration in the virtual surface. As shown in figure 5, the fingertip haptic interface was devised to bring the final plate into contact with the fingertip with different orientations, according to the direction of the perpendicular to the virtual surface in the point of contact. Moreover, the contact can occur at different points of the fingertip surface, depending on its orientation in respect of the virtual surface. These requirements can be satisfied by a kinematics with five DoF, three translational and two rotational ones. A hybrid kinematics, consisting of a first parallel translational stage and a second parallel rotational stage, resulted the most suitable solution. The translational stage has the same kinematics of the supporting haptic interface, with 3-UPU legs. In each leg the cable connected to the motor and a compression spring are mounted aligned to the centers of the universal joints. The spring works in opposition with the motor, in order to generate the required actuation force and to guarantee a pre-load on the cable. The control was implemented with local position controllers at the joint level. An inverse kinematic module was used to convert the desired position expressed in cartesian coordinates to the corresponding joint coordinates. The non-linear term due to the spring pre-load and to the wieght of the device was compensated, by a feedforward term in the control loop. 7

8 2.2 Can local haptic cues improve haptic perception? Performances were estimated by simulating the contact with a virtual sphere having a radius of 70mm. Fingertip positions and the interaction forces were monitored during the interaction: when the finger is out of the sphere, the platform is moved far apart of a given offset from the finger. When the finger comes in contact with the sphere, the two positions coincide, meaning that the plate is in contact with the finger. In figure 6, the black continuous line represents a scaled representation of force (with an offset only for the purpose of superimposing it to the plot) generated by the supporting haptic interface: the force is null when the finger is out of the sphere. Figure 6: Experimental plot showing the response of the two devices A psychophysics assessment was carried out in order to measure the discrimination threshold in the perception of curvature, with only kinesthetic devices and with the additional of informative geometric local cues, through the new device. The procedure was based on the Theory of Signal Detection (TSD). Four participants were recruited for the experiment, three males and one female. They were completely novices to haptic interfaces and did not present any dysfunction of the fingers. The Same-Different procedure of TSD was adopted for the determination of the difference threshold. The test consisted in exploring in the virtual environment a pair of curved surfaces. The exploration was carried out in a restricted workspace, consisting in a vertical cylinder with a diameter of 25mm. Figure 7 shows a planar scheme of the displayed haptic cues (virtual spheres with given curvature) and their position in the space, relatively to the device. The curvature of the two presented surfaces could be either the same or different. The two stimuli were randomly presented to the observers; each series was composed by 100 trials with the same probability to have different or equal stimuli. The observer s task was to judge if the curvature of the two surfaces was different or the same. The test was carried out in two different modalities, A and B: in condition A both the kinesthetic and the local geometry haptic cues were provided to the observers, while in condition B only the kinesthetic feedback was provided. In modality A, the mobile platform of the fingertip device was kept in contact with the fingerpad when the user was 8

9 Figure 7: Representation of the displayed haptic cues Curvatures Δcurvature Series 1 5m-1/6m-1 1m-1 Series m-1 / 6 m m-1 Series 3 4m-1/6m-1 2m-1 Table 1: Presented curvature values in contact with the surface, with an orientation tangent to the displayed virtual surface. As a result the observer was perceiving in the contact point the indentation of the platform, oriented along the direction of the normal force applied by kinesthetic device. In the second modality the mobile platform was substituted by a fixed thimble, into which the user was required to insert its finger. In this case the only haptic cue applied to the fingerpad was the force perpendicular to the virtual surface generated by the supporting haptic interface; no local geometry information was provided. The sequence of the series in the two modality changed for each subject, in order to minimize the influence of the participant s learning for the procedure. For each series the hit rate p h and the false alarm rate p f were calculated. The hit rate corresponds to the different responses percentage when the two surfaces of the pair had a different curvature, while the false alarm rate is the percentage of different responses for equal surfaces. The rates are converted to z-score of the normal distribution and the sensitivity measure d is calculated as the difference between the two values: d = Z h Z f (1) The Just Noticeable Difference (JND) was identified as the difference between curvatures for which d was equal to 1, according to the criterion most commonly adopted in literature. Three series were presented to each participant, changing the value of the difference between the two possible curvatures, according to the values reported in table 1. For each Δ of curvature, the sensitivity measure d was obtained and the three points were linearly interpolated, as shown in the plot of figure 8 for one subject in both the two conditions. Finally the JND was calculated from the interpolating function, defining the overall JND is defined as the mean of the participants JNDs. The JND values resulted 1.51 m 1 in1 in modality A and 2.62 m 1 in modality B, with a statistically significant difference (p <0.05). For each subject the improvement in 9

10 Figure 8: Experimental results for one subject: modality A (red upper line) and B (blue lower line). curvature discrimination using the new device was evident. This allows us to conclude that an enhancement of haptic perception of shape cues can be reached by complementing pure kinesthetic feedback with haptic cues, applied locally at the fingertip, informative of the contact geometry, and that this mechanism appears to be a fundamental and physiological component of haptic perception of real shapes. 3. Virtually altered force feedback to generate illusions in fingertips exploration. Another alternative approach to enhance haptic perception is to adopt haptic rendering algorithms that can generate suitable haptic illusions during active exploration. In fact, the level of sensation provided with kinesthetic haptic devices below fingers is realistic, but still far to be exact: an important open issue that occurs then (Basdogan and Srinivasan, 2002; Hayward et al., 2004) is to understand to which extent such system can be employed for real tasks where high sensitivity is needed, such as the ability to render the force feedback of thin objects such a s needles and, in general, of objects having spatially sharp features. As matter of fact, during haptic interaction some information is lost due to bandwidth and stiffness limitation of the mechanical devices, moreover, as smaller the device and its mechanical parts are, the softer it result. It has been found that force feedback information alone can elicit complex perception relating to haptic texture and shape (Minsky, 1995; Morgenbesser and Srinivasan, 1996; Ho et al., 2004). Such perceptions can be considered as haptic perceptual illusion of texture and shape. In what follows we report available results on the ability of using haptic effects to render the sensation of sharp smaller objects. 10

11 Figure 9: Selected shape profiles. 3.1 Experiment design The reduced reliability of sensations produced by haptics was already proven in the past. For instance, the use of lateral forces (Robles-De-La-Torre and Hayward, 2000, 2001) already showed the existence of illusory effects that may be generated on fingertips. In order to understand the level of human perception sensitivity in fine exploration tasks, a haptic to vision matching design was employed. The experiment was designed in the following way. A kinesthetic haptic interface was employed. The users placed the right index finger in the device thimble in order to touch the 3D virtual surface. Hence the user may move his/her finger all over a virtual surface modeled in software. During such a motion he/she may perceive geometric features (such as corners, edges, curvatures,... ). During the experiment the user was asked to distinguish among sizes, distances and spatial relationships among elements. Additionally during the experiment subjects were instructed to maintain their finger inserted into the thimble all over the time and to keep the same standing orientation, subject had to explore the virtual surface sideways (left to right). This instruction allowed them to maintain the properties of the exploration independent from the feature position in the space, while preventing accidental contact with other parts of the mechanical device. Subjects explored the surface at their own pace without any time constraint. Each exploration had to be performed with closed eyes. During the experiment a series of haptic surfaces was presented. The surfaces, having different shapes, were presented in a randomized order. Each presentation defined a trial and the shape of the surface was changed between different trials. When subjects finished haptically exploring each shape,they matched the haptic shape to a visually displayed profile. A set of shape profiles was visually shown to subjects on a computer screen placed in front of the subject, as shown in figure 9. Each profile had a number. Subjects used a computer keyboard to enter the number of the profile that they believed to be closest to the profile of the shape that was haptically explored. If they were not completely certain about the shape, subjects were instructed to give their best guess. The following variables were saved during the experiment: trajectory of device thimble during exploration, trial explo- 11

12 Haptic Shapes Matched Visual Shape (% of all subjects) Sine Sine and small Sine and small Saw gaussian Saw 1. SineSeg (Profile 1) SineLFGauss (Profile ) 3. Saw (Profile 4) SineSaw (Profile 3) SineGauss (Profile 2) Table 2: Average Haptic to Visual matching for all subjects. The highlighted cells indicate the visual shapes to which the haptic shapes should be ideally matched. For each haptic shape, the difference in matching performance is statistically significant (ANOVA, p<0.01) ration time, haptic shape presented and visual shape matched. A test consisted of 90 trials, and typically lasted for 25 minutes. Ten right-handed subjects ages participated in the experiment. All of them had previous experience with haptic interface but were navice as to the purpose of the analysis carried out. Five different haptic surfaces were used in this experiment: 1) a sinusoidal segment, 2) a sinusoidal segment with a lateral-force-based (Robles-De-La-Torre and Hayward, 2000, 2001) illusory Gaussian shape, 3) a sawtooth segment, 4) a sinusoidal segment with a small sawtooth bump, and 5) a sinusoidal segment with a small Gaussian bump. The force-feedback and geometrical features of each surface are described below. A surface was rendered with lateral (Fx, along the x-axis) and vertical forces (Fy, along y-axis). The vertical forces were served to maintain the vertical position of the haptic manipulandum as close as possible to the target geometry. The virtual surfaces were displayed within a 310mm workspace defined along the x-axis. The center of the feature (c) was randomly placed and changed among trials. All features were 10mm large and 3mm high. The haptic control loop was performed at high frequency (2.5KHz). The mathematical definition of the shapes was described in (Portillo et al., 2005). Table 2 shows subjects performance in the haptics-to-vision matching task. The table relates the frequency with which a given haptic surface was matched to one of the visual profiles described before. This frequency is expressed as a percent of the overall matching performance for all subjects. Subjects consistently matched the haptic surface SineSeg to the sinusoidal shape (Row 1, Column 1). However, subjects matching performance was completely different when exploring SineLFGauss (Row 2, Column 2). Note how SineSeg and SineLFGauss had the same geometry. It is striking how the Saw haptic surface was very frequently matched to the sinusoidal shape segment (Row 3, Column 1). But the converse was not true: the haptic sinusoidal segment (SineSeg) was rarely matched to the sawtooth segment (Row 1, Column 4). This helps highlight the difficulties of consistently rendering a good sawtooth shape by using a 12

13 Figure 10: An user s fingertip trajectory while exploring a SineSaw shape (Subject 1, trial 34) and the trajectory data corrected for vertical offset and ready for MSE computation. literal approach. Even though the stimulus had an approximation to a real, sharp edge, the results suggest that subjects did not perceive an object with a sharp feature. This contrasts with subjects matching performance for SineLFGauss (Row 2). This haptic surface was rarely confused with the sinusoid. It was sometimes classified into different categories by some subjects, but overall it was mostly matched to the sinusoidal surface with a small Gaussian bump (figure 9, Profile 2) and to the sinusoidal surface with a small sawtooth bump (figure 9, Profile 3). This suggests that SineLFGauss was perceived by subjects as a sinusoidal segment with a sharp feature (figure 9, Profiles 2 and 3), rather than as a large sawtooth shape (such as the one in figure 9, Profile 4). In contrast to the matching performance for the haptic Saw (row 3), haptic surfaces SineSaw and SineGauss were rarely matched (4.4 % and 6.1 % respectively) to the visual sinusoidal segment (Column 1, rows 4 and 5). This suggests that the perception of sharp features depends on the context in which they are presented. For example, when exploring haptic shapes SineLFGauss, SineSaw and SineGauss, there was a decrease in force as the top of the sinusoidal position of the stimuli was reached, and then the sharp feature or the Gaussian lateral force provided a large increase in force. Compare this to the constant forces experienced when ascending/descending the slopes of Saw (see Haptic Shapes in Methods). This may not be surprising from the perceptual point of view, but it is potentially useful for haptic rendering purposes. Finally, subjects consistently matched SineSaw and SineGauss to the equivalent visual shapes: 68.9 % and 60 % of the time, respectively. Subjects finger trajectories during object exploration were compared to the ideal geometries of each haptic surface through Mean Squared Errors (MSEs). This allowed examining how close subjects finger trajectories were to the geometry of each stimulus. Figure 10 shows a typical trajectory when exploring a SineSaw stimulus. Subjects finger trajectories had a slight vertical offset relative to the ideal geometry of the objects. This was due to the finite stiffness used (which in this experiment was set to 2 N/mm). The offset was eliminated before computing MSEs. MSEs were calculated within the range x (c 0.025m, c m). MSE analysis is summarized in table 3. Results are expressed as a percent of the trials in which a given trajectory was closest to the geometry of a haptic shape. For example, 13

14 Haptic Shapes Explored Closest geometry (% of total trials for each haptic shape) 1. SineSeg 2. Saw 3. SineSaw 4.SineGauss 1. SineSeg SineLFGauss Saw SineSaw SineGauss Table 3: Comparison of the exploration trajectories used by subjects to the geometry of the haptic shapes. Note that SineLFGauss stimuli had the same geometry as the Sinusoidal segment (SineSeg). For each haptic shape, the difference in the trajectories followed is statistically significant (ANOVA, p<0.01). consider all the trials in which SineSeg was explored by subjects (row 1). The table shows that in 98.33% of those trials, subjects finger trajectories were closest to the ideal SineSeg geometry. The table also indicates that, very rarely, subjects finger trajectories were closest to the ideal geometries of Saw, SineSaw or SineGauss. Table 3 helps to assess the difficulty to render the geometry of some sharp features. While rendering the geometry of SineSeg and Saw was simple, this was not the case for SineSaw and SineGauss (rows 4 and 5). Subjects finger trajectory when exploring SineSaw and SineGauss did not, in general, closely follow the target geometry of these surfaces. In contrast, rendering the geometry of SineLFGauss was possible (row 2). As mentioned above, SineLFGauss had the same geometry as SineSeg (a smooth sinusoid), but was consistently matched to visual shapes with sharp features. The difficulties to render the geometry of SineSaw and SineGauss suggest that maintaining surface contact with these objects during exploration was made difficult by the surfaces sharp features, which is something commonly observed when rendering these in general. In contrast, table 3 suggests that a good surface contact with SineLFGauss (the surface with an illusory Gaussian bump) was simply achieved. Two major points are suggested by the second table also. The first is that the geometry of Saw was accurately rendered (table 3, Row 3, Column 2), but this did not result in subjects consistently matching this haptic surface to the visual Sawtooth profile (table 2, Row 3, Column 4). In contrast, the haptic Saw was consistently matched to the sinusoidal visual profile (table 2, Row 3, Column 1). This suggests that accurate rendering of the geometry of sharp objects does not always result in subjects perceiving the sharp features of objects. The second major point is that, even when the geometry is not very accurately rendered (perhaps due to unstable contact with the surface of the object due to spatially sharp features), subjects still could be able to perform an accurate haptic to visual match. This is suggested by the matching performance for SineSaw (table 2, Row 4, Column 3). The results suggest that lateral-force-based haptic shape illusions (such as the one used in SineLFGauss) can be combined with a smooth object geometry to haptically render sharp features of objects. The results also suggest that this can be achieved while maintaining a stable contact with the object, which is what happens, in general, during haptic interaction 14

15 with real objects with spatially sharp features. It is not possible to stress enough the desirability of such stable interaction with haptically rendered, spatially sharp objects, particularly in applications such as surgery simulators. However, the variability found during haptic to visual matching for SineLFGauss suggests that there are more factors determining subject perception in these cases. For example, subjects may have had a bias toward choosing one of the three visual figures, or perhaps subjects expectations regarding the haptic features of stimuli were modulated by the visual shapes. More research is needed to clarify these possibilities. 4. Vibrotactile haptic feedback in manipulation and exploration in virtual environments Vibro-tactile feedback cues can significantly enhance touch perception for virtual environment applications with minimal design complexity and cost (Okamura et al., 1998). From a developmental psychological perspective, touch plays an essential role in our perceptual construction of spatial environmental layout. Combining touch and vision allows the simultaneous extraction of perceptive process invariants, crucial for establishing the reciprocal connections that allow for higher order perception and categorization of objects and environments (Haans and IJsselsteijn, 2006). To date, there are several types of data-glove commercially available (Sama et al., 2006; Kessler et al., 1995; Tarasewich, 2002), but not so many researchers (Murray et al., 2003) or ( 1993) have integrated these glove with prototypical vibro-tactile pads. For the most of commercial Data-gloves previously cited, quantitative assessment of rigid range of motion (RoM) is required and a measuring procedure must be done (Simone and Kamper, 2005) and (Wise et al., 1990). The Data-glove application presented in this section represent a unique combination of absolute goniometric and tactile stimulators (See Figure 11). The Data glove is based on Hall Effect goniometric sensors (Salsedo et al., 2004) and a specific mechanical design that allows the device to be dressed by anyone without requiring a specific pre-calibration procedure. The prototype can track any human gesture (grasping simple objects, touching surface,... ) in order to perform psychophysics experiments and rehabilitation procedures. The data-glove is equipped with at least two sensors of different length for each finger; The vibro-tactile actuators are displaced on the back of each fingertip to stimulate the human cutaneous receptors when the virtual hand comes in contact with an object in the virtual environment.they measure the angular displacement of proximal (MCP) and medial phalanxes (PIP) with respect to the back of the hand. The difference between the two signals can be implemented via software in order to obtain the relative angular displacement between the two phalanxes. The adduction-abduction movement of each MCP finger is not measured, because it has not to be needed to perform the main gestures useful for the foreseen applications. A third sensor has been added to the thumb; it bends in a plane normal to the flexo-extension of the other two sensors. 4.1 Goniometric Sensor The working principle of the goniometric sensor relies on the fact that a flexible beam having a deformed elastic line lying on a plane has the property that the longitudinal elongation 15

16 Figure 11: Hand joints representation and PERCRO data glove Figure 12: Main components of the goniometric sensor. of the fibers depends linearly on their curvature and their distance from the neutral axis of the beam. The total elongation of the fiber is a function of the angle between the two endpoints of the flexible beam and it is independent from its specific elastic line. Hall effect sensor measures the intensity of a magnetic field produced by a magnet attached to the movable end of the wire. The goniometric sensor is composed of only four parts: a commercial cylindrical permanent magnet, a commercial miniaturized Hall Effect sensor with a built-in signal amplifier, a multi wire flexible steel cable; a flexible thin beam made of plastic with a square cross section a longitudinal hole. The beam ends with a casing for the magnet and the Hall effect sensor (Salsedo et al., 2004). The sensor (see figure 12) is composed of a transducing bulb and a sensing flexing bar. When a relative bending angle dθ to a beam element of length dl is imposed, the length of the neutral fiber of the beam remains unchanged while the fiber, positioned at a distance e from the neutral axis, changes its length by a quantity of: dl = e dθ (2) 16

17 Figure 13: Calibration performances (Tension vs angle) Integrating this variation along the entire beam, in case of constant e, we obtain: L = L 0 dl = e Δϑ 0 dϑ = e Δϑ Therefore, the total elongation of the fiber is a function of the angle between the two endpoints of the flexible beam and it is independent from its specific elastic line. One time calibration and test have shown high and reliable stability of sensor measurements 13 both in terms of accuracy and performances: 180 o range of motion, ±1.5 o degree of accuracy (worst condition) 4.2 Vibromechanical stimulator Neural mechanisms that involves the sensation of touch have been studied extensively from many years ago. These studies demonstrate the sensory capacity of a human by functional proprieties of the sense organs in the skin, rather by mechanisms within the central nervous system (Johansson, 1978). The tactile units in the skin area of the human hand are of four of different types: two fast adapting, RA (Meissner corpuscles) and PC (Pacinian corpuscles), and two slowly adapting types (Merkel cells). The slowly adapting are sensitive to low frequency stimulation (<10Hz) and primarily encode pressure, texture and form of the object. The Meissner corpuscles are most sensitive to vibro-tactile frequencies of 30 Hz and response to the flutter, slip, and motion of objects. The Pacinian corpuscles are most sensitive to high frequency vibration centered around 200 Hz. 17

18 Figure 14: Vibro-mechanical actuators and human hand receptors In order to stimulate the Pacinian corpuscles vibrating motors have been attached on the palm-side of each fingertip. These actuators consist of small motors commonly used as vibrational alarms in pagers, mobile phones and many vibro-tactile game controllers, which can be made to rotate at different speeds, and so different frequencies. Vibration intensity is controlled varying the voltage through a PWM command between 0 and 5V. Due to the voltage constant of such actuators, at the maximum voltage command the motors, with an eccentric mass (0.16gr) mounted on its shaft, the frequency reaches a theoretical value of about 500 Hz. Acquisition and control interface The hardware control architecture acquires the analog signals from the goniometric sensors, and convert them to digital signals, then send the digital data to a host computer via a serial communication protocol. Two types of microcontrollers (μc) PIC18LF4420 and PIC18LF443 have been used to develop a master-slave mini-network; each of them operates with an external oscillator of 40 Mhz. 11 analog inputs are used to read the input signals from the goniometric sensors using 12 bits ADCs. The communication between the μc and the host computer is performed through RS232 interface at 115,200 bps. The communication among the master and slaves is based on the Serial Peripheral Interface (SPI) that guarantees 12MBit/s. Each slave can activate four motors with different PWM signals trough a Darlington array. Rehabilitation test scenario A test scenario which includes a whole hand avatar has been implemented. The environment includes a dining table, plates and kitchen utensils. A rehabilitation main task (to move and pick-up the objects around the table) was designed to estimate how user feels the tactile sensation made by hand-object contact. 18

19 Figure 15: Data-glove electronic architecture. Figure 16: Virtual hand and kitchen environment (Courtesy of Graphics and Hypermedia Lab at the University of Cyprus) A graphic representation of the hand and the kitchen environment was developed and implemented through a virtual 3D platform called XVR (Carrozzino et al., 2005; Tecchia, 2006). To allow the use sweep the VE, a 6DoF magnetic tracker by Polhemus has been embedded on the data-glove. Such a sensor permits to acquire the position and the orientation of the user s hand respect to a physical receiver. For this application, the angular positions of the skeletal joints are associated to the obtained angles, and then 3D hand meshes are deformed applying geometrical transformations to the point of the mesh associated to the corresponding point of the skeletal, as shown in the application example of figure

20 The integration with Physics AGEIA has been performed for providing a more realistic dynamic interaction with the virtual environment and for relaying back to the user information about the contact of the virtual hand fingers against the objects. References CA Avizzano, S. Marcheschi, M. Angerilli, M. Fontana, M. Bergamasco, T. Gutierrez, and M. Mannegeis. A multi-finger haptic interface for visually impaired people. In Robot and Human Interactive Communication, pages , P. Jenmalm A.W. Goodwin and R.S. Johansson. Control of grip force when tilting objects: Effect of curvature of grasped surfaces and applied tangential torque. The Journal of Neuroscience, 18(24): , F. Barbagli, R. DeVengenzo, and K. Salisbury. Dual-handed virtual grasping. In International Conference on Robotics and Automation, ICRA, volume 1, pages , F. Barbagli, A. Frisoli, K. Salisbury, and M. Bergamasco. Simulating human fingers: a soft finger proxy model and algorithm. In 12th International Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, pages 9 17, 2004a. F. Barbagli, K. Salisbury, and R. Devengenzo. Toward virtual manipulation: from one point of contact to four. Sensor Review, 24(1), 2004b. C. Basdogan and M.A. Srinivasan. Haptic Rendering in Virtual Environments. Lawrence Erlbaum Associates, M. Carrozzino, F. Tecchia, S. Bacinelli, C. Cappelletti, and M. Bergamasco. Lowering the development time of multimodal interactive application: the real-life experience of the xvr project. In Proceedings of the 2005 ACM SIGCHI International Conference on Advances in computer entertainment technology, pages ACM Press New York, NY, USA, G. Cini, A. Frisoli, S. Marcheschi, F. Salsedo, and M. Bergamasco. A novel fingertip haptic device for display of local contact geometry. In Haptic Interfaces for Virtual Environment and Teleoperator Systems, WHC First Joint Eurohaptics Conference and Symposium on, pages , H. Dostmohamed and V. Hayward. Trajectory of contact region on the fingerpad gives the illusion of haptic shape. Experimental Brain Research, 164(3): , S. A. Douglas and A. E. Kirkpatrick. Application based evaluation of haptic interfaces. In Proceedings of the Tenth Symposium on haptic interfaces for virtual environment and teleoperator systems, A. Frisoli, F. Barbagli, S-L. Wu, E. Ruffaldi, and M. Bergamasco. Comparison of multipoint contact interfaces in haptic perception. In Workshop on Multipoint Interaction in Robotics and Virtual Reality. International Conference on Robotics and Automation,

21 A. Frisoli, M. Bergamasco, S.L. Wu, and E. Ruffaldi. Multi-point interaction with real and virtual objects. Springer tracts in advanced robotics, volume 18, chapter Evaluation of Multipoint Contact Interfaces in Haptic Perception of Shapes, pages Advanced Robotics, A. Haans and W. IJsselsteijn. Mediated social touch: a review of current research and future directions. In Virtual Reality, volume 9, pages Springer, V. Hayward, O.R. Astley, M. Cruz-Hernandez, D. Grant, and G. Robles-De-La-Torre. Haptic interfaces and devices. Sensor Review, 24(1):16 29, PP Ho, BD Adelstein, and H. Kazerooni. Judging 2d versus 3d square-wave virtual gratings. In Haptic Interfaces for Virtual Environment and Teleoperator Systems, HAPTICS 04. Proceedings. 12th International Symposium on, pages , Cybertouch. Technical report, Immersion, G. Jansson. Effects of number of fingers involved in exploration on haptic identification of objects. excerpt from pure-form: The museum of pure form; haptic exploration for perception of the shape of virtual objects. Technical report, Technical report, EU-PURE FORM, G. Jansson and L. Monaci. Touch, Blindness and Neuroscience, chapter Haptic identification of objects with different numbers of fingers. UNED Press, Madrid, G. Jansson and L. Monaci. Identification of real objects under conditions similar to those in haptic displays: providing spatially distributed information at the contact areas is more important than increasing the number of areas. Virtual Reality, 9(4): , G. Jansson, M. Bergamasco, and A. Frisoli. A new option for the visually impaired to experience 3d art at museums: manual exploration of virtual copies. Visual Impairment Research, 5(1):1 12, VI Johannes and M. Green. Role of the rate of application of the tangential force in determining the static friction coefficient. volume 24(3), pages , RS Johansson. Tactile sensibility in the human hand: receptive field characteristics of mechanoreceptive units in the glabrous skin area. The Journal of Physiology, 281(1): , R.S. Johansson and G. Westling. Roles of glabrous skin receptors and sensorimotor memory in automatic control of precision grip when lifting rougher or more slippery objects. Experimental Brain Research, 56: , G.D. Kessler, L.F. Hodges, and N. Walker. Evaluation of the cyberglove as a whole-hand input device. ACM Transactions on Computer-Human Interaction (TOCHI), 2(4): ,