Flexible Active Touch Using 2.5D Display Generating Tactile and Force Sensations

This is the accepted version of the following article: ICIC Express Letters 6(12):2995-3000 January 2012, which has been published in final form at http://www.ijicic.org/el-6(12).htm Flexible Active Touch Using 2.5D Display Generating Tactile and Force Sensations Satoshi Tsuboi 1 and Masahiro Ohka 1 1 Graduate School of Information Science Nagoya University Chikusa-ku, Furo-cho, Nagoya, 464-8601, Japan tsuboi.satoshi@h.mbox.nagoya-u.ac.jp Received August 2011; accepted November 2011 ABSTRACT. We are developing a multi-modal 2.5D display capable of stimulating tactile and force sensations to achieve a new human-friendly interface. In the present study, we extended the versatility of the operation in 2.5D space for object manipulation in 3D space. Although the surface rotation of the virtual tactile pad was limited around the z-axis in the previous display, the rotations around the x-axis and y-axis are managed by the centroid position of the operator s fingers in the new display. We conducted three series of evaluation experiments. In Experiment I, we confirmed that the centroid estimation was stable around the center of the display pad. In Experiment II, we revealed the proper proportionality constant for the vertical movement of the virtual tactile pad. In Experiment III, subjects tried to compare the size of two virtual spheres to evaluate the 3D shape presentation capability of the present display. Since the sensation thresholds for each subject were low enough, we proved that operators could recognize a fine distinction of sphere radii generated by the proposed display. Keywords: Haptics, Virtual reality, 2.5D display, Tactile sensation, Object handling 1. Introduction. In the realm of virtual reality (VR) and tele-existence, it is effective to present tactile and force sensations in addition to visual and auditory information for object manipulation because this concerns the tactile sensation and reaction force caused by touching objects. Thus, tactile and force displays have attracted the attention of researchers. So far, they have developed several types of haptic displays such as mouse-type [1, 2], encounter-type [3, 4], wearable type [5-7], and grip-type [7, 8] haptic devices. Although the abovementioned experimental haptic devices are investigated further, we can use some conventional 3D haptic devices such as the PHANToM series [9]. However, even if they are commercial devices, they have a problem: operators grow tired since they have to maintain the same vertical movement in 3D space. Without the vertical movement, it is possible to reduce the consumption energy of the operator. Accordingly, the motivation of this study is to enable the operator to work comfortably in VR. In preceding papers [10, 11], we have developed an advanced multi-modal 2.5D display for tactile and force sensation to develop a new human-friendly interface for various fields

such as virtual reality and tele-existence. The display is comprised of a master arm with a tactile display with a 4-by-12 array of stimulus pins driven by micro-actuators and an articulated manipulator. We arranged the handle of the display like a mouse so that anyone can comfortably use it without a priori knowledge. In addition, the tactile display has a relatively large display pad, and an operator can naturally touch an object with three fingers. Furthermore, we suggested a new concept of virtual 2.5D space to reduce fatigue while 3D manipulations for long term tasks, where we achieve virtual quasi-3d space by adding vertical pointing controlled by compressive force to 2D spacing. In the present display, vertical movement of the virtual tactile pad in the virtual world is determined in proportion to compressive force applied on the display pad with reference to TrackPoint [12], so that operators can manipulate the virtual tactile pad with minimum force. In addition, since operators hands are supported by the manipulator in the vertical direction, they don t get tired during manipulations in 3D space. Moreover, we developed a presentation system to assist us for building virtual blocks in the appropriate position. Then, we performed a series of experiments and verified that the alignment precision is improved for pressure and force presentation. In the present study, we focus on increasing the efficiency of the proposed display for tactile recognition to advance the previous study. This study includes three objectives: the first is to extend the versatility of the operation in 2.5D space for object manipulation in 3D space; the second is to discover the appropriate parameter for the vertical movement of the virtual tactile pad; the third is to verify the presentation capability of the proposed display. In the previous display, since the surface rotation of the virtual tactile pad was limited around the z-axis, the contact region during object exploration was restricted within a plane. Therefore, we propose that the rotations around the x- and y-axes are managed by the centroid position of the operator s fingers. In the present display, compressive forces applied on the display pad can be obtained from three pressure sensors installed under the actuator array. Operators can perceive the subtle curvature and surface information of virtual objects because they can control the contact region of the virtual tactile pad flexibly while touching objects in virtual space. On the other hand, vertical movement of the virtual tactile pad is controlled in proportion to compressive resultant force with respect to the previous display. Consequently, operators can perform flexible works in 3D space in spite of their manipulations in 2.5D space. To evaluate our display, we conducted three evaluation experiments. In Experiment I, we verified the measurement precision of the centroid estimation. To ensure reliable operation for the contact region of the virtual tactile pad, it is necessary to confirm the stability of estimation. In Experiment II, subjects tried to compare the heights of two virtual rectangular solids. This experiment is conducted in order to determine the appropriate proportionality constant for the vertical movement of the virtual tactile pad. If operators can distinguish the height of a virtual object, they can perform intuitive works in VR. In Experiment III, subjects judged the diameters of two spheres to prove the recognition capability for virtual objects by using the proposed display. 2. Display System. 2.1 Haptic Display. Figure 1 shows the 2.5D display generating combined tactile and force

sensations developed on the basis of the previous study. The haptic display consists of a 3-link planar manipulator used as a force display and a tactile display on the handle of the manipulator. The tactile display is composed of a 4-by-12 array of stimulus pins driven by piezoelectric bimorph actuators of a Braille cell [13] (SC9, KGS, Co.) as shown in Fig. 2. The stimulus pins perform a protruding motion between intermediate values from 0 to 1 mm. An operator can naturally touch an object with three fingers because the tactile display has a relatively large display pad. In addition, since we arranged the handle of the manipulator like a mouse, anyone can use it comfortably without a priori knowledge (Fig. 3). The position of the virtual tactile pad is controlled by the following procedure. The horizontal position is fixed with respect to the display pad in the actual display. The horizontal information is obtained from kinematics since the configuration of the manipulator is determined by the joint angles. On the other hand, although the movement of the manipulator is planar, the vertical movement of the virtual tactile pad in the virtual space is regulated by compressive resultant force applied on the display pad. In this study, three pressure sensors are installed under the display pad as depicted in Figs. 4 and 5 even though one pressure sensor was installed in the previous display. Thus, we can use this device as the pointing device in 3D space despite the manipulations in 2.5D space. Since operators hands are supported by the manipulator, they can work as if they have put on their hands on a hand rest. Therefore, they don t get tired during manipulations in 3D-space. In virtual space, the contact interaction between the fingertips of the operator and the virtual object is calculated to obtain reaction force and distributed pressure. The force and torques are computed from statics and physical models. 2.2 Altitude Control. The vertical movement of the virtual tactile pad is controlled in proportion to compressive resultant forces applied on the display pad with reference to the previous display. Figure 5 describes the principle of the technique for the vertical position control. In the initial condition, the virtual tactile pad is put in the air apart from the virtual objects, maintaining a default distance (Fig. 5a). Operators can manipulate this display as if their fingertips work through the tactile pad, while they grasp the end effector designed like a mouse and put their three fingers (index, middle, and ring finger) on the display pad (Fig. 3). If they increase contact force of the fingertips by shifting part of their weight to the fingertips for vertical direction, they can move the virtual tactile pad in the vertical direction. The vertical movement is controlled as it is in proportion to the compressive resultant force obtained from the three pressure sensors (Fig. 5b). Then, operators can perceive the tactile information on surface unevenness of the virtual objects according to pin-protrusion change. Thus, they can recognize the movement of virtual tactile pad in 3D space although the real display pad remains static. This technique is similar in principle to pseudo-haptics feedback [14, 15]. Although the abovementioned pseudo-haptics is obtained by using only visual feedback, in our display, the distributed pressure is presented by stimulus pins according to applied force instead of visual feedback. Consequently, operators can feel not only the height of the object but also surface unevenness with minimum force.

2.3 Orientation Control. For tactile recognition of objects, many researchers have studied active and passive touch [16-18]. Active touch is defined such that human subjects actively feel the object surface with their fingers. On the other hand, passive touch is defined such that human subjects remain stationary and a stimulus is applied to their fingers. Active touch is effective for tactile scanning since we perceive the external environment by touching actively. However, in our previous display, operators couldn t manipulate the virtual tactile pad flexibly because the surface rotation of the virtual tactile pad was limited around the z-axis. The contact region during object exploration was restricted in the plane. In the present display, the rotations of the virtual tactile pad around the x- and y- axes are managed by the centroid position of the operator s fingers. Consequently, operators can perceive the curvature and surface information of virtual objects because they can flexibly manipulate the contact region of the virtual tactile pad while touching objects in virtual space. In the present display, compressive forces applied on the display pad can be obtained from three pressure sensors installed under the actuator array as shown in Fig. 6. Three pressure sensors are arranged on a circle whose central position is the same as the center of the display pad. Then, the centroid position x T G, y G is calculated as: 3 i i i i T i1 i1 xg, yg, (1) 3 3 Fi F i i1 i1 where x T i, y i and F i are the position of pressure sensors and contact forces applied on each sensor. Figure 7 shows the principle of the orientation control of the virtual tactile pad. The surface rotations of the virtual tactile pad around the x- and y-axes are controlled by the following procedure. In the initial condition, the virtual tactile pad remains horizontal to the x-y plane. If operators increase contact force of the fingertips and shift their center of gravity toward the horizontal direction, the centroid position is moving. Because of the distance between the center of the display pad and the centroid position, the rotation around the x-axis and y-axis are regulated by the distance for the y-axis and x-axis directions, respectively. Thus, the orientation of the virtual tactile pad is obtained from the set of angles consisting of roll, pitch, and yaw, which is widely used in robotics to describe rigid body orientation. Figure 8 3 T F x F y illustrates the procedure for the rotations. First, the coordinate frame O xyz is rotated about the z axis by the yaw angle (Figure 8a). Secondly, the new coordinate frame O xyz is rotated about the y axis by the pitch angle (Figure 8b). Finally, the newest coordinate frame O x yz is then rotated about the x axis by the roll angle. The resultant coordinate frame O xb yb zb is depicted in Figure 8c. As a result, the rotation matrix R is computed as:

where CC SC C S S S S C S C R SC CC S S S C S S S C (2) S C S C C C cos, C cos, C cos (3) S sin, S sin, S sin (4) Although we assumed that the virtual tactile pad rotates 2.0 degrees counterclockwise per 1 mm for roll and pitch angle in this paper, this angle rate is easily adjusted according to objectives. Consequently, operators can perform flexible touching easily because operators just shift the direction of contact force of the fingertips in order to manipulate the surface rotation of the virtual tactile pad. 3. Experimental Procedure. 3.1 Verification of Centroid Estimation (Experiment I). To ensure reliable operation for the contact region of the virtual tactile pad, we performed an evaluation experiment to verify the stability of centroid estimation. In the preliminary experiment, we verified that the centroid position of fingers could be guessed as long as operators put their fingers on the display pad. In Experiment I, we compared measurement values of centroid estimation with the exact position on the actual display pad. We added vertical compressive force on the stimulus pins of the tactile display with a stylus as shown in Figure 9. The target pins are the first line and sixth column. Then, we measured the position of the centroid estimation. We repeated this five times for measurement of each point. 3.2 Height Discrimination (Experiment II). For practical use of the proposed display, it is necessary to define the appropriate proportionality constant for the vertical movement of the virtual tactile pad. If the proper one is obtained, operators can perform intuitive works in VR. In addition, we can suggest the optimal design method for manipulation with a 2.5D display. In Experiment II, we intended to examine the height felt by the operator based on the following procedure. 1) The virtual tactile pad is put in the air apart from the ground, keeping a default distance of 16 mm as depicted in Fig. 5a. 2) If the operators increase contact force of the fingertips by shifting part of their weight to the fingertips for vertical direction, they can move the virtual tactile pad in the vertical direction. 3) The proportionality constant for the vertical movement of the virtual tactile pad is changed during this experiment. The variations of proportionality constants are 2.5, 3.5, 5.0, 10.0, and 20.0 mm per 1.0 N. 4) The operator examines the height of the standard rectangular solid, which is 20 mm in width, 20 mm in thickness, and 8 mm in height. 5) The operator inspects the height of comparison rectangular solids, which are 2, 4, 6, 10, 12, and 14 mm in height. Figure 11a and b depict the standard object and the lowest comparison object.

6) The operator tries to compare the heights of two virtual rectangular solids. 7) The contact interaction between the fingertips of the operator and the virtual rectangular solid is calculated to obtain distributed pressure (Figure 11c). 8) The tactile information on surface unevenness of virtual rectangular solids is presented by the tactile display as shown in Figure 5b. To define the appropriate proportionality constant for the vertical movement of the virtual tactile pad, we performed a series of experiments with five human subjects (right-handed, average age: 24), and they compared the heights of two virtual rectangular solids. The positions of stimulus pins in virtual world correspond with actual pins as depicted in Fig. 10. Stimulus pins protrude in proportion to the normal component of displacement caused by contacts between the virtual tactile pad and the virtual rectangular solid. Operators perceive the height of virtual rectangular solids according to pin-protrusion change. Procedures (1) (8) were repeated 300 times by using a constant stimuli method for each subject without visibility. 3.3 Sphere Diameter Discrimination (Experiment III). We adopted a psychological experiment to evaluate the presentation capability of the present display by using a constant stimuli method. In this experiment, a sensation threshold corresponding to sensitivity was obtained. If the threshold was low enough, an operator could recognize a fine distinction of stimuli generated by the display. In this experiment, we intended to investigate the size felt by the operator based on the following procedure. 1) The virtual tactile pad is put in the air, keeping a default distance 5 mm from the sphere (Fig. 12a). 2) If the operators increase contact force of the fingertips by shifting part of their weight to the fingertips for vertical direction, they move down 3.5 mm per 1.0 N. 3) The operator examines the size of the standard sphere, which has a diameter of 30 mm. 4) The operator inspects the size of comparison spheres with diameters of 24, 26, 28, 32, 34, and 36 mm. 5) The operator tries to compare the size of two virtual spheres. 6) The contact interaction between the fingertips of the operator and the virtual sphere is calculated to obtain distributed pressure (Fig. 12b). 7) The tactile information about the curvature and the surface information of the virtual sphere are presented by the tactile display according to pin-protrusion change. To confirm the effectiveness of the proposed display for tactile recognition, we conducted a series of experiments with the same subjects as Experiment II, and they compared the size of the standard and comparison virtual spheres. The positions of stimulus pins in virtual world correspond to actual pins in common with Experiment II. Stimulus pins protrude in proportion to the normal component of displacement caused by contacts between the virtual tactile pad and the virtual sphere (Fig. 13). Since operators can control the contact region of the virtual tactile pad flexibly while touching a sphere in virtual space as described in Fig. 7, they minutely perceive the curvature and the surface information of the virtual sphere by the tactile display according to pin-protrusion change. Procedures (1) (7) were repeated 60 times for each subject in a no-visibility situation.

4. Experimental Results and Discussion. Here are the main results in this paper... 4.1 Verification of Centroid Estimation (Experiment I). To verify the stability of centroid estimation, the positions were measured for each target point. Figure 14 depicts the averages of measurement positions for each point. The errors between the measured position and the target position were reduced toward the center of the display pad. On the other hand, the estimation positions were deviated from the accurate positions with increasing distance from the center of the display pad. We guess that it is difficult to tilt the display pad greatly by compressive force because the display pad is fixed on the haptic display. However, since operators put three fingers on the display pad while touching objects, it is seldom that their centroid position moves extensively away from the center of the display pad. From these experimental results, we confirmed that the centroid estimation is stable around the center of the display pad. Consequently, since the contact region of the virtual tactile pad is naturally changed by the operators, they can perform intuitive manipulations in virtual space. 4.2 Height Discrimination (Experiment II). The differential limens (DLs) for each subject were measured to obtain the appropriate proportionality constant for vertical movement of the virtual tactile pad. Figure 15 shows the relationship between the proportionality constant and DL for each subject. The smallest DLs for four subjects (subjects A, C, D, and E) were obtained when the proportionality constant was 3.5 mm per 1.0 N. On the other hand, the smallest DL for subject B was measured when the proportionality constant was 2.5 mm per 1.0 N. However, the DL for subject B was also low enough when the proportionality constant was 3.5 mm per 1.0 N. From these results, it is obvious that DL is going to converge at a low level of about 1.0 mm with the decreasing of the proportionality constant. Thus, we can adopt the proper proportionality constant for the vertical movement of the virtual tactile pad as 3.5 mm per 1.0 N. 4.3 Sphere Diameter Discrimination (Experiment III). Presentation capability is examined using the diameter discrimination precision of the subjects as an evaluation parameter. The experimental results for each subject are summarized in Table 1. According to Weber s law, the ratio of DL I to standard stimuli I is constant. The DLs for four subjects (subjects A, B, C, and D) are low enough although the DL for subject E is not very high. Subjects except for subject E made their fingers travel over virtual spheres to inspect their size. On the other hand, subject E examined diameters of virtual spheres by touching lightly in a short time compared to other subjects. In other words, subject E judged the size by a slight tap. Since it is essential for operators to explore virtual objects carefully, the result obtained from subject E is ignored in the following discussion. We can verify the effectiveness of the proposed display for tactile recognition in four subjects (subjects A, B, C, and D) because the sensation thresholds were low enough. Since the Weber s ratio of around 0.2 was small enough, operators could recognize a fine distinction of sphere radii generated by the display.

TABLE 1. Sensation threshold for size of virtual sphere Subjects A B C D E DL [mm] 4.7 6.3 4.6 4.6 10.8 I I 0.16 0.21 0.15 0.15 0.36 5. Conclusions. We are developing a multi-modal 2.5D display capable of stimulating the muscles and tendons of the forearms and tactile receptors in fingers, with the aim of developing a new human-friendly interface. In the present paper, we extended the versatility of the operation in 2.5D space for object manipulation in 3D space. Although the surface rotation of the virtual tactile pad was limited around the z-axis in the previous display, we proposed that the rotations around the x- and y-axes are managed by the centroid position of the operator s fingers. The position is estimated by compressive forces applied on the display pad, which are measured by three pressure sensors installed under the actuator array. Then, operators can minutely perceive the curvature and surface information of virtual objects because they can control the contact region of the virtual tactile pad flexibly while touching objects in virtual space. Since operators can manipulate vertical movement and attitude of the virtual tactile pad with minimum force, they don t grow tired during manipulations in 3D space. To evaluate our display, we conducted three evaluation experiments. In Experiment I, we compared the measurement position of centroid estimation with the exact position on the actual display pad to verify the precision of the centroid estimation. The results show that the centroid estimation is stable around the center of the display pad. Since the contact region of the virtual tactile pad is naturally changed by operators, they can perform intuitive manipulations with minimum force in virtual space. In Experiment II, subjects compared the heights of two virtual rectangular solids. Then, the DLs for each subject were measured to find the appropriate proportionality constant for vertical movement of the virtual tactile pad. As a result, we revealed the proper proportionality constant for the vertical movement of the virtual tactile pad as 3.5 mm per 1.0 N. In Experiment III, subjects tried to compare the size of two virtual spheres to evaluate the presentation capability of the proposed display. From the experimental result, since the sensation thresholds for each subject were low enough, operators could recognize a fine distinction of stimuli generated by the present display. Although these psychological experiments were conducted in no-visibility situations, subjects recognized the slight changes in VR. Consequently, it is expected that this display will reduce the burden of the operators. Moreover, this display can be applied to the tactile-vision substitution systems (TVSS) for visually impaired persons [19, 20] in future work. Finally, since we translated three of six degrees of freedom into the degree of force and moment, we achieved the 3D effects by using six degrees of freedom. It is impossible for the conventional device to present them. Acknowledgment. This work was supported by the Hori Science and Arts Foundation.

REFERENCES [1] M. Akamatsu and S. Sato, A multi-modal mouse with tactile and force feedback, International Journal of Human-Computer Studies, vol.40, no.3, pp.443-453, 1994. [2] G. Yang, K. Kyung, Y. Jeong, and D. Kwon, Novel Haptic Mouse System for Holistic Haptic Display and Potential of Vibrotactile Stimulation, Proc. of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2005), pp.1980 1985, 2005. [3] K. Sato, K. Minamizawa, N. Kawakami, and S. Tachi, Haptic Telexistence, Proc. of SIGGRAPH 07ACM SIGGRAPH 2007emerging technique, no.10, 2007. [4] H. Kawasaki, Y. Doi, S. Koide, T. Endo, and T. Mouri, Hand Haptic Interface Incorporating 1D Finger Pad and 3D Fingertip Force Display Devices, Proc. of 2010 IEEE International Symposium on Industrial Electronics (ISIE), pp.1869-1874, 2010. [5] T. Oomichi, M. Higuchi, and K. Ohnishi, Design Method of Multifingered Master Manipulator with Force and Tactile Feed-back Free from Operation Restriction by Mechanism (in Japanese), Journal of the Robotics Society of Japan, vol.16, no.7, pp.942-950, 1998. [6] Y. Shiokawa, A. Tazo, M. Konyo, and T. Maeno, Hybrid Display of Roughness, Softness and Friction Senses of Haptics, Proc. of the18th International Conference of Artificial Reality and Teleexistence (ICAT ), pp.72-79, 2008. [7] M. McLaughlin, G. Sukhatme, W. Peng, W. Zhu, and J. Parks, Performance and Co-presence in Heterogeneous Haptic Collaboration, Proc. of the 11th Symposium Haptic Interfaces for Virtual Environment and Teleoperator Systems (HAPTICS), pp.285-291, 2003. [8] H. Yao, V. Hayward, and R. E. Ellis, A Tactile Enhancement Instrument for Minimally Invasive Surgery, Computer Aided Surgery, vol.10, no.4, pp.233 239, 2005. [9] PHANToM. http://www.sensable.com/products/phantom_ghost [10] S. Tsuboi and M. Ohka, Object Handling Using Combined Display of Tactile and Force Sensation, Proc. of the 4th International Conference on Manufacturing, Machine Design and Tribology (ICMDT), pp.57-58, 2011. [11] S. Tsuboi and M. Ohka, Virtual Building Blocks Using a 2.5D display Generating Tactile and Force Sensations, 2011 International Annual Symposium on Micro-Nano Mechatronics and Human Science (MHS), pp.416-420, 2011. [12] TrackPoint. http://www.pc.ibm.com/ww/healthycomputing/trkpnt.html [13] Braille cells. http://www.kgs-jpn.co.jp/epiezo.html [14] A. Lécuyer, S. Coquillart, and A. Kheddar, Pseudo-Haptics Feedback: Can Isometric Input Devices Simulate force feedback?, Proc. of IEEE International Conference on Virtual Reality, pp.83-90, 2000. [15] M. Tatezono, K. Sato, K. Minamizawa, H. Nii, N. Kawakami, and S. Tachi, Effect of haptic feedback on pseudo-haptic for arm display, ICROS-SICE International Joint Conference 2009(ICCAS-SICE 2009), pp.4332-4337, 2009. [16] J. Gibson, Observations on active touch, Psychological Review, vol.69, no.6, pp.447-491, 1962. [17] S. Lederman, The perception of surface roughness by active and passive touch, Bulletin of the Psychonomic Society, vol.18, no.5, pp.253-255, 1981. [18] H. Dostmohamed and V. Hayward, Trajectory of Contact Region on the Fingerpad Gives the Illusion of Haptic Shape, Experimental Brain Research, vol.164, no.3, pp.387-394, 2005. [19] M. Shinohara, Y. Shimizu, and A. Mochizuki, Three-Dimensional Tactile Display for the Blind, IEEE Transactions on Rehabilitation Engineering, vol.6, no.3, pp.249-256, 1998. [20] S. Wall and S. Brewster, Sensory Substitution Using Tactile Pin Arrays: Human Factors, Technology, and Applications, Signal Processing, vol.86, no.12, pp.3674-3695, 2006.