PROPRIOCEPTION AND FORCE FEEDBACK Roope Raisamo and Jukka Raisamo Multimodal Interaction Research Group Tampere Unit for Computer Human Interaction Department of Computer Sciences University of Tampere, Finland
Outline Force sensing and control Force feedback devices Haptic rendering 1
2 Force sensing and control
Force feedback interface Force feedback interfaces can be viewed as having two basic functions (Tan et al.,1994) to measure the positions and contact forces of the user s hand (and/or other body parts), and to display contact forces and positions to the user. 3
Haptic interface design experiences The major perceptual issues are the following: force sensing under quasi static and dynamic conditions, pressure perception, position sensing resolution, and the level of stiffness required for rigidity simulation. The major manual performance issues are the following: the maximum forces humans can produce, the precision with which humans can control a force, and the control bandwidth of the force. The issues of ergonomics and comfort are also important 4
Force sensing In order for the user to perceive the forces displayed by the device as smoothly varying, the force display resolution of the device should match or exceed human sensing resolution. just noticeable difference (JND) for human force sensing is around 7% 5
Unintended vibration One of the most noticeable disturbances in a force reflecting device is the level of unintended vibration. A significant level of vibration can quickly destroy the feeling of free motion or disturb the perception and control of virtual objects in contact. The human body is very sensitive for vibration thus careful attention to hardware design and control software is needed. 6
Pressure perception 1/3 Many force reflecting exoskeletons are attached to the user s forearm in order to display contact forces at the fingertips. An unbalanced contact force at the fingertip is mechanically grounded at the forearm, and not supported by the entire upper body as it is in the real world. The rest of the user s body does not experience the effects of this force. 7
Pressure perception 2/3 The perceptual effectiveness of such a nonrealistic display is unknown. The true ground illusion might be successfully produced if the pressure distribution and its changes at the grounding location are below the absolute detection and discrimination thresholds, respectively, for the human user. 8
Pressure perception 3/3 Pressure JND decreases as a function of contact area. People are more sensitive to pressure changes when contact area is enlarged. Independent of location The contact area at the attachment points of exoskeletons for mechanical ground should be minimized and the perimeter of the area should be maximized, in order to improve the illusion of true grounding. 9
Position sensing resolution The position sensing resolution that is desired in the device depends on the position resolution of the human operator. Joint angle resolution is better at proximal joints than at distal ones. JNDs for joint angle resolution: Wrist (2.0 ), elbow (2.0 ) and shoulder (side and front both 0.8 ) Help humans to control end points (i.e., fingertips) accurately. An error of 1 in shoulder joint angle sensing would result in an error of 1 cm at the index fingertip Concerning finger joint, the corresponding error would be 0.08 cm 10
Stiffness Mechanical behavior of most solid objects in a virtual world is modeled with elastic stiffness. Many virtual objects are supposed to appear rigid. In the virtual environments stiffness and rigidity are perceptual notions that need to be decided casespecifically. What is the stiffness required to convince a user that an object is rigid? Perceived rigidity depends on both the stiffness of the interface hardware and whether the comparison is done among a set of virtual object simulations or between a simulation and a real object. For example, users can consistently judge the relative stiffness of different virtual walls even though they are never as rigid as the real walls due to hardware limitations. 11
Force output range To match human performance, the maximum force exerted by the device should meet or exceed the maximum force humans can produce. A study on the maximum controllable force humans can produce with joints of the hand and the arm. The subjects were asked to exert a maximum force and maintain it for 5 seconds. Maximum force increased from the most distal joint (finger) to the most proximal joint (shoulder) Very good control over force output with the shoulder joint compared to finger joint. 12
Force output resolution The resolution at which the force on a joint linkage must be controllable. In order to present a perceptually smoothly varying force, the forces displayed by the device must be controllable to at least the level at which humans can sense and control force. The precision with which humans can produce a mid range force with joints of the hand and the arm. The resolution is somewhat lower for proximal joints but they can control a lot larger forces. 13
Force bandwidth The force control and perceptual bandwidths of a human operator are quite different. Vibrotactile stimuli can be perceived up to about 500 Hz. The upper bound of force control bandwidth is from 20 to 30 Hz. However, the actual bandwidth is considerably less, and has been reported to be about 7 Hz. The bandwidth of the device when it is backdriven by the human operator should at least match the force control bandwidth of the operator. 14
Ergonomics and control Sizing and fatigue are also important issues, especially for exoskeletal devices Bad ergonomics can easily ruin otherwise excellent haptic display. 15
Human haptics Kinesthetic information Net forces along with position and motion of limbs Coarse properties of object Large shapes, spring like compliances 16
Degrees of Freedom (DOF) Degrees of freedom are limited: Degrees of freedom for force feedback display 6DOF for position and orientation Large number of joints and contact points Contact point: grasping versus stylus. Usually focus on hands: resolution, detection, discrimination ability increase distally. However, movement speed decreases if natural movements are constrained. 17
Human arm degrees of freedom Caldwell et al. (1995) 18
Force feedback for display of texture and shape A B In figure A, the sensation of a textured surface can be produced via a stylus that moves according to the virtual surface texture In figure B, a stylus can be used to probe the surface characteristics of a virtual object 19
Analysis of grasps 20 Cutkosky and Howe (1990)
Sensing and control bandwidth Sensing bandwidth Frequency with which tactile and kinesthetic stimuli are sensed. Control bandwidth Speed with which humans can respond. Sensing bandwidth is much larger than the control bandwidth. Output loop or the control bandwidth has a typical force response of 5 10 Hz bandwidth while the kinesthetic / proprioceptive sensing bandwidth is about 20 30 Hz. 21
Sensing and control bandwidth 22
23
24
Physical modeling for virtual objects Surface deformation Compliance & texture Grasping Haptic interface control Hard contact Collision detection Physical constraints HAPTIC INTERFACE 25 (Adopted from Burdea, 1996)
Haptic system architecture Haptic feedback User Visual, audio feedback Distributed computing platform High level control Haptic interface Bi directional energy flow Low level control Interface controller One way information flow (Adopted from Burdea, 1996) 26
Force Feedback Devices 27
Force feedback devices 1 Degree of freedom Steering Wheels Hard Driving (Atari) Ultimate Per4mer (SC&T2) 2 Degrees of freedom Pens and Mice Pen Based Force Display (Hannaford, U. Wash) MouseCAT/PenCAT (Hayward, Haptic Tech., Canada) Feel It Mouse (Immersion) Joysticks Force FX (CH Products) Sidewinder Force Feedback Pro (Microsoft) 28
1D and 2D gaming devices 29
PenCAT 30
Force feedback mice In 1999 Immersion Corporation s Feel It force feedback mouse was introduced as Logitech Wingman Force Feedback Mouse 31
Force feedback devices 3 Degrees of freedom PHANToM (SensAble Technologies) ForceDimension Delta and Omega Novint Falcon Impulse engine (Immersion) 6+ Degrees of freedom PHANTOM Premium 6 DOF, ForceDimension Delta 6 DOF, Teleoperator masters (MA 23, Argonne, CRL) Freedom 6/7 (Hayward, MPB Technologies) 6DOF (Cybernet) 32
Sensable Technologies: PHANTOM http://www.sensable.com/ 33
ForceDimension Omega and Delta 34
Novint Technologies: Falcon http://www.novint.com/ 35
FCS Systems: HapticMASTER 36
3 DoF Force Feedback Joystick Impulse Engine 2000 37
Medical Force Feedback Systems 38
Immersion CyberGrasp 39
What makes a good haptic interface? Must work with human abilities and limitations Approximations of real world haptic interactions determined by limits of human performance 40
A good haptic interface Free motion must feel free Low back drive inertia and friction No motion constraints Ergonomics and comfort Pain, discomfort and fatigue will detract from the experience 41
A good haptic interface (2) Suitable range, resolution and bandwidth User should not be able to go through rigid objects by exceeding force range No unintended vibrations Solid objects must feel stiff 42
Haptic Rendering Courtesy of Reachin Technologies AB, www.reachin.se 43
Haptic rendering Haptic rendering is the process of computing and generating forces in response to use interactions with virtual objects, based on the position of the device. Haptic rendering of an object can be seen as pushing the device out of the object whenever it moves inside it. The human sense of touch is sensitive enough to require a processing speed of at least 1000 Hz in terms of haptic rendering. 44
Haptic rendering The haptic rendering needs to provide forces that push the user out of the object. The further inside the object you move, the greater the force pushing you out. This makes the surface feel solid. m 45
1000 Hz Performance Requirement The user becomes a part of the simulation loop. 1000 Hz is necessary so that the whole system doesn t suffer from disturbing oscillations. The PHANTOM haptic devices run their control loop at 1000 Hz. The consequence: we are very limited on the amount of computation that we can do. 46
1 khz haptics rendering: update speeds Real time loop (~1000 Hz) Necessary due to the high sensitivity of human touch. Not necessary to look at every object in the scene 1000 times per second. Scene graph loop (~60 Hz) Looks at every object in the scene and generates surface instances that are rendered at 1000 Hz. 47
1 khz haptics rendering Finger Force Scene graph object The real time surface is a parametric surface. This means that it can be curved to closely match the real surface curvature locally. The finger is the actual position of the haptic device 48
1 khz haptics rendering Haptic position T S The real time surface has a 2D coordinate space to allow programmers to define haptic surface effects as a function of position and penetration depth. 49
1 khz haptics rendering 3 DOF haptic devices are rendered in the API using a spherical proxy. The proxy stays on the surface of objects, and is maintained such that it is at the closest point on the surface of an object to the haptic device. 50
Device support Reachin API currently supports: SensAble PHANToM family ForceDimension Delta and Omega haptic devices. Immersion Laparoscopic Surgical Workstation and Impulse Engine. Virtuose devices from Haption Novint Falcon Devices are generally configured for each computer rather than for each application. The same application can be used with different hardware settings without changing the code. 51
Degrees of Freedom Haptic rendering for pen type devices: 3DOF Haptics (3D force output) Point Object interaction 6DOF Haptics (3D force + torque output) Object Object interaction DOF = Degree of Freedom 52
Haptic interaction 53
3DOF Haptics: Introduction Output: 3D force 3DOF haptics Limited to applications where point object interaction is enough. Haptic visualization of data Painting and sculpting Some medical applications Point Object Point object Object object Object Object 54
6DOF Haptics: Introduction Output: 3D force + 3D torque For applications related to manipulation. Assembly and maintenance oriented design. Removal of parts from complex structures. Typical problem: peg in the hole. There is a net torque 55
Haptic rendering Two parts: collision detection, response 56
Two types of interactions Point based haptic interactions Only end point of device, or haptic interface point (HIP), interacts with virtual object When moved, collision detection algorithm checks to see if the end point is inside the virtual object Depth calculated as distance between HIP and closest surface point 57
Two types of interactions (2) Ray based haptic interactions Probe of haptic device modeled as a linesegment whose orientation matters Can touch multiple objects simultaneously Torque interactions 58
Haptic texturing (3) 59
History Argonne 54, first master slave systems. Master=slave. Salisbury 80, independent master and slave. Cartesian control of robot arm. Later, interaction with computer simulated slave (VR). GROPE project UNC 67 90, molecular docking. Minsky 90, the Sandpaper. Massie & Salisbury 94, the PHANTOM device. Early 90 s to 97, 3DOF haptics. Late 90 s to today, 6DOF haptics. 60
References Basdogan, C., Srinivasan, M.A. Haptic rendering in virtual environments. http://network.ku.edu.tr/~cbasdogan/ Papers/VRbookChapter.pdf Chen, E. Six degree of freedom haptic system for desktop virtual prototyping applications. Proc. First International Workshop on Virtual Reality and Prototyping, p. 97 106, 1999. Gregory, A., Lin, M., Gottschalk, S. and Taylor, R. A Framework for Fast and Accurate Collision Detection for Haptic Interaction. Proc. of the IEEE Virtual Reality (VR 99), p. 38 45, 1999. Mark, W. et al. Adding force feedback to graphics systems: issues and solutions. Proc. ACM SIGGRAPH 1996. Massie, Thomas H. and Kenneth Salisbury. The PHANTOM haptic interface: a device for probing virtual objects. Proc ASME Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, 1994. McNeely, W., Puterbaugh K., and Troy, J. Six degree of freedom haptic rendering using voxel sampling. Proc. ACM SIGGRAPH 1999. 61
References (2) Ruspini, Kolarov and Khatib. The haptic display of complex graphical environments. Proc. ACM SIGGRAPH 1997. Salisbury, J.K. et al. Haptic rendering: programming touch interaction with virtual objects. Proc. ACM SIGGRAPH 1995. Salisbury, J.K. and Srinivasan, M.A. Phantom based haptic interaction with virtual objects. IEEE Computer Graphics and Applications, 17(5), p. 6 10. Salisbury, J.K. Making graphics physically tangible. Communications of the ACM, 42(8), p. 74 81. Srinivasan, M.A. and Basdogan, C. Haptics in virtual environments: taxonomy, research status, and challenges. Computers & Graphics, 21(4), p. 393 404. Zilles, C.B. and Salisbury, J.K. A constraint based god object method for haptic display. Proc. IEE/RSJ International Conference on IntelligentRobots and Systems, Human Robot Interaction, and Cooperative Robots, Vol. 3, p. 146 151, 1995. 62
References R. J. Adams and B. Hannaford. A Two port Framework for the Design of Unconditionally Stable Haptic Interfaces. In Proc. of IEEE/RSJ Int. Conference on Intelligent Robots and Systems, 1998. C. Basdogan, C. H. Ho and M. Srinivasan. A Ray based Haptic Rendering Technique for Displaying Shape and Texture of 3D Objects in Virtual Environments. In the Proc. of ASME Dynamic Systems and Control Division, 1997. G. Burdea. Force and Touch Feedback for Virtual Reality. John Wiley and Sons, 1996. S. A. Ehmann and M. C. Lin. Accurate and Fast Proximity Queries between Polyhedra Using Surface Decomposition. In Proc. of Eurographics, 2001. A. Gregory, M. C. Lin, S. Gottschalk and R. M. Taylor II. H Collide: A Framework for Fast and Accurate Collision Detection for Haptic Interaction. In Proc. of IEEE Virtual Reality Conference, 1999. A. Gregory, A. Mascarenhas, S. Ehmann, M. C. Lin and D. Manocha. Six Degree of Freedom Haptic Display of Polygonal Models. In Proc. of IEEE Visualization, 2000. 63
References V. Hayward and B. Armstrong. A New Computational Method of Friction Applied to Haptic Rendering. In Lecture Notes in Control and Information Sciences, Vol. 250, Springer Verlag, 2000. Y. J. Kim, M. A. Otaduy, M. C. Lin and D. Manocha. Six Degree of Freedom Haptic Display Using Localized Contact Computations. To appear in Proc. of Haptic Symposium, 2002. W. R. Mark, S. C. Randolph, M. Finch, J. M. Van Verth, and R. M. Taylor II. Adding Force Feedback to Graphics Systems: Issues and Solutions. In Proc. of ACM SIGGRAPH, 1996. W. A. McNeely, K. D. Puterbaugh and J. J. Troy. Six Degree of Freedom Haptic Rendering Using Voxel Sampling. In Proc. of ACM SIGGRAPH, 1999. D. C. Ruspini, K. Kolarov, and O. Khatib. The Haptic Display of Complex Graphical Environments. In Proc. of ACM SIGGRAPH, 1997. C. Zilles and K. Salisbury. A Constraint based God Object Method for Haptic Display. In Proc. of IEEE/RSJ Int. Conference on Intelligent Robots and Systems, 1995. 64
This presentation is partly based on presentations by the following people: Pierre Boulanger, Department of Computing Science, University of Alberta Max Smolens, University of North Carolina at Chapel Hill Ming C. Lin, University of North Carolina at Chapel Hill Ida Olofsson, Reachin Technologies AB 65