Integration of a Force Feedback Joystick with a Virtual Reality System

Transcription

1 Integration of a Force Feedback Joystick with a Virtual Reality System Alfredo C. Castro, José F. Postigo, Jorge Manzano * Instituto de Automática. Facultad de Ingeniería.Universidad Nacional de San Juan Av. San martín 09 (oeste) San Juan. ARGENTINA. Tel.: Fax: {acastro,jpostigo}@inaut.unsj.edu.ar Keywords: teleoperation, force feedback, virtual environment, network, man-machine interface. Abstract The integration of force feedback with a complete real-time virtual environment system shows some more difficult problems than those found in building a typical force feedback system. Particularly, bulky computations for graphics or simulation require decoupling the haptic servo-loop from the main application loop if highquality forces (realistic) are to be obtained. As an approach to this problem, a control system is developed here for a force feedback joystick (Impulse 000 -Immersion Corp.) along with its integration to a virtual environment. Technical issues on the development of stable control architectures for Internet-based exchange of haptic information are discussed. Some experiments were developed to show the performance of both the force feedback joystick and the virtual reality system. This work was carried out in the Laboratories of the Robotics and Informatics Division of ENEA under an ENEA s project grant. Rome, Italy. Introduction The integration of force feedback to a complete realtime virtual environment system shows several problems which are, in general, more difficult to solve than those found in building a standard force feedback system. In particular, bulky computations for graphics or simulation require a decoupling of the haptic servo-loop from the main application loop if high-quality forces are to be produced (Mark et al., 996). Likewise, to use haptic devices over the Internet is a much more challenging problem than to transfer audio and video data, because these devices must remain stable in spite of the typical fluctuations of performance of the Internet. Typically, Internet data present random timedelay and packet losses, which would make a control system unstable if it were used to close the control loop over Internet (Fiorini and Oboe, 997). In this work, we present an approach to solve these problems by especially proposing a controller for the force-feedback joystick Impulse Engine 000 by Immersion Corp. Its integration to a virtual environment and the possibility of exchanging haptic information through Internet are also described here. In a Virtual Reality (VR) training environment, although 3D visualization is a useful tool for learning to teleoperate a complex procedure, it does not suffice for developing a physical skill which requires a haptic interaction as a necessary component of the training process (Colgate et al., 995). This work was performed within the framework of the ENEA projects TINA and RAS. The former project is financed by the Italian Ministry for Scientific Research and it is aimed at improving ENEA s telemanipulator MASCOT. The RAS project is within the Italian Scientific Research Program of Antarctica and it is aimed at implementing a remotely operated vehicle equipped with a teleoperated arm (Manzano, 997; Fichera et al., 997).. The Mascot teleoperation system In 96 the researchers from ENEA developed the first Mascot unit, a telemanipulator for nuclear plant operation. This manipulator was, and still is, one of the best machines available in the world as regards the force feeling it can transmit back to the operator (force feedback signals). The Mascot is a Master/Slave telemanipulator of the force feedback type (see Fig. ). Each arm has seven servocontrolled joints: six links for six degrees-of-freedom plus a gripper. Each joint is driven by its own actuator through gears or steel cables. The control algorithm is based on comparing the position and velocity of the joints of the Master and Slave arm, sensed at the same instant. These values, the position and velocity errors, are then multiplied by adequate proportional coefficients to determine the torques to be applied to the Slave arm actuators (which is thus forced to follow the Master) and to those of the Master arm, to generate the force feedback to the operator. These torques are applied in order to minimize the position errors.. Work objectives The RAS and TINA projects impose several new features to the last version of MASCOT. Some of these, addressed here, are: To increase the field of applications via VR: To use the real master with a Mascot virtual slave for training with virtual reality techniques or to use other device as an alternative master. In this work, virtual environment applications were developed. * Divisione Robotica e Informatica Avanzata. Dipartimento Innovazione. Ente per le Nuove Tecnologie, l Energia e l Ambiente (ENEA). Rome, Italy.

2 To lower the costs: Use of less expensive devices as an alternative to the master arm. This device may be a market-available device or another developed at the ENEA. In this work, we use a market-type device. Figure. ENEA s Mascot Sytem. To increase the distance between master and slave sites: The long-distance communication between master and slave would be established via a radio link or a network link. Here, we discuss the possibility of using a network link. Force Feedback Joystick The main features of different market-type force feedback devices were studied and analyzed at ENEA Lab which led to select the Impulse Engine 000 TM joystick (see Fig. ) of Immersion Corporation. The analysis was based on a price-benefits criterion.. Main Features This quality force feedback joystick for research has the following features, as supplied by Immersion Corporation: Two degrees of Freedom. ISA bus interface (PC). Cable-driven capstan transmission. 6" x 6" (5. x 5. cm) Workspace Size " (00 dpi) Position Resolution. lbs. (8.9N) Max Force Output. <0.5 oz (0.4 N) Backdrive Friction. 0 Hz Bandwidth. 3 Joystick Controller The software development kit of the force-feedback joystick allows for writing applications that basically read the joystick s position and to write its output force, all under a DOS and Win 95/NT environment. Since Immersion Corporation does not supply any kind of controller software or hardware with the joystick, the need arises for developing a digital position controller to be used as a master or slave in a teleoperation system. A brief description of the stick model of the joystick follows. 3. Simplified model of the stick A simplified model of the stick was used for the controller design, as shown in Figure 3. y θ I l mg θ τ x Figure 3. Simplified model of the stick. The dynamic equation which describes the stick s behavior is: Figure. Impulse 000 Force feedback joystick. The Impulse Engine 000 TM is a force feedback joystick that accurately tracks a two-degree-of-freedom motion and features a PC interface for communication between PC and joystick, at rates of up to 0KHz (Immersion Corporation, 997). τ = I θ& bθ& mgl sin θ ()

3 where: m I l θ mass of the stick. inertia of the stick. distance from joint to mass center of stick. stick tilt angle (referred to the vertical). By applying Laplace Transform to Eq.() and using the trigonometric approximation sinθ θ, we obtain, The digital controller is designed by considering the human operator as an external perturbation. The complete control system is shown in Fig Non-symmetric handle problem The handle asymmetry generates an additional error in the joystick control. This is because the gravity center is not located on the symmetry axis of the stick. This introduces a difference θ in the measured tilt angle of the stick with respect to that of the ideal stick (see Fig. 5). Τ(s) = I Θ(s) s b Θ(s) s mgl Θ(s) () y θ θ Then, the open-loop transfer function G(s) of the plant is, θ Θ( s) G( s) = = (3) Τ( s) Is bs mgl Since Eq. (3) is a continuous transfer function and the system is digital, we must use the z operator to obtain the proper transformations. Using bilinear transformation, the transfer function of Eq. (3) in the z plane becomes: b 0 b z b ( ) z G z = (4) a z a z where: b b a a 0 = b T = sampling time Ref. θ T = bt mglt T = bt mglt ( mglt ) = bt mglt bt mglt = bt mglt Σ - Controller Σ τ Operator Figure 4. Closed loop system. G (z) Out θ Figure 5. Handle of the stick. By inlcuding this difference θ in Eq. () it yields, τ = I θ& bθ& mgl sin( θ θ ) (5) The difference θ is constant and adds a steady-state error to the system response when a PD controller is used. 3.3 Control Structures Three control schemes were tested in order to eliminate the steady state error. Main results are summarized as follows: a) PID Controller If we add the integrative factor into the controller structure, the steady state error is eliminated. Nevertheless, an integrative action often makes the system to be more sensible to small perturbations, which can lead the system to instability (Ogata, 987). b) State Space Controller A state space controller for the above model (Eq. 4) was designed using Matlab (Ogata, 987; Messner and Dawn, 997). Simulation results of the closed loop system were satisfactory, but when this controller was applied to the real system, we noticed that it worked in the limits of stability. We concluded that to use a simplified model was not enough for the controller design. c) PD with gravity pre-compensation controller This structure is shown in Fig. 7 and will be x

4 described in section 4.4. In this structure, the tilt difference angle θ is known a priori and the output could be predetermined in this way. A gravity precompensation PD controller is obtained by implementing a standard PD controller adjusted to compensate this tilt angle difference. This pre-compensation is used only on the y-axis, where the handle is asymmetric. The main features of the PD controller are: Fast computing algorithm. Robust algorithm. Stability warranted by using the PD controller. 4 Software Application The software application is the main objective of this work. Here, we used a client-server architecture where JoysSrvr is a server application that controls the force feedback joystick Impulse 000. It communicates easily with others client applications that need to interact with the joystick. The JoysSrvr application runs under the Microsoft Windows NT system and uses two modes for communicating with the client application: via shared memory link and via network link. The digital controller for the JoysSrvr application is a PD with precompensation of gravity. 4. Client-Server Architecture Client application Client application Joystick Client-Server Application JoysSrvr NT Driver inp.sys Client Application Figure 6. Developed Client -Server Architecture. The well-known client-server architecture was used for designing the JoysSrvr application. The possibilities to link the client to the server are shown in the scheme of figure 6. The main characteristics of this architecture are: Easy and "clean" Interface with the rest of the system. The change of the force feedback device implies only to develop a new server application. The architecture allows for decoupling the force servo loop from other loops (graphics, simulation, etc). Ready architecture for extension to the network scenario. 4. Application features The JoysSrvr application was developed in Microsoft Visual C 5.0. The Windows interface code is completely programmed in C. C language was used for the controller algorithm because the API multimedia timer calls for C functions (Immersion, 997; Microsoft, 997). The platform used in the development is Microsoft Windows NT, and the application features are: Multi-thread application: A high priority thread for the controller and a normal priority thread for the main application. Use of multimedia timer: High precision timers supplied for Windows NT. KHz sampling frequency for the controller. This is the maximum frequency attainable by the multimedia timer. Windows socket support: This is a specification for network programming interface of Microsoft Windows. Low-resource usage: The experiments show that the application used only the 0% of the machine resource for control the joystick. User-friendly interface: The interface was developed using the graphics resources supplied by Microsoft Windows. On-line help: The application was provided with a hypertext help file. 4.3 Operation modes JoysSrvr is a server application featuring two modes of communication with the client application: using shared memory or using sockets. Both modes will be described in the following. Direct force Sel. Ref. signal Position Input signals (from client application) 0 Reference Selector PD Σ Controller - Gravity precompensation (only in y-axis) Σ Joystick Buttons state Velocity Position Output signals (from server application) Figure 7. JoysSrvr controller structure.

5 a) Shared memory In this mode the two independent processes (server and client running in the same computer) transfer the information using a shared memory area. b) Winsock In this mode the two independent processes (server and client running in the same or in different computers) transfer the information using the network protocol TCP/IP. The operation mode is selected at the beginning of the JoysSrvr application start-up. Only one mode is allowed during the execution of this process. 4.4 JoysSrvr Controller JoysSrvr features a PD controller with gravity precompensation, as shown in Fig. 7. The proposed structure has as input signals, the information sent from the client application to the server and as output signals, the information from the server to the client application (see Figure 7). The input signals supplied by the client application, are the following: Direct force: This signal specifies a direct force applied to the d.c. motors. It is added to the controller output and to the human operator force in the structure of Fig. 7. This allows the simulation of damping, texture, spring and other effects. The range of this signal (force) is.0 to.0 in single precision numbers. Reference Selector signal: This is a digital signal that indicates the input reference to the controller. The digital signal can be: Reference Closed: The reference is the client position. It is used for systems without significant time delays. Closed loop control is performed by the client application. Reference Open: The reference is the position of the joystick (free space simulation). This is used for systems with significant time delays. Reference Zero: The reference is zero. This is used for mapping the joystick position client side to a velocity signal. Position: This signal is the reference position generated at the client application. The range of the position signal is from -.0 to.0 in single precision numbers. The output signals of the control structure (that is signals to the client application) are: Buttons state: This signal indicates the state of the button of the joystick to the client application. The states for each button may be pressed or unpressed. Velocity: This output is the velocity of the joystick, that is, position variation in one sampling time. Its range is from.0 to.0 in single precision numbers. Position: Actual position of the joystick. The range of this output position is from.0 to.0 in single precision numbers. 5 Integration with the virtual environment The real time virtual environment incorporates some additional problems to the use of a force feedback device (Mark et al., 996), such as: Bulky (lengthy) computations for rendering the visualization of the virtual environment. These are timeconsuming computations. Difference of loops updating. Reasonable display update rates, with around 40 Hz are reached, but it isn t enough for the force feedback signals, where higher update rates are required (generally greater than 500 Hz). The proposed solution to solve the above problems is to decouple the different loops of the application, that is, Decoupling the haptic servo-loop from the remaining application loops. This is done using the highest priority for executing this loop. Decoupling the graphics loop from the haptic servo loop. This time, the idle priority is used to execute this loop. WorldToolKit (WTK) is a portable, cross-platform software development system for building high performance, real time, integrated 3D applications (Sense8, 996). Virtual slaves were developed using WTK, enhancing the two main problems of virtual environment integration to the joystick: Mapping the multi-degree-of-freedom virtual slave into the two-degree-of-freedom joystick. The application developed with this in mind is called ArmMFC. Adding force feedback feature to the virtual slave. The application developed keeping this in mind is called ModelD. A brief description of both applications follows. 5. Application ArmMFC Figure 8. ArmMFC Application The ArmMFC application (see Fig. 8) maps the 4- dof-virtual robot arm onto the -dof joystick and onto the joystick s buttons. Besides, the position and

6 velocity control swap is possible by pressing the joystick buttons. The interface of the application for Windows was developed in Microsoft Visual C, but the WTK code is in C for a portable platform. 5. Application ModelD Figure 9. ModelD Application The application ModelD (see Fig. 9) is a twodegree-of-freedom virtual slave. This slave with force feedback maps the joystick s two degrees and simulates a simple virtual environment. This application is an example of a haptic interface because it allows simulating texture, damping, spring and other effects in a virtual environment. This slave is also tested with network link between the master and slave site as is shown the Fig Master slave connection using TCP/IP protocol 6. Communication through Internet One of the standard tools of Internet communication is based on TCP/IP protocol. This protocol includes reliability feature, such as error recovery, large packet splitting and reordering. Nevertheless, TCP/IP protocol introduces an excessive overhead and due to this, it produces long time delays (Fiorini and Oboe, 997). 6. Problems and solutions The use of a computer network for communicating the master and the slave sites in a teleoperation system generates some additional problems. Among the more important ones are (Funda, 990; Fiorini and Oboe, 997): Time delay: The delay in a network can be defined as a time-varying random process. Limited bandwidth. Data losses. Addition of the communication loop to the closed loop control system. The above items can degrade or even destroy the sense of telepresence and therefore reduce the efficiency of task performance (Funda, 990). The solution proposed is to use the so-called Model Based Teleoperation. With this technique, the following features are incorporated to the original system: Information exchange of the model from the slave site to the master site. In this sense, another application that is a client to the joystick and a server to the slave site, was added. This application executes the simulation loop execution of the slave site in the same computer where the master site is physically located. Therefore, this fact increases the intelligence and autonomy of the master site. Model App. Joys App. Shared memory link Computer slave site JoysSrvr App. Computer master site Network Link Hardware Link Figure 0. Master-slave teleoperation system.

7 Mascot App. JoysSrvr App. Computer master site Computer slave site Network Link Hardware Link Figure. Virtual Mascot system Decoupling of the communication loop from the simulation loop. The communication loop has a variable rate, depending on network traffic. The simulation loop requires a higher updating rate, greater than 500Hz. 6.3 Internet master-slave system The system runs two applications in the computer with the joystick (in the master site). The former application is the joystick controller and the second is the slave site simulation. Both applications communicate between each other via a shared memory link and run under a Windows NT system. The ENEA s virtual mascot system was developed in WorldToolKit. This virtual slave was linked via network to the joystick, through the application JoysSrvr in network mode. The virtual slave runs in a Silicon Graphics machine and the joystick is attached to a PC WinNT system (see Figure ). 6.4 Experiments and results After using different clients and server applications in an Intranet available at ENEA s laboratories, the following values were reached: Data exchange without visualization, only with the information exchange loop. Maximum of 500Hz in a low traffic Intranet. With 3D rendering, obtained with a SGI machine. It yields a maximum of 40 Hz. This value depends on the model complexity to render the visualization. With 3D rendering and collision computation. It yields from 8 to 0 Hz, depending on the model complexity. 7 Conclusions In the present work we have demonstrated the feasibility of implementing a real time controller under a Windows NT for a force feedback device. The stability of the system is warranted whenever: A PD controller is used (this grants a non-intensive computation control algorithm). The priority for each loop is scheduled. The priority order is crucial to maintain the system stability. In a decreasing order, the priorities are:. Controller.. Simulation. 3. Sockets. 4. Graphics. Real time is the best priority to be assigned to the controller loop, but it must be carefully used because an intensive computations could jeopardize the stability of the Win NT system. It is very essential to put all the graphic functions in an idle priority loop. Satisfactory solutions that prevent the degradation of haptics performance of the virtual environment or the mechanical systems coupled via Internet were found by: decoupling the simulation and graphic loops. adding a simulation loop to the master site (intelligence layer). Further research in this area includes the development of a telerobotic system with a force feedback joystick in the master site and a real robot in the slave site minimizing the network delay stability problems. Acknowledgments The work reported here was partially supported by the Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET), Argentina, and by the Ente per le Nuove Tecnologie, l Energia e l Ambiente (ENEA), Rome, Italy. References Colgate, J. Edward; Stanley, Michael C.; Brown, J. Michael. Issues in the Haptic Display of Tool Use. Proceedings of the 995 IEEE/RSJ International Conference on Intelligent Robots and Systems, Pittsburgh, PA, pp Funda, Janez. Teleprograming: Overcaming communica-tion delays in remote manipulation. Computer and Information Science Department.

8 University of Pennsylvania. June 990. Fichera, P.A.; Manzano, J.; Moriconi, C. RAS - A Robot For Antarctica Surface Exploration. International Advanced Robotics Program Workshop. Proceedings - Genoa, October 3-4, 997. Fiorini, Paolo and Oboe, Roberto. Internet-Based Telerobotics: Problems and approaches. 997 International Conference on Advanced Robotics (ICAR'97), Monterey (CA), July 997. Immersion Corporation. Impulse Engine 000 Software Development Kit. Release 3.0. Immersion Corporation. September 997. Manzano, J. Specifiche per l Ingegnerizazzione del MASCOT. TINA project report, October 997. Mark, W. R.; Randolph, S. C.; Finch, M.; Van Verth, J. M.; Taylor II, R. M.. Adding Force feedback to graphics systems: Issues and Solutions. Proceedings of SIGGRAPH 96 (New Orleans, Louisiana, August 4-9, 996). In Computer Graphics Proceedings, Annual Conference Series, 996, ACM SIGGRAPH, pp Messner, William and Tilbury, Dawn. Control Tutorial for Matlab. Mechanical Engineering at Carnegie Mellon and University of Michigan, Microsoft Corporation. Visual C Programmer's Guide. Microsoft Corporation, 997. Ogata, Katsuhiko. Discrete-time control systems. Prentice-Hall, Englewood Cliffs, NJ, 987. Sense8 Corporation. WorldToolKit Reference Manual. Sense8 Corporation, July 996.