Real-Time Bilateral Control for an Internet-Based Telerobotic System

708 Real-Time Bilateral Control for an Internet-Based Telerobotic System Jahng-Hyon PARK, Joonyoung PARK and Seungjae MOON There is a growing tendency to use the Internet as the transmission medium of the telerobotic system since the Internet is inexpensive and available all over the world. However, the bilateral real-time teleroperation can be unstable due to the random time delays of the Internet. The Event based teleoperation is one of the fine approaches to overcome the random time delays. Since the event based control uses a non-time action reference, the stability and response performance of the system are affected by the time which an event is held for. This study introduces the variable holding time to secure stability and response performance at once. The proper holding time for each event is obtained depending on the characteristics of the task through the fuzzy logic. The proposed control scheme was implemented on a robot arm over the Internet to show its effectiveness. Key Words: Internet, Time Delay, Fuzzy Logic, Event Based Control 1. Introduction Teleoperation requires data transmission between human operator and remote system to interchange control information. The Internet as a transmission medium gives the teleoperation the advantage of worldwide availability inexpensively since it has spread almost all over the world. However, the Internet has a problem to be solved; irregular time delay can lead instability of the bilateral teleoperation system. The irregular time delay is an inevitable and unpredictable phenomenon which is caused by the routing way of the network. To avoid the effect of the delay, many studies take the non-real time control method, the unilateral control method, or the assumption that the delay has upper bound (1) (5). However, the real time bilateral control is indispensable to offer the operator a high level telepresence which enables reaction to the variation of the remote environment. Only when the control scheme is real time and bilateral, the robot can trace operator s improvised motion and the operator can be provided useful sensory feedback information from the remote environment; as a very common sensory feedback, force reflection can significantly improve the teleoperation (6). Besides, any as- Received 2nd May, 2003 (No. 03-5059) School of Mechanical Engineering, Hanyang University, Haengdang, Seongdong, Seoul 133 791, Korea. E-mail: smoon@hanyang.ac.kr Department of Mechanical Design, Hanyang University, Haengdang, Seongdong, Seoul 133 791, Korea sumption on transmission delay like an upper bound is unable to be ensured on the Internet. Recent works that purposed real time bilateral teleoperation via the Internet are still open for further improvement in performance terms (7), (8). Xi and Tarn made an event based teleoperation system which is controlled by non-time action reference to immune to time delay (9) (11). This system secured stability successfully but time delays existing between each event may slow system response down. In this paper variable holding time is introduced to improve the response of the event based control system. This method holds each event for proper holding time according to characteristics of the task of the system to secure stability and response at once. Fuzzy logic is implemented to infer the proper holding time from factors that tells the characteristic of the task. In order to show its effectiveness, this method is demonstrated experimentally by operating a SCARA robot via the Internet. 2. Real Time Bilateral Teleoperation Scheme 2. 1 Event based control via the internet A robotic system needs the action reference variable by which a robotic system is synchronized and controlled. Generally, most of control methods use the time as the reference variable because they are easy to build and synchronize different components in the system. If there is irregular time delay on the control loop, however, the time based control method loses the synchronization among different components and the stability. To overcome the

709 Fig. 1 The event based control through the Internet, After receiving a command, R(s), the slave follows the command and transmits feedback data, Y(s + 1), When feedback data are arrived, the master produces new command, R(s+1) effect of the time delay, the event based control is driven by the non-time action reference which is called an event. Because the event is based on the sensory output of the robot which is independent of time, each component in the entire system can refer same action reference and maintain the synchronization regardless of the communication time delay. The master system doesn t transmit a new command, R(s + 1) until feedback data, Y(s + 1) resulted from previous event are received from the slave system as shown in Fig. 1. The other side, the slave system halts the robot by feeding zero velocity or previous position data into the local controller and no feedback data will transmit to the master system until next command is arrived. In this manner, the operator on the master side generates a new control command which is based on latest status of the remote robot and the system can be immune to the irregular time delay. 2. 2 Holding time The irregular time delay still exists between each event of the event based teleoperation. It means the robot moves at irregular intervals. Receiving a new event command, the slave performs it for one sampling time of local controller, transmits feedback data to the master and stops until next event arrives. This is to maintain the stability but on the other hand, overall process is slowed down and chattering may be occurred. Generally, to improve response against small time delay between each event, the control input is held for several ten milliseconds after transmitting feedback data. If new event arrives while the previous event is being held, the system can follow next reference input without a stop. Therefore, the system can respond fast and smoothly. But the state arrived to the master is not the real state of the remote robot. Between the states, there is a difference as much as minor of the holding time and RTT (Round Trip Time). Because it means the synchronization is distorted as much as the time, the system can be unstable depending on the characteristics of the task when the holding time is large. The existing event based control derives the holding time experimentally taking predictable working conditions into account to secure stability and response. However, the robot system can be supposed to perform various tasks under unpredictable circumstance. Therefore, one pre-defined holding time is not sufficient to cover the whole working range of the robot. So the event based control scheme which has variable holding time is developed. The variable holding time appropriate to the present working condition will enable the remote robot to carry out various tasks. 2. 3 Deciding variable holding time To find proper holding time for each event automatically, the factors that reflect working condition have to be investigated. Operator s intention, similarity of commands and environmental information are most useful factors that tell the characteristic of the task and can be referred to decide proper holding time. 1 ) Operator s intention: Observing the feedback data, the operator can transmit information about how the future task will be to the remote system. If the operator wants to perform precise position or force control with the robot, the holding time should be small. The small holding time ensures the strong synchronization between the components and enables precise control though it gets slow response. On the other hand, in order to operate rapidly in the working space which does not have obstacles, the holding time should be increased to generate events continuously. 2 ) Similarity of commands: If the remote system receives the same control inputs for several events, it is mostly like to take place that the next control input is the same too. In this case, the system can avoid unnecessary rests by taking large holding time. On the other hand, when the control inputs are variant, the holding time should be reduced to deal with different control input. 3 ) Environmental information: The robot system can detect surroundings with various sensors. If the robot is close to an obstacle or in contact with other objects, the operation should be performed carefully. Since even slightly distorted synchronization may cause the system damage, the new event based on real state of the remote system, has to be generated by reducing the holding time. 2. 4 Fuzzy logic for the holding time The component to get proper holding time from those factors is constructed as Fig. 2. The operator s intention can be obtained by additional input device like a jog wheel on the master side. If the operator wants very careful operation, he should set the jog wheel to 1. On the contrary, very rapid operation can be realized by setting the jog wheel to 1. The similarity analyzer is placed on the

710 Operator s intention factor Fig. 2 System block for deciding the holding time. The fuzzy logic infers the proper holding time, h(s) from sensed force, S (s), user s intention, U(s), and similarity of commands, V r (s) Similarity of reference input (%) slave system to get similarities between new control input and previous inputs which were stacked in buffer memory. 20 control inputs are stacked in buffer memory and the latest one of them has higher priority. If new control input is arrived, the analyzer compares it with previous inputs in terms of its direction and magnitude. Finally, the similarity of control inputs is expressed in percentage. And aforce/torque sensor supplies force values exerted on the robot. In this application, the force exerted by surroundings just indicates whether the robot is in contact with surroundings or not. So the fuzzy membership functions were constructed within the small range of the force; from 0 to 2 kgf. Fuzzy logic is applied to deciding the proper holding time from those three data. The fuzzy membership functions for three inputs and one output are in the shape of triangle as shown in Fig. 3. The fuzzy rule implemented in this application is represented by rule surfaces as Fig. 4. Mamdani s fuzzy implication and the max-min composition were used for the fuzzy reasoning. And the defuzzi- Fig. 3 Sensed force (kgf) (d) Output: Holding time (ms) Fuzzy membership functions for operator s intention factor, similarity of reference input, sensed force, and (d) holding time fication strategy which is implemented is center of gravity method (12). Fig. 4 Fuzzy rule surfaces which show the relations between similarity of commands and user s intension, user s intension and sensed force, and similarity of commands and sensed force

711 Fig. 5 Block diagram of the event based teleoperation system with variable holding time Fig. 7 System configuration for the experiments Table 1 Specification of experimental apparatuses Fig. 6 Algorithm of the event manager. It produces new reference input for the local controller 3. Implementation Event based teleoperation system with variable holding time is constructed as shown in Fig. 5. The arrival of the feedback data containing position of the robot, P(s) and force reflection, F(s) trigger for operator to generate new event data which consist of control input, V r (s) and his intention information, U(s). After receiving new event data, the slave system decides the proper holding time, h(s) referring operator s intention, U(s), similarity of commands, and sensor value, S (s). Simultaneously, system status, P(s) ands (s) are transmitted to the master system and the control input, V r (t) is feed to local controller for the holding time. This process is supervised by the event manager of which algorithm is summarized in Fig. 6. The robot used is a Samsung FARAMAN AS1-i robot with 6 degree of freedom. Logitech wingman extreme force feedback joystick is selected as the haptic device with which the operator gives reference velocity and takes force feedback. Since the joystick has two degree of freedom, a plane work space is chosen and the robot is set up to move only on the surface using three joints; the second, third and fifth joint (13). The measurement of the force exerted on the robot is assigned to a Bl. Autotech 6-Axis

712 (d) Fig. 8 Reference and actual velocity of the system when the holding time is 30 ms, 200 ms, or variable. The variable holding times are shown on (d). (d) Fig. 9 Target and actual position of the system when the holding time is 30 ms, 200 ms, or variable. The variable holding times are shown on (d). force/torque sensor. Being located between the sixth arm and the gripper, the force/torque sensor observes the force exerted on the end point of the gripper. All software for the system is developed on Windows NT base and the Internet connectionisestablishedwithtcp/ip protocol. The physical configuration of the system is observed in Fig. 7 and

713 Fig. 10 (d) Force control error of the system when the holding time is 30 ms, 200 ms, or variable. The variable holding times are shown on (d). the specifications of experimental apparatuses are summarized in Table 1. 4. Experimental Results The operator was connected to the remote robot located 400 kilometers away via the Internet. The average RTT between two points was 291 milliseconds during the experiment. To verify that the proposed system ensures stability and response in various conditions, three tasks of which characteristics are different one another were selected. And the system is compared to two existing systems which have the pre-defined holding times. One has 30 millisecond holding time to place weight on stability, whereas the other has 200 milliseconds to get a quick response. Figure 8 shows the velocity of the robot which is moving from one working region to another without obstacles nearby. The operator moves the gripper of the robot toward the point which is 1 m away from start point on the working surface with 0.1 m/s velocity. It can be served as one example of this situation to carry an assembly part from stack region to assembly region. Grasping and locating the part need to be controlled precisely and carefully but, in the middle of these operations, the carrying process is less critical since we assumed that obstacles are eliminated. So in this case, the response is more important than the precision. In this point of view, the velocity of the system which selected 30 milliseconds as the fixed holding time represents too careful behavior. When the holding time is 200 milliseconds or variable, the system is more rapid to respond. The proposed system adjusts its variable holding time to 380 milliseconds as shown on Fig. 8 (d). Secondly, the task to locate the robot on target position precisely was executed. The operator moves the gripper of the robot to target point which is 0.35 m away from start point on the working surface precisely. The position of the robot is plotted on Fig. 9. The distorted synchronization of the 200 millisecond holding time system makes it difficult to converge on the desired target. Though the system having 30 millisecond holding time can place the robot on the target precisely, yet it takes too much time. The motion of the proposed system changes from the rapid to the careful gradually as the robot approaches the desired position. In other words, the variable holding time decreases rapidly as shown on Fig. 9 (d). As the result of applying the variable holding time, the robot can reach the target precisely and quickly. Thirdly, the operator gives 50 N vertical force to the object which is in contact with the robot. The force/torque sensor located between the sixth arm and the gripper measures actual force which is exerted on the object. RMS error values are represented on Fig. 10 when the force regulation control is performed. Since the robot is in contact with environment, the operation should be performed extremely carefully. In this situation, small holding time ensures precise control. It is clear that the error of the system which has 30 millisecond or variable holding time is smaller than the 200 millisecond holding time system s. The proposed system restricts its holding time to 20 milliseconds during this experiment to ensure precise control

714 as shown on Fig. 10 (d). 5. Conclusion The bilateral teleoperation system which exchanges velocity and force information via the Internet has been developed. This system employed the event based control to secure the stability against irregular time delay. Moreover, the variable holding time was introduced to make the system quick to respond. This scheme could be realized by implementing the fuzzy logic which maintains the proper holding time based on the characteristic of the present task for every event. The experimental results show that the proposed method secures stability and response simultaneously. References ( 1 ) Brady, K. and Tarn, T.J., Internet-Based Remote Teleoperation, Proceedings of the 1998 IEEE Int. Conf. on Robotics & Auto., Vol.1 (1998), pp.65 70. ( 2 ) Anderson, R. and Spong, M., Asymptotic Stability for Force Reflecting Teleoperators with Time Delay, The Int. Journal of Robotics Research, Vol.11, No.2 (1992), pp.135 149. ( 3 ) Goldberg, K., Gentner, S., Sutter, C. and Wiegley, J., The Mercury Project: A Feasible Study for Internet Robots, IEEE Robotics & Automation Magazine, Vol.7, No.1 (2000), pp.35 40. ( 4 ) Grange, S., Fong, T. and Baur, C., Effective Vehicle Teleoperation on the World Wide Web, IEEE Int. Conf. on Robotics & Auto., Vol.2 (2000), pp.2007 2012. ( 5 ) Niemeyer, G. and Slotine, J., Stable Adaptive Teleoperation, IEEE Journal of Oceanic Engineering, Vol.16, No.1 (1991), pp.152 162. ( 6 ) Sheridan, T.B., Telerobotics, Automation, and Human Supervisory Control, (1992), The MIT Press. ( 7 ) Bemporad, A., Predictive Control of Teleoperated Constrained Systems with Unbounded Communication Delays, Proceedings of the 37th IEEE Conf. on Decision & Control, Vol.2 (1998), pp.2133 2138. ( 8 ) Niemeyer, G. and Slotine, J.-J.E., Towards Force- Reflecting Teleoperation over the Internet, Proceedings of the 1998 IEEE Int. Conf. on Robotics & Auto., Vol.3 (1998), pp.1909 1915. ( 9 ) Xi, N. and Tarn, T.J., Action Synchronization and Control of Internet Based Telerobotic Systems, Proceedings of the 1999 IEEE Int. Conf. on Robotics & Auto., Vol.1 (1999), pp.219 224. (10) Xi, N. and Tarn, T.J., Stability Analysis of Non- Time Referenced Internet-Based Telerobotic Systems, Robotics and Autonomous Systems, Vol.32 (2000), pp.173 178. (11) Elhajj, I., Xi, N. and Liu, Y., Real-Time Control of Internet Based Teleoperation with Force Reflection, Proceedings of the 2000 IEEE Int. Conf. on Robotics & Auto., Vol.4 (2000), pp.3284 3289. (12) Passino, K.M. and Yurkovich, S., Fuzzy Control, (1998), pp.21 72, Addison Wesley Longman. (13) Asada, H. and Slotine, J.-J.E., Robot Analysis and Control, (1986), pp.51 70, John Willey & Sons.