A Novel Distributed Telerobotic System for Construction Machines Based on Modules Synchronization

A Novel Distributed Telerobotic System for Construction Machines Based on Modules Synchronization E. Rohmer and K. oshida Department of Aerospace Engineering Tohoku University Aoba 6-6 - 1, Sendai, 98-8579, JAPAN Email: {eric, yoshida}@astro.mech.tohoku.ac.jp Eiji Nakano Robot Technology Center Chiba Institute of Technology, 2-17-1 Tsudanuma, Narashino, Chiba 275-16, JAPAN Email: nakano.eiji@it-chiba.ac.jp Abstract On a scene of a natural disaster, rescuers require heavy duty hydraulic machines to answer their need of large forces, or to approach an unreachable spot. Those industrial vehicles are hydraulically actuated arms kinematically different, and equipped by specific tools to fit the task to accomplish. They require trained operators because of their complex use. As time and accuracy are primordial factors to save lives in such a situation, the use of automation and robotic concepts on the control of those machines becomes necessary. The standard approach is to use manual teleoperation based on a bilateral master/ slave system, to enhance the maneuverability of the machines, but their diversity is nearly never taken into account. We propose here, a modular platform based on an original synchronization of module states that allows easy manipulator changes and flexibility of use. We will describe the proposed concept of Intelligent Interface, and verify its utility and performances through some pertinent configurations and experiment. I. INTRODUCTION When a natural disaster such as earthquakes occurs, quick and efficient recovery programs to minimize the loss of human life and facilitate rapid restoration and reconstruction are necessary. This process requires precise tools, a large variety of heavy duty construction machines, effectively trained machines operators and the assurance that health and safety risks for the operator are reduced to a minimum. The use of telerobotic concepts offers the possibility of contributing to each of those requirements. Hydraulic machines are for most of them, similar to serial robots of different kinematics complexities, equipped by a specific tool to accomplish a specific task. But their traditional way of control is based on an independent joint control, where the operator is using a complex system of lever to operate separately each angle of the arm. The precise control of the trajectory of the tool requires a long training on machines. To be efficient, rescuers need to operate on simulated disaster scene. In some other cases, the operator needs to work from outside the cab, to get either another point of view, or to work from a remote safer place in case of a hazardous working site. Regarding these problems, the design of a new architecture for civil engineering machines must take into account the following constraints: Maneuverability of the machines: how to cope with the complex system of levers and the complexity of the tasks, Interoperability: how to add remote control features to the machines; replace a real machine by a virtual one, etc., Diversity: how to deal with the numerous kinds of civil engineering machines and tasks. In this paper, we present a solution based on a new software architecture to allow the use of robotic and automation concepts with the civil engineering machines, that deals with those three issues. This architecture is based on the integration of a computer system using a modular approach based on synchronization of virtual images of manipulators, instead of a more common master-slave approach. In the first part, we show the problem inherent to traditional hydraulic machine use. In the second part, we describe the concept of the modular intelligent interface and describe its implementation. Finally, we show performances and advantages of the proposed platform, though some pertinent configurations. II. MANEUVERABILIT AND DIVERSIT OF MACHINES Here we detail the problem inherent to conventional machines control that requires long training, to show the necessity of finding an enhanced way to manipulate civil engineering machines. At the same time, one should be able to notice the diversity of tasks and machines available to accomplish the very different kind of works that require as well a special training. A. Maneuverability of the conventional machines Due to their counter intuitive system of control, the operator needs a special training for each kind of machine, and each different task to be quick and efficient. 1) Controlling the position and orientation of the tool in the space: The conventional device to control hydraulic machines is corresponding to a system of levers directly connected to the valves that controls the pressure in the cylinders. The operator is then controlling each joints of the hydraulic arm separately,

Bucket Fig. 1. Dipper Boom Boom Bucket Cab A slope task with a conventional excavator Dipper Cab positioning and orienting the tool in not intuitive joint mode. For complex motions required by a task like digging a slope with a regular excavator, the operator has to smoothly change three of the angles of the machine during the task, by operating gradually changing diagonals with the system of levers (Fig. 1). For other kind of tasks, on kinematically different machines, the problem remains the same. As some of them have more than 1 degrees of freedom, the operator uses a complex control box. The difficulty of controlling quickly and properly a hydraulic arm increases drastically with the number of degree of freedom of it, and is depending of the kind of task to do. Another maneuverability issue concerns the repetition of a task. For a mass excavation that consists in removing a huge amount of soil, the operator needs to repete the sequence of digging an area to fill up the bucket, and dump it into a truck a great number of time. The operator gets tired, as he has to follow the same pattern several times, and can be a source of accidents in the working area. 2) Controlling the force in the tool: On a conventional machine, the operator has few feedbacks. The visual and audio one he gets from his cab allows him to have an idea about the forces inside the tool. The machine loading or the force inside a gripping tool can only be determined by experienced operators relying on such cues as cab motion and engine audio [1]. Those parameters are not reliable and can not assure safety on a disaster field while hauling crumbles to rescue someone trapped under it. With a more reliable way of getting his feedback, the operator could accomplish this duty faster and reducing the risks, as it has been proved for long that an assembly task for instance, can be completed much faster and more accurately when the operator has a feel for forces and torques caused by contact [2]. B. Modularity and Interoperability The previous section showed that a lot of different machines and tasks need special training for the operators to let them become efficient. The rescuers are then confronted to a modularity problem as they have to be trained on several kinds of machines to accomplish different tasks. Sometimes, an interoperability problem between the control box and the machine appears. The operator needs to be extracted from the cab in order to work from outside of it. The worksite can be dangerous for him or sometimes the task requires him to get a different point of view than from the cab III. CONCEPT OF MODULAR INTELLIGENT INTERFACE Numerous researches [3] [4] [5] have been conducted to implement bilateral master-slave architectures to control a specific industrial machine to improve its maneuverability providing force feedback, and Cartesian control. Recently, some of them like Carnegie Mellon university automatic excavator [6] or Tmsuk T-52 Enryu hydraulic two armed robot [7] are providing solutions that handles interoperability allowing remote work, and very few of them like [8] or [9] lead to application that handle the diversity of machines. Our proposed platform offers a solution that manages maneuverability, interoperability and modularity issues in order to offer several configurations to accomplish any kind of tasks on several types of machines. A. The Intelligent Interface specifications We introduced in our research the concept of Intelligent Interface between master manipulators and slave manipulators using a Supervisory Control. This interface is not a single Indpdt Class Function Content Manip. Defining Manip. s status cm, tm, cs, tm Type and type. c:current, t:target, m:master, s:slave Motion Specify a joint angle or angu- θ =[θ 1,.., θ n] or θ = lar speed Specify a position or linear speed and orientation of the end effectors Transmit raw manip. data (i.e. not considered in the kinematic chain, Buttons states or Commands Returns an error or status message Sensor Transmit force and torque from joint[i] Transmit force ans torque data of the end effector sensor Inputs from the manipulator s environement (promimity, laser range,...) [ θ 1,.., θ n] = [x, y, z, u, v, w] Ẋ =[ẋ, ẏ, ż, u, v, w] A[i] = Axis values, B[i] = Button state, Cmds msg = text message F j [i] =[f, n] F = [fx,fy,fz, nx, ny, nz] TABLE I NORMALIZED PROTOCOL OF COMMUNICATION BETWEEN MODULES onboard computer, but a system of computer with a variable complexity (one or many communicating computers connected to each others by a Local Area Network (LAN)), and has the following characteristics: It is an interface between master and slave manipulator that needs an abstracted a communication layer with a normalized protocol of communication, described in Table I It owns a current virtual image (geometry, specifications, current angular configuration etc.) of both master and

Master Modules θcm A[1..i] B[1..j] Fcm Intelligent Module FKm(θcm)=cm FKs(θcs)=cs Master Status Report Current Virtual Images Status Report θcs A[1..k] B[1..l] Fcs msg Modules Master Manipulators serial, FPGA,... Master process (Wireless) LAN Ftm High level Control Ftm θtm Target Virtual A[1..i] Images Master Target Brain process B[1..j] Command msg IKm(tm)=θtm Intelligent Interface ForceGain(Fcs)=Ftm ts (Wireless) LAN Target Command IKs(ts)=θts process PCI, FPGA,... Fts θts A[1..k] B[1..l] Cmds msg pan-tilt cam Manipulators Fig. 2. Virtual image synchronization slave manipulators and the effects on their environment on them, updated as often and as fast as the communication media allows it. And target virtual images of the manipulators that saves the target to reach, A higher level of control to synchronize the current images of the manipulators to store them in the target one, with computation capabilities as: Forward, Inverse, Velocity Kinematics etc. required to be computed for the management of the control of both master and slave manipulators; Enhanced capabilities (Computer Aided Task Management) in order to let the user select different working mode to help him in his task (Cartesian, slope, pattern, etc. mode), Monitoring capability to display information from the master and slave manipulators and eventually visual feedback of the remote site, A modular design in order to deal with several master and slave configurations and communication mediums to accomplish different kind of tasks. A lower level of control for each manipulators close to them, to handle the control of the actuators in order to reach the target provided by the higher level, and eventually to accomplish automated basic tasks considering the environment, to assure a minimum supervisory control of the tasks. To implement this concept of Intelligent Interface, with all the described features, a modular approach using client-server architecture was investigated. B. A modular approach based on client-server architecture: Synchronization of virtual images Figure 2 shows an overview of the Intelligent Interface. Its implementation is composed of three modules (master, slave, and intelligent). A module is any kind of manipulators (real manipulator or 3-D simulation of it,...) with a dedicated hardware and an attached process. So each module owns a process whose function is to manage communication between the other modules and its attached manipulator communication and control. The Master and the slave module respectively handle the low level control of their attached manipulator, and the Intelligent module handles the synchronization of the status of them thanks to High-level control it owns. The software implementation is then three separates processes (master, slave and brain) that can be run on different computers connected to the network, using a client-server application and exchanging messages with the normalized protocol of communication. The Brain process is the server application that owns the virtual images of the manipulators current and targeted status, and the high-level control. It manages synchronization of clients and the communication between them through its instance of class. The master and the slave processes are client applications that connect to the brain process through the (wireless) LAN to send the status of their attached manipulators, and receive from it the reference target to reach. They are managing the low-level control of their attached manipulators through fast es. The first step to start a task is to define the topology of the Intelligent Interface, through a system of scriptable options for each process. When the Brain process is started with the appropriate options to define which manipulators are used, and which machine owns the master and slave modules, it is waiting for the master and slave process to connect it. Once identified, if the connection succeeded, the work can start. Each manipulator is sending reports on their status

(angular configuration, sensor data, buttons status, user defined info,...) to the brain process. The operator is perturbing the master manipulator in the same way than the slave one is perturbed by its environment. Once the brain process receives the latest status from a module, it is updating its corresponding current image. The end-effector position and orientation of each manipulators using the forward kinematics models implemented in the images are then calculated from the current angular configuration of the manipulators (θ cm and θ cs ) and the sensors information are saved in their virtual image. When both current images are updated by new information, the Highlevel control can synchronize the two current images to define what the next target to reach for each manipulator is, following implemented operating modes and control policies described in III-C. The result of the synchronization are two set of new targets to reach ts and tm (position and orientation of the tool frame) of slave and master manipulators and what force to reflect master side F tm. Those are stored in the manipulators target virtual images. Then, the inverse kinematics models described in [1] in the Intelligent Interface are converting the Cartesian space of the slave and master new target ts and tm, into their corresponding joint spaces θ ts and θ tm, and normalizing the force vectors F tm to fit the master manipulator force space. The sets of angles θ ts and θ tm and the force vector F tm become the new target to track and reach for the slave and master manipulator low-level of control. If a new target to reach is received by a module, the old one becomes obsolete. One should notice that other commands comming from the buttons status operator side are sent through the vector Cmds to the slave manipulator in order to control cab rotation or to drive the machine for example. Master Image θ cm F tm Joint Cartesian Slope Gs: speed gain Gp: position gain α : the angle of slope Fig. 3. Boundary Check Master model Boundary check FKm Cartesian Mode selection θ cm Force Feedback mode switch ForceGain Slope space Converter cm α Master model High Level Control Gp Gs Image IKs F cs. θ ts or. ts θ ts or ts Control selection Brain Process Mode and Control selector overview inside the brain process C. Control types and operating modes Depending on the task to accomplish, the operator will use a specific master and slave manipulator. Fig. 3 shows how he can select an operating mode and a control policy implemented inside the Brain process. To test the performances, we developed two control types that are applicable to three operating modes: Position Control: also known as zero order control. It refers to the control of the position of the manipulator joints or tool frame relative to an input signal coming from the master manipulator. Rate Control: This control policy also known as velocity or speed control refers to the direct control of the speed of a manipulator s joints or tool frame relative to an input signal provided by the master manipulator. Processes: brain master slave Computer running under RT Linux Intelligent Module PCI port PA1 controler FPGA torque sensor board module Fig. 4. serial pan tilt system sh4 FPGA Network cameras Torque sensor Experimental system for the onboard work Master Module pan-tilt joystick monolever Here are the implemented operating modes: Joint control mode: is controlling the movement of each section of the arm separately. The operator must acquire through practice a mental model of the inverse kinematics of the manipulator to manipulate the tool. In this control mode, one degree of freedom of the master manipulator is linked to one degree of freedom of the slave manipulator. Cartesian control mode: is also referred as coordinate control, coordinate motion control and boom-tip control. The Intelligent Interface is computing the inverse kinematics of the slave manipulator to map the manipulator s workspace into joint motions. The operator therefore does not need to develop a mental model of it and can focus on the task to perform. Slope control mode: is similar to the Cartesian mode but considering an angle of slope. In case of digging a well that would not be vertical, or in order to generate a slope with a fixed slope angle, this mode increases the productivity and accuracy of the task. A Cartesian

mode refers to a frame whose z-axis is normal to the ground. Here, the z-axis becomes normal to the slope. A horizontal command signal from the master manipulator is corresponding to a motion parallel to the slope ground of the tool frame. A vertical command signal one is corresponding to a motion of the tool frame normal to the slope ground. The control of the tool frame orientation is done in absolute angle i.e. in the same reference frame as the operator. To set this mode on, the operator needs to define the angle of slope by selecting two points in the slave workspace. IV. PLATFORM EVALUATION THROUGH SOME PERTINENT CONFIGURATIONS OF THE SSTEM In this section we will present some pertinent configurations of the system that are helpful to accomplish different kind of jobs. We will show as well the performances of the proposed platform in each configuration, through some experiences. A. Onboard task to show the maneuverability enhancement This configuration depicted in Fig. 4 is related to the manual teleoperation studied by Nakano in [4]. The operator is inside the cab of the machine and is controlling it with an enhanced maneuverability. The configuration is using an onboard computer that runs the three processes. The operator is using a Slope mode operating mode and speed control and gets some force feedback. A camera at the end effector, and 3D visualization model of the manipulators are helping the operator in his task. For this configuration, we are using a PA1 robot to replace the hydraulic machine to grab a ball, test the force feedback, and follow a slope in different controls and operating modes. Fig. 5 shows some results obtained using the Intelligent Interface while following a slope in different modes and control policies in order to evaluate the system performances of tracking in each mode, and in order to determine which combination of policy and mode was the most accurate to follow a pattern. As one could expect, the slope mode gives the most accurate results. Speed control give the best tracking result as, once the operator sets a proper angle, he just has to control the speed of the tool, in a horizontal motion of the master manipulator. In position control, he needs to continue his horizontal motion in order to follow the pattern. The results provided by a Cartesian mode are less accurate. In speed mode the operator has to found the appropriate diagonal. Once he finds the correct angle with the master manipulator, he does not have to change the position of the manipulator to continue the task. But if he needs to reduce or increase the speed of the tool frame, or rotate the bucket, it becomes more difficult to follow the pattern. In case of position control in Cartesian mode (the worst case), the operator have trouble to avoid the stair effect while following the slope. The author did not compare those results with joint mode, as it is obvious that for a non-experienced operator, following a slope requires a long training, as explained earlier. Computer running under Linux Intelligent Module LAN port Processes: brain master serial master manipulator controller sh4 fpga Master Module master manipulator 14 12 14 12 Wireless LAN 9 6 3 14 speed control - cartesian mode 3 6 9 12 14 9 6 3 14 position control - cartesian mode 3 6 9 12 14 Processes: slave LAN port module simulation of a 5DOF komatsu Keyboard and mouse to navigate in the 3D environment This computer is running under windows P 12 12 Fig. 6. Experimental system for the remote work 9 6 3 position control - slope mode 3 6 9 12 14 Fig. 5. 9 6 3 speed control - slope mode 3 6 9 12 14 slope task tracking results in different operating modes B. Remotely control task to show the interoperability enhancement This configuration is allowing a remote work to accomplish a task from outside the machine. There, two computers are linked through a wireless LAN 82.11 g (1 onboard 1 remote) and are running the three processes as depicted in Fig. 6, a pan tilt camera can provide visual feedback to emulate the sight of the operator from inside the cab or on the worksite, and a 3D visualization model shows the operator the geometry of

the manipulator. For this experiment, we are using simulation of a komatsu 5 DOF machine in order to evaluate the respond of the platform. The results of the tracking experience in Fig. 7 shows the accuracy of the system through a wireless network. The operator could accomplish a task of grabbing a box and dropping it on another one, in the simulator of the komatsu machine equipped with a gripper in Cartesian mode, with force feedback, with the monolever master manipulator, without any problem. This kind of task would require some training in joint mode in a traditional way. No perceptible delay was encounter during the task using the wireless LAN. θ (deg) 4 35 3 25 2 15 1 5 Input angle boom angle -5 1 2 3 4 5 6 7 8 9 h (m) 4 3.5 3. 2.5 2. Boom Cylinder Input signal slave cartesian axis 1 θ (deg) 5-1 -15-2 -25-3 Input angle Stick angle 1.5.5 1. -.5 1 2 3 4 5 6 7 8 9 1 1 2 3 4 5 6 7 8 9 1 Fig. 7. Cartesian in -5-35 1 2 3 4 5 6 7 8 9 h (m) Stick Cylinder 4 Input signal 3.5 slave cartesian Z axis 3. 2.5 2. 1.5 1. Cartesian in Z Response of a slave manipulator in a remote work V. CONCLUSIONS AND ONGOING WORK We successfully implemented a distributed telerobotic system based on a modular software architecture that is: A new approach for the design of the controlling system of the hydraulic machines that deals either with manoeuvrability of the machines, and diversity of the tasks to do and machines configuration, A multiplatform modular solution that allows several configurations for testing the control of several machines, training the operator on simulators, accomplishing a task remotely, and monitoring a task. An open object oriented architecture for a fast implementation of new manipulators, with a normalized protocol of communication based on an abstracted communication layer for an accurate interoperability between the components of the system. The first point allows any rescuers to perform difficult tasks faster and better without specific training on different kind of machine used on a disaster field. The second point allows simulation of disaster. The last point considerably increases the developer s productivity, by re-usability of the code and thanks to the standardized protocol between the modules. 1 Generally, manual teleoperation using a bilateral master slave control is the solution used by robotic systems for construction machines. They deal with the first point for only one specific application on a specific machine. The superiority of our system on the other one is that its open modular architecture allows the use of any hydraulic machine, and the possible configurations allows training, simulation, remote work over the Internet or a (wireless) local area network. The flexibility allowed by the modular system does not exist for most of the other solutions. The unmanned construction using teleoperated humanoid robots that can handle any kind of machines designed for human operators have restricted features and cannot enhance the maneuverability The ongoing research concerns first manual teleoperation and the development of a new slave module and its low level control, based on a reversed engineered 5 DOF Komatsu excavator. The second one concerns a Human-Machine interface for the automation of mechanical earth work, specifically automatic excavation, loading and hauling. REFERENCES [1] N. R. Parker, S. E. Salcudean, P. D. Lawrence, Application of Force Feedback to Heavy duty Hydraulic Machines In Proc. IEEE Robotics and Automation Conference, Atlanta, May 1993 [2] D. Stokic, M. Vukobratovic, and D. Hristic, Implementation of a force feedback in manipulation robots, International Journal of Robotics Research, vol. 5, pp. 66-67, Spring 1986. [3] M. Ostaja-Starzwensky, M. Skibiniewski, A Master- Manipulator for Excavation and Construction Tasks, Robotics and Autonomous Systems, 1989 [4] E. Nakano, N. Tsuda, K. Inoue, K. Kayaba, H. Kimura, T. Matsukawa, Development of an Advance Way of Improvement of the Maneuverability of a Backhoe Machine, 9 th ISARC, June 3-5, Tokyo 1992 [5] P.D. Lawrence, B. Sauder, U. Wallersteiner, and J. Wilson, Teleoperation of forest harvesting machines, in Robotics in Forestry, (Vaudreuil, Quebec), pp. 36-39, September 199 [6] A. Stentz, J. Bares, S. Singh, P. Rowe, A Robotic Excavator for Autonomous Truck Loading, Autonomous Robots Vol. 7, N. 2, pp. 175-186, 199 [7] http://www.enryu.jp/ [8] K. okoi, K. Nakashima,. anagihara, A Teleoperated Humanoid Robot Drives a Backhoe in the Open Air, Proc. of the 23 IEEE/RSJ Intl. Conference on Intelligent Robots and Systems IROS23, Las Vegas, October 23 [9] T. Miyata, T. Sakaki, K. Kawashima, Development of a remote Control System of Construction Machinery Using Pneumatic Robot Arm. Proc. of the 24 IEEE/RSJ Intl. Conference on Intelligent Robots and Systems IROS24, Sendai, Japan, Sept. 28 - Oct. 4 24 [1] E.Rohmer, T. Takahashi, E. Nakano, Modular Intelligent Interface to Assist Human in Operating Construction Machines, Doctoral dissertation, Graduate School of Information Science, Tohoku University, 25