A VIRTUAL REALITY TELEOPERATOR INTERFACE FOR ASSEMBLY OF HYBRID MEMS PROTOTYPES

Transcription

1 Proceedings of DETC ASME Design Engineering Technical Conference September 13-16, 1998, Atlanta, GA DETC98/MECH-5836 A VIRTUAL REALITY TELEOPERATOR INTERFACE FOR ASSEMBLY OF HYBRID MEMS PROTOTYPES JOSEPH ALEX BARMESHWAR VIKRAMADITYA BRADLEY J. NELSON Research Assistant Research Assistant Assistant Professor Mechanical Engineering Mechanical Engineering Mechanical Engineering University of Illinois at Chicago University of Illinois at Chicago University of Illinois at Chicago Chicago IL, Chicago IL, Chicago IL, jalex1@uic.edu bvikra1@uic.edu bnelson@uic.edu ABSTRACT In this paper we describe a teleoperated microassembly workcell that integrates a VRML-based virtual microworld with visual servoing micromanipulation strategies. Java is used to program the VRML-based supervisory interface and to communicate with the microassembly workcell. This provides platform independence and allows remote teleoperation over the Internet. A key aspect of our approach entails the integration of teleoperation and visual servoing strategies. This allows a supervisor to guide the task remotely, while visual servoing strategies compensate for the imprecisely calibrated microworld. Results are presented that demonstrate system performance when a supervisor manipulates a microobject remotely. Though Internet delays impact the dynamic performance of the system, teleoperated relative parts placements with submicron precision is successfully demonstrated. 1. INTRODUCTION With the development of increasingly complex microelectromechanical systems (MEMS), the assembly and testing of prototype devices requires increasingly sophisticated micromanipulation techniques. Although monolithic microfabrication has been a requirement for the commercial success of MEMS devices in the past, such as pressure sensors and accelerometers, the future of MEMS will require increasingly sophisticated assembly of hybrid components. For example, if a microdevice must be made of different materials, has a complicated geometry, or is manufactured using incompatible processes, assembly is required. With these developments on the horizon, the need for sophisticated teleoperated micromanipulation environments for the assembly and testing of prototype devices is apparent. Our past work has demonstrated the advantages of applying visual servoing techniques to micropositioning of MEMS components for purposes of assembly (Vikramaditya and Nelson 97). Because of the high precision required for parts placement in assembling MEMS devices (often requiring submicron precision for relative parts placement), conventional open loop precision assembly devices used in industry are inadequate (Slocum 92). However, we have shown that visual servoing strategies combined with high resolution optical systems are able to achieve the required submicron precision. This is due to the ability of closedloop vision feedback to compensate not only for inaccurate sensor and manipulator kinematic models, but also the difficulties associated with thermal expansion and the complex microphysics that are inherent to the microworld. In this paper, we propose a framework for visually servoed teleoperated micromanipulation that uses an expectation-based approach to task execution. A key aspect of our approach entails the integration of a virtual microworld with visual servoing strategies to perform the task in the real microworld. The virtual microworld is represented using VRML (Virtual Reality Modeling Language). A supervisor interacts within this virtual microworld by selecting and dragging objects to be manipulated. As objects are moved, the changing desired visual representation of the microworld is determined by the VRML environment. This constantly changing desired visual representation is transmitted as a vector of reference feature states to a visual servoing agent that executes the required visually servoed micromotion. The

2 interface between VRML, the supervisor, and the visually servoed microassembly workcell uses the Java programming language. This provides a significant level of portability across computing platforms for providing the graphical user interface, i.e. the virtual microworld, to the supervisor and for communication, thus allowing for remote teleoperation over the Internet. Experimental results are presented that demonstrate system performance. Though Internet delays do impact the dynamic performance of the system, we are able to demonstrate teleoperated relative parts placements with submicron precision. Within this paper, we first present previous work related to microassembly and teleoperation using the Internet. In Section 3, we describe our system framework, and in Section 4 we discuss our hardware and software implementation. Section 5 presents experimental results, and Section 6 concludes the paper. 2. BACKGROUND 2.1. Microassembly manipulation probe glass fiber V-groove Figure 1. Manipulating a 308µm dia. glass fiber into a 270µm wide V-groove for constructing an electron column for a miniature scanning electron microscope (Feinerman et al. 92). Currently, microdevices requiring complex manipulation are assembled by hand using an optical microscope and probes or small tweezers, and is a crude form of teleoperated micromanipulation. For example, specially trained technicians use this technique to assemble precision optical and magnetic devices (Yamagata and Higuchi 95). In the Microfabrication Applications Laboratory, we have assembled many different microdevices by hand using optical microscopes. Some of the devices include miniature fiber optic assemblies, micropumps, and electron columns for miniature scanning electron microscopes (Feinerman et al. 92) (see Figure 1). A primary goal of this project is to develop more robust teleoperated micromanipulation strategies for these types of assembly tasks. Many researchers are actively pursuing strategies for manipulating micron sized objects for various applications. For example, researchers have used feedback from a scanning electron microscope (SEM) to teleoperatively guide micromanipulation (Sato et al. 95); look-and-move techniques for remote teleoperation of micro/milli sized structures have been developed (Hannaford et al. 97); vision based methods have been proposed (Koyano and Sato 96) (Sulzmann et al. 97) (Vikramaditya and Nelson 97); and microassembly workcells are being built (Menciassi et al. 97), to name a few of the efforts in this area. Our approach uses visual servoing techniques, as opposed to lookand-move, to guide and provide feedback on relative parts placement over large ranges. The visual servoing approach is integrated with a virtual microworld for providing a graphical user interface to the supervisor performing the task. The 3D representation of the microworld is also used to develop an expectationbased framework for micromanipulation, as will be discussed in Section Teleoperation Using the Web and Java One of the first and most popular web-based teleoperation systems is described in (Goldberg et al. 95). This system allowed any remote web user to position a manipulator arm and discharge compressed air to uncover sand covered objects. An image was sent to show the effect of the air discharge. Due to the limitations of the HTTP protocol at the time (Java was not used), only openloop motion commands could be sent to the manipulator and single snap shots of the workcell were returned as feedback. In (DePasquale et al. 97), the issue of teleoperation of a robot using a Java applet running on a web page from anywhere on the Internet was first explored. A painting robot was used for demonstration purposes, though a VRML interface was not incorporated. The possibility of combining Java and VRML utilizing CGI scripts (Common Gateway Interface) is mentioned in (Hirukawa et al. 97), though a full implementation is not described. Our implementation of a remote teleoperation system is more general in that a Netscape web browser is used, rather than a specific software tool such as CGI. This paper describes our successful integration of a remote supervisor, VRML, and our micromanipulation workcell through the use of Java VRML VRML 2.0, Virtual Reality Modeling Language, is a scene description language that can be used to describe 3D models of objects and scenes with the capabilities of interactive operations on them. These models can be viewed using a web browser with a plug-in for VRML 2.0. The capabilities of navigation and viewpoints are built into VRML and, thus, can be used as a graphical display engine. VRML inherently supports an event driven model, which allows routing of the field values inside the

3 nodes to other values thus changing the scene. To communicate with the real world, a programming language is needed to link VRML with the real world. For this purpose, Java is used. This requires that the VRML browser supports the Java-VRML interface. The two existing interfaces are the Script Authoring Interface and the External Authoring Interface (EAI). We use EAI to link Java and VRML due to implementation issues on our chosen supervisor interface development platform, a Silicon Graphics machine. 3. SYSTEM FRAMEWORK Our overall system framework is based on the concept of an expectation or verification based approach to scene understanding (Dickmanns 92) (Roth and Jain 91). A key point of both the expectation and verification-based approaches is that strong internal models of the recent world state are maintained. Neisser s view of the human perceptual cycle (Neisser 76), as Jain points out (Jain 89), is similar in many ways to a verification or expectation based approach. Figure 2 shows a modified representation of Neisser s perceptual cycle. This figure illustrates our view of the relationship between the VRML representation of the microworld, the real world, the visually servoed micromanipulator, and the supervisor. The counter-clockwise flow of information represents the cyclical nature of the system; sensory data updates the VRML representation which accepts plans from a supervisor; VRML s desired world view (in terms of image-based visual features) guides the visually servoed micromanipulator; which provides sensory data obtained from the real world to VRML. This cycle illustrates the interaction between perception of the world, actions taken within this world by the visually servoed micromanipulator, and plans made about the world by the supervisor. Supervisor Sensory Data VRML Real World Control Directives Sensing and Actuation Visual Servoing Agent Figure 2. A modified perceptual cycle for visually servoed manipulators VRML - Supervisor Interface The VRML world contains 3D models of objects and manipulators in the real world. Through a camera-lens model approximately equivalent to our microscope-ccd system, a virtual image of the scene is created; for our experiments a 50µm diameter polysilicon gear and a microgripper are modeled. The supervisor moves a microobject by clicking on its corresponding virtual object on the screen and dragging it. This motion in VRML creates a change in image plane coordinates for visual features located on the object. As fast as the browser and the Internet will allow (currently approximately 10Hz), these new desired feature states are transmitted to the visual servoing system located at the microassembly workcell Visual servoing with an optical microscope The job of the visual servoing system is to accept a vector of desired feature states from VRML and determine a motion control command that will result in the desired image despite system disturbances. In formulating the visual servoing component of our system, the Jacobian mapping from real world task space to CCD sensor space is used. We desire a Jacobian for the camera-lens system of the form x S = J v ( φ)x T where x S is a velocity vector in sensor space; J v ( φ) is the image Jacobian matrix and is a function of the extrinsic and intrinsic parameters of the vision sensor as well as the number of features tracked and their locations on the image plane; and X T is a velocity vector in task space. For a microscope that is allowed to translate and rotate, J v ( φ) is of the form J v where s x and are pixel dimension on the CCD; the total linear magnification m is given by m = h 2 h 1 = ( gc) ( f o 'f e ') (3) where g is the optical tube length; and c is the distance that the CCD lies behind the posterior principal focal plane of the eyepiece and is shown in Figure 3. Generally several features are tracked. Thus, for n feature points the Jacobian is of the form. where J i (t) is the Jacobian matrix for each feature given by (2). A complete derivation of (2) can be found in our previously published work (Vikramaditya and Nelson 97). The state equation for the visual servoing system is created (1) Z m y m c s s s x x s = x (2) m Z c m x s s x J v = J 1 ()... t J n () t T (4)

4 by discretizing (1) and rewriting the discretized equation as x( k + 1) = x() k + TJ v ()u k () k (5) where x(k) R 2M (M is the number of features being tracked); T is the sampling period of the vision system; and T u() k = X T Y T Z T ω XT ω YT ω ZT is the velocity in the task space of the manipulator end-effector. The Jacobian is written as J v () k in order to emphasize its time varying nature due to the changing feature coordinates on the image plane. The intrinsic parameters of the camera-lens system are constant for the experimental results to be presented. h 1 objective f o f o g h f e mechanical zoom elements tube optics f e c h 2 4. IMPLEMENTATION 4.1. Hardware Implementation Experiments were conducted with the microassembly workstation shown in Figure 4. The workstation is centered around a Wentworth MP950 Integrated Circuit Probe Station with a Mitutoyo FS60 optical microscope. The probe station has been retrofitted for motion control using high precision Kollmorgen brushless DC motors and a Queensgate NPS3330 3DOF piezoactuated nanopositioner. Image processing and visual servoing control calculations were performed with a vision system consisting of a digitizer and multiple TMS320C40 DSP s. The vision system is able to track up to 5 16x16 feature templates at 30Hz. The hardware architecture of the visual servoing system is shown in Figure 5. Two micromanipulators have been integrated with the system, a Wentworth HOP 2000 and a Sutter MP285 micromanipulator. Both manipulators provide 3 DOF. Silicon micro vacuum grippers have been fabricated in our Microfabrication Applications Laboratory and are used for grasping objects with dimensions greater than 10µm. Figure 3. Ray diagram for a microscope optical system. Optimal control techniques are used to arrive at the following expression for the control input. T 1 T u() k = ( TJ v ()QTJv k () k + L) TJv ()Qxk k [ xd ( k + 1) ] (6) The vector x D ( k + 1) is the vector that is sent from the virtual desired image created by VRML. This vector represents the desired image that the supervisor wants the visual servoing system to achieve through motion of the probe stage or micromanipulator. The weighting matrices Q and L allow the user to place more or less emphasis on the feature error and the control input. Extensions to this system model and control derivations that account for system delays, modeling and control inaccuracies, and measurement noise have been experimentally investigated (Papanikolopoulos et al. 92). The measurement of the motion of the features on the image plane, where the features are described by x(k), must be done continuously and quickly. The method used to measure this motion is based on an optical flow technique called Sum-of- Squares-Differences (SSD). The method assumes that the intensities around a feature point remain constant as that point moves across the image plane. The displacement of a point p a =(x S,y S ) at the next time increment to p a =(x S + x, y S + y), is determined by finding the displacement x=( x, y) which minimizes the SSD measure e(p a, x). A pyramidal search scheme is used to reduce the search space. A more complete description of the algorithm and its implementation can be found in (Nelson et al. 93). Figure 4. Microassembly workcell 4.2. Software Implementation The communication implementation among the client and the visual servoing agent is shown in Figure 6. The main system components consist of the remotely located client and the PCbased visual servoing system that controls the microassembly workcell VRML Figure 7 shows the VRML interface provided to the supervisor. Shown in the figure is the 3D virtual model of a 50µm polysilicon gear and a manipulation probe. A live view of the image

5 Microscope Camera C40 Vision System PCI/ISA Bus virtual image very closely. Initially, the position of the images are set to the corresponding VRML coordinates by using the Initialize button in the applet. Tim40 Carrier Probe Station Stage and Micromanipulators PMAC-PC manipulation probe 50µm dia. gear Figure 5. PC-based visual servoing hardware architecture. Supervisor Microworld Java applet Event handler Client Browser with VRML plug-in Java Socket Server Internet PC Visual servoing system: vision system motion control Application Server (Visual C++) Figure 7. Supervisor interface as shown in VRML running under Netscape The upper left window is a live image transmitted over the Internet by the microassembly workcell. Figure 6. Communication scheme for remote teleoperation using VRML and Java. can also transmitted over the Internet. The supervisor moves a microobject by clicking on its corresponding virtual object on the screen and dragging it. This motion in VRML is detected using the PlaneSensor node attached to the virtual image. This changes the translation field value of the image and sends out the translation_changed EventOut. The particular eventout triggers the callback function in the EAI applet. The function transforms the VRML coordinates to the pixel coordinates of the camera image using a camera-lens model and sends the data through the socket connection to the application running on the remote server. Due to security restrictions, an applet can open a socket connection only to the server from which it is downloaded. Here, the application running on the SGI accepts the connection and opens another socket connection to the PC which transforms the pixel coordinates to the actual movement to the controller. The controller sends the data to the workcell to affect the movement. As the micropart moves, its current position is tracked by the vision system and is sent back to the EAI applet through the application on the server. Upon receiving the pixel coordinates, the applet performs the transformation to VRML coordinates and the model of the real micropart is updated in the VRML world by routing an EventIn to the translation field of the realgear node. Since the PlaneSensor outputs events at nearly a continuous rate (10 to 15 Hz), the real image follows the desired EAI The Java-VRML interaction takes place through the EAI supported by CosmoPlayer 1.02 for IRIX. This gives the functionality to extend the features of VRML by adding the power of a programming language, Java. EAI allows an external program to access the nodes in a VRML scene using the existing VRML event model. The communication takes place with the browser plug-in interface that allows embedded objects on a web page to communicate with each other. We use a Java applet embedded in an HTML page, which also contains a VRML world. The interface allows access into the VRML scene by reading the last value sent from eventouts of the nodes inside the scene upon getting notification when these events occur. These can be used to modify the fields of the nodes inside the scene by sending events to eventins of nodes inside the scene. For this, the applet has methods (user-defined functions) that are called when the specified eventout occurs. A method is registered with an eventout of a given node and is called when the specific eventout event is generated. The framework provides a tool for the user to interact directly with the real world and receive feedback on the manipulated world. The ability of Java bytecode to run on different platforms makes it possible to view the HTML page and perform teleoperation across the Internet from a client side machine with the plug-in.

6 5. EXPERIMENTAL RESULTS The workcell is remotely teleoperated over the Internet under the web browser Netscape running VRML. We have used a Silicon Graphics O2 and various PC s running Windows95 for providing a virtual environment to the supervisor. We have also performed remote teleoperation from West Lafayette, Indiana of the microassembly workcell in Chicago. Experimental results demonstrating system performance are shown in Figure 8. The top two figures show object location in x, and y image coordinates. The solid curves represent VRML virtual image coordinates as a supervisor moves a virtual 3D object in an approximate circle, while the dashed curves show the delayed CCD image coordinates resulting from these commanded motions. A time delay of approximately 260ms was present when remotely teleoperating the system over the Internet while the remote host was located on the same hub as the microassembly workcell. From the plots one can see good tracking performance of the real image with the desired image created within VRML. A 20x objective lens was used to collect these results. Using microfabricated calibration grids, we estimate the relative positioning precision that is attainable with this optical configuration is approximately 0.4µm. 6. CONCLUSION In order to develop more complex hybrid MEMS devices, teleoperated micromanipulation strategies must be enhanced. The integration of a complex virtual environment with visual servoing strategies provides a remote environment with the tools for developing complex micromanipulation strategies. By incorporating a VRML environment with a Java-based interface, we are developing a remotely teleoperated micromanipulation system that can be operated over a variety of platforms. To date, our system has demonstrated submicron precision in relative parts placement using a variety of remote platforms, including an SGI O2 and a variety of PC s running Windows95. x (pix) y (pix) (sec) (sec) 7. ACKNOWLEDGMENTS This research was supported in part by the National Science Foundation through Grant Numbers IRI , CDA , and IRI , by the Office of Naval Research through Grant Number N , and by DARPA through Grant Number N µm 8. REFERENCES DePasquale, P., Lewis, J. and Stein, M. R., 1997, A Java Interface for Asserting Interactive Telerobotic Control, Proc. of SPIE, Vol. 3206, Dickmanns, E. D., 1992, Expectation-based Dynamic Scene Understanding, in Active Vision, Blake, A., and Yuille, A., ed., , The MIT Press, Cambridge. Feinerman, A. D., Crewe, D. A., Perng, D. C., Shoaf, S. E. and Crewe, A. V., 1992, Sub-centimeter micromachined elec- (sec) Figure 8. Experimental results showing object location in VRML pixel coordinates (solid) and CCD image coordinates (dashed) in x and y, and X (solid) and Y (dashed) motion of the object in the real world.

7 tron microscope, J. Vac. Sci. Technol. A, 10(4), pp Goldberg, K., Mascha, M., Gentner, S. and Rothenberg, N., 1995, Desktop Teleoperation via the World Wide Web, in Proc IEEE Int. Conf. on Robotics and Automation, Hannaford, B., Hewitt, J., Maneewarn, T., Venema, S., Appleby, M. and Ehsresman, R., 1997, Telerobotic remote handling of protein crystals, Proc IEEE Int. Conf. on Robotics and Automation, Hirukawa, H., Matsui, T. and Onda, H., 1997, Prototypes of Teleoperation Systems via a Standard Protocol with a Standard Human Interface Proc IEEE Int. Conf. on Robotics and Automation, pp Jain, R., 1989, Environment Models and Information Assimilation, Technical Report RJ 6866(65692), IBM-Yorktown Heights. Koyano K. and Sato, T., 1996, Micro object handling system with concentrated visual fields and new handling skills, Proc IEEE Int. Conf. on Robotics and Automation, pp Menciassi, A., Carroza, M. C., Ristori, C., Tiezzi, G. and Dario, P., 1997, A workstation for manipulation of micron sized objects, Proc th Int. Conf. on Advanced Robotics, pp Neisser, U., 1976, Cognition and Reality, W.H. Freeman and Co., New York. Nelson, B. J., Papanikolopoulos, N. P. and Khosla, P. K., 1993, Visual servoing for robotic assembly, Visual Servoing Real-Time Control of Robot Manipulators Based on Visual Sensory Feedback, Hashimoto, K., ed., River Edge, NJ:World Scientific Publishing Co. Pte. Ltd. pp Papanikolopoulos, N. P., Nelson, B. J. and Khosla, P. K., 1992, Full 3-d tracking using the controlled active vision paradigm, Proc IEEE Int. Symp. on Intelligent Control (ISIC- 92), pp Roth, Y. and Jain, R., 1991, Verification versus Discovery in Vision-Based Systems, Technical Report CSE-TR , The University of Michigan. Sato, T., Kameya, T., Miyazaki, H. and Hatamura, Y., 1995, Hand-eye System in the Nano Manipulation World, Proc IEEE Int. Conf. on Robotics and Automation, pp Slocum, A. H., 1992, Precision Machine Design, Prentice Hall Ṡulzmann, A., Breguet, J. M. and Jacot, J., 1997, Micromotor assembly using high accurate optical vision feedback for microrobot relative 3D displacement in submicron range, Proc Int. Conf. on Solid-State Sensors and Actuators (Transducers 97), Vikramaditya, B. and Nelson, B. J., 1997, Visually Guided Microassembly Using Optical Microscopes and Active Vision Techniques, Proc IEEE Int. Conf. on Robotics and Automation, pp Yamagata, Y. and Higuchi, T., 1995, A Micropositioning Device for Precision Automatic Assembly using Impact Force of Piezoelectric Elements, Proc IEEE Int. Conf. on Robotics and Automation, pp