The European Robot Initiative for Strengthening the Competitiveness of SMEs in Manufacturing The robot capable of understanding human-like instructions
Contents of this Presentation Introduction Human robot interaction devices and methods investigated General topics: technologies, that are applicable in a variety of use cases Success stories: solutions for specific tasks that allow human-like instructions Summary 2
Common Robot Programming Online programming using the teach pendant Offline programming in virtual environments Requires training Requires experience Takes a lot of time on the robot (approx. 1 min / point) Requires training Requires experience Requires equipment Takes a lot of time Needs: Fast and intuitive Programming! 3
Outlines and Goals Use Case Analysis What data do we need? Interaction Devices How can this data be retrieved and conveyed (low-level)? Device Interface, Multimodal Interface How can the information be transferred? High Level Programming Methods How can the information be processed (high-level)? Success Stories Integration in Demonstrations 4
Use Case Analysis Use Cases Analysis of user scenarios and interaction methods Locksmith Schmidt Joiner Mohring Metal works Treffler Conducted SME visits: System integrator: Automazioni Industriali, Kine, Friul Robot Joiners workshop: Som, Mohring Locksmithery: Petry, Schmidt Other: K.Met, Europress, Treffler, SD GmbH, ZF-Lenksysteme, Suitable use cases Milling of wooden kitchen elements Sanding of wooden stair elements Marking sheet metal for punching Handling / Cooperation in ramp-up phase Welding and bending in small batch sizes 5
Overview on Methods and Devices Programming devices and methods in different areas have been investigated Multimodality: Tactile Verbal Visual Different Intelligence levels: Pure point to point teaching Integration of process know how High Level Programming Automatic Programming Investigation in test beds, integration in demonstrations 6
State of the Art: Survey and Assessment Rate and match interaction devices to use cases 7
State of the Art: Survey and Assessment Result: Five main requirements more intuitive definition of directions and orientations in 3D more intuitive definition of points, trajectories, surfaces and objects in 3D definition of process parameters, such as physical values, e.g., velocity or force need for small, wireless and hand-held devices that could be operated with one hand, e.g. PDAs command a robot without using hands and fingers 8
Evolution of Devices IPA InTeach IPA PDA Commercial HW Reis Flybook ADDF PDA Comau SafeGamePad Comau PDA KUKA 2D Visual servoing KUKA inertial jogging 1 Handed KUKA 3D (teach wand) Visual servoing KUKA new teach wand Anoto Pen KUKA Anoto Pen as teach wand ADDF Anoto Pen for CAD IPA Anoto Pen for Sketches IPA/ADDF speech interfaces Manual Guidance and Robot Handles (ABB, IPA, ) 9
Overview: General Topics Manual Guidance: Jogging the Robot Digital Pens: Understanding of Drawings Speech Interaction: Commanding the Robot High Level Programming: Abstract Task Specifications Ontologies: Understanding the Task 10
General Topics: Manual Guidance Distinction Low Level Force Control (ITIA CNR, COMAU, DLR) Manual Guidance (KUKA, ITIA, COMAU, ABB, IPA, ADDF) Programming by Demonstration (ITIA, ABB, KUKA, IPA, AAF) Results: Set up of test beds and demonstrators Interaction with safety aspects e.g. in collision situations 11
General Topics: Speech Interaction (1/2) What has to be done? Intuitive, hands-free communication Flexible set of commands that can be changed without hardware reconfiguration Main problems? Robustness, high recognition rate High noise level Direct effect no long processing Approach? Use of existing speech recognizers Definition of a benchmarking process Integration of different microphone types to achieve better signal to noise ratio 12
General Topics: Speech Interaction (2/2) Benchmark Tool has been developed Preparation of comparable test data Evaluation methods Throat microphone > 90% background noise cancellation Algorithms for mapping the frequency content of the throat microphone to standard speech signals Demonstrator Integration D1 (based on Microsoft SAPI) D3 and D4 (based on Sphinx4 framework) 13
Ontologies for Custom Programming Input: AppConf from ontology Intermediate: Transformations use an XSLT stylesheet, afterwards eps with interaction. Based on Apache FOP. Output: Visual layout and interaction as Anoto-Doc Web surface Voice XML (using TellMe) Ontolog Produc t data Process data Configu a ble part Rule s and modality-specific constraints XML appconf E-form Anoto P Gestures User inter action Val ues Produc t data and proces execution Sensor Noncong urable part Spoken dialogue (Voic exml/sa 14
Overview: Success Stories Robot Programming for Everyone Intuitive Programming for Welding Tasks Teach Wand: Camera based Programming Touch Screen Based Programming in a Joiners Workshop High Level Programming 15
Success Story: Robot Programming for Everyone Lead-Through Programming Server Today s robot programming techniques are to time consuming to enable robots to finish low quantities of castings A new robot programming concept was developed at ABB that enables robot programming by manual guiding enables speech communication offers an adjustable instruction set according to the tasks and personal needs enables integration of Plug n Produce * devices *Plug n Produce will be abbreviated by PnP in the following 16
Success Story: Robot Programming for Everyone - PnP A simplified robot programming system has to include external devices Integration of XIRP-protocol for PnP functionality into Lead-Through Programming System Automatic Device-discovery Server provides PnP services to operators automatically Direct integration of PnP into final robot program XIRP-protocol implemented on cheap microcontroller as server for simple external devices Grippers Cameras Sensors External devices in robot applications 17
Success Story: Robot Programming for Everyone - Safety Based on a risk analysis a special handling device was developed for safe Lead-Through Programming Six integrated acknowledge switches enable continuous guiding with regrasping 18
Success Story: Robot Programming for Everyone Graphical visualization of instruction sequence Lead-Through Programming System Server generates robot programs in native robot language automatically No manual writing of code needed 19
Success Story: Robot Programming for Everyone - Demo Candy feeder with PnP functionality Demo set up at the Automatica 2008 fair 20
Success Story: Intuitive Programming for Welding Tasks at the Treffler Workplace (1/4) Introduction Application: MAG arc welding of small lot size work pieces End User: Treffler Maschinenbau, a SME in the area of Augsburg, Germany. Contract manufacturing of welding products, approx. 70 workers. Devices and Methods: Programming by Demonstration Guiding the robot using a Force Torque Sensor Path recording and post processing Guiding tool 21
Success Story: Intuitive Programming for Welding Tasks at the Treffler Workplace (2/4) Design Criteria Low interfering contour Guiding in a natural way Functionality Safety: emergency stop, three stage button, collision switch Functionality: force torque sensor, record button, laser scanner Evolution Three versions made of aluminium profiles and SLS, iterated with comments from Treffler 22
Success Story: Intuitive Programming for Welding Tasks at the Treffler Workplace (3/4) Guiding Admittance control for manual guidance Moving the table with the same interface Programming Trajectory compression using intelligent algorithms or simple interfaces Assigning welding parameters using simple interfaces or learning algorithms Adaption of process parameters using a 3-D interface Increasing the path accuracy using local sensor data (douglas-peucker, regiongrowth algorithms) 23
Success Story: Intuitive Programming for Welding Tasks at the Treffler Workplace (4/4) Result 1: Reduction of programming time for less experienced people up to 50% Result 2: Only small reduction for experienced robot programmer Result 3: Programming the robot for process experts is possible after a short training (<2 h) 24
Success Story: KUKA Teach Wand PDA Teach Wand Camera Teach Wand Ball Joint Enabling Switch + PDA integration + cheaper tracking system + enabling switch + flexible mounting for pointing device + lightweight pointing device + form of a welding torch + new design for LEDs 25
Success Story: Smart worker interface for the teach wand Definition of Motions for Welding Seams, Approach Movements and movement Parameters Online Feedback of Robot Pose, online correction of orientation and translation 26
Success Story: Smart worker interface for the teach wand Some Screenshots of the PDA screen while programming a weld seam Programming Time for one trial run: Criteria KCP Teach Wand # of viapoints 28 28 # of welding points 42 46 # of welding seams 13 13 Seam length [mm] 2940 2940 Duration of the teaching [min] 100 26 Duration test and correction [min] 20 23 Duration complete [min] 120 49 # of points complete 70 74 27
Success Story: Smart worker interface for the teach wand 28
Success Story: Touchscreen based Programming in a Joiners Workshop (1/4) Final status of D4 demonstrator cell (30 Jan 2009) At Schreinerei Som: Full demonstrator At Fraunhofer IPA: Reduced version to demonstrate a subset of functionalities 29
Success Story: Touchscreen based Programming in a Joiners Workshop (2/4) Integration and evaluation of a graphics based user input device Flybook Results / future use: Flybook is used in the development phase of a new teach pendant generation Evaluation of a standard PDA as user input device MDA compact Results / future use: PDA not suitable as real input device for geometry adaption PDA can be used offline to generate sketches to be processed by the Sketch Editor 30
Success Story: Touchscreen based Programming in a Joiners Workshop (3/4) User interfaces for milling, drilling and spraying Exemplary screen shots Drilling Milling Spraying 31
Success Story: Touchscreen based Programming in a Joiners Workshop (4/4) User interface of Sketch Editor Exemplary screenshot Time Comparison Milling of a free form table: Manual: 30 min (including template preparation) Woodworking Ass.: 8 min 32
Success Story: High Level Programming (1/3) Automatic High Level Programming Workflow control High-level programming (HLP) commands SOA services Model-based Automatic robot program generation Collision avoidance 33
Success Story: High Level Programming (2/3) Interactive High Level Programming Set up of robot cell Interactive defintion of surfaces Generation of robot path Interactive adjustment of robot path 34
Success Story: High Level Programming (3/3) 35
Summary, Formal Aspects, Outlook Impact: Many devices and methods for humanlike instructing of robots have been developed. They were integrated in the four Demonstration scenarios and also many other testbeds Improvements towards the SMErobot goals: Reduction of programming time: up to 58% when programming manual, >90% for automatic programming Simplicity of use: simple, easy to understand interfaces Safe operation: permitted Scientific Impact: Publications, conferences, fairs But, most important: Set ups at the end user sites! 36
Thank you for your attention! 37