HRI overview. Industrial Robotics: Human Robot Interaction. Safety in commercial systems. Human-Robot Interaction. Robot Safety

Size: px

Start display at page:

Download "HRI overview. Industrial Robotics: Human Robot Interaction. Safety in commercial systems. Human-Robot Interaction. Robot Safety"

Pierce Dickerson
5 years ago
Views:

HRI overview Industrial Robotics: Human Robot Interaction Henrik I Christensen CSE @ UCSD Human-Robot Interaction Safety

generation human-robot interaction Robot safety is crucial to deployment of systems anywhere Commercial systems have

1 HRI overview Industrial Robotics: Human Robot Interaction Henrik I Christensen UCSD Human-Robot Interaction Safety in commercial systems Robot Safety Robot Programming Application Packages Robot Simulation (for deployment studies) Next generation human-robot interaction Robot safety is crucial to deployment of systems anywhere Commercial systems have extensive mechanisms in place to assist in achieving good standards for robot safety Material from KUKA, ABB and A. GT

Safety consideration for deployment in industry Safety consideration for deployment in industry Requirements from Customer (1/7) Requirements from Customer (2/7) According to today s safety

hit the safety fence Cost effective safety fences are not designed to stop a robot Inside production the robotic application requires more floor space (theoretical work envelope red) than it

500 $ per m 2 ) If the customer wants to put safety fences directly next to the robot s actual work envelope he has to use: Either safety fences that are designed to stop a robot movement

use of mechanical axis range monitoring systems (mechanical proximity switches) 6 Safety consideration for deployment in industry Safety consideration for deployment in industry Requirements

are not wear- and maintenance-free The monitoring functionality is only available for axis 1 to axis 3 The monitoring functionality is not available for additional axis and linear axis

2 Safety consideration for deployment in industry Safety consideration for deployment in industry Requirements from Customer (1/7) Requirements from Customer (2/7) According to today s safety requirements it is not allowed to put safety fences inside the theoretical work envelope of the robot Due to a failure of the robot controller the robot might leave it s programmed path and hit the safety fence Cost effective safety fences are not designed to stop a robot Inside production the robotic application requires more floor space (theoretical work envelope red) than it really needs (actual work envelope white) The customer wants to reduce the required floor space to a minimum (approx $ per m 2 ) If the customer wants to put safety fences directly next to the robot s actual work envelope he has to use: Either safety fences that are designed to stop a robot movement those fences are very expensive as they use macrolon, reinforced frames and special fixtures Or a possibility to monitor the workspace of the robot in a failsafe way this was possible with the use of mechanical axis range monitoring systems (mechanical proximity switches) 6 Safety consideration for deployment in industry Safety consideration for deployment in industry Requirements from Customer (3/7) Requirements from Customer (4/7) If the customer uses the ABÜ technology he has to deal with many disadvantages: Mechanical wearing parts like proximity switches and cams are not wear- and maintenance-free The monitoring functionality is only available for axis 1 to axis 3 The monitoring functionality is not available for additional axis and linear axis Unacceptable long commissioning time due to installing and testing mechanical parts and cams Unacceptable long downtime when exchanging robots because mechanical parts have to be exchanged - downtime today approx hours 7 If the application requires a manual loading and/or unloading of parts the customer has to meet several safety requirements: Either the part has to be placed by the worker into an additional table with positioning pins - the table adds costs and needs additional floor space Or when putting the part directly into the gripper the drives have to be safely deactivated this adds cycle time due to a time delay when activating the drives again 8

Safety consideration for deployment in industry Safety consideration for deployment in industry Requirements from Customer (5/7) Requirements from Customer (6/7) To reduce the required floor space

robot Signal is communicated via failsafe bus system to robot Centralized safety concepts (one Safety PLC for up to 12 robots) are cost-intensive and rather complex: High cost for cabling as all

additional failsafe I/O modules Huge number of components are connected to the line Safety PLC - slows down communication and cycle time Response time includes Safety PLC cycle time and several

3 Safety consideration for deployment in industry Safety consideration for deployment in industry Requirements from Customer (5/7) Requirements from Customer (6/7) To reduce the required floor space for a robotic application it is very important to reduce the stopping distance of the robot in case of a failure: The monitoring of the robot by a Safety-PLC using ABÜtechnique adds reaction time to the stopping time (approx. 150 ms - yellow rectangle in top diagram) Failure robot hits proximity switch Signal is communicated via failsafe bus system to Safety-PLC 2 PLC cycles (worst case) until PLC sets signal to stop the robot Signal is communicated via failsafe bus system to robot Centralized safety concepts (one Safety PLC for up to 12 robots) are cost-intensive and rather complex: High cost for cabling as all safety components have to be connected to the Safety PLC (either hard-wired or via a safety bus system Most of the safety components can not be connected directly to a safety bus system adds cost for additional failsafe I/O modules Huge number of components are connected to the line Safety PLC - slows down communication and cycle time Response time includes Safety PLC cycle time and several communication delays: approx ms Robot stops 9 10 Safety consideration for deployment in industry Product Example - SafeOperation Requirements from Customer (7/7) KPS ESC-UL-10A Summary: Reduction of required floor space Use of low price safety fences inside the theoretical work envelope of the robot Cost-effective axis range monitoring of all axis in a wearand maintenance-free way Reduction of commissioning time for safety functions Reduction of downtime when exchanging robots Reduction of costs for manual loading systems (table, floor space and additional safety components) Reduction of costs for Safety PLCs, safety components and safety communication approx. 150 $/ft 2 approx. 30 $/ft up to 700 $/m approx. 200 $/Robot approx. 200 $/Robot up to $ up to 30% Second Main Contactor KPC 2GHz,512MB, MFC3-Tech CI3-Tech Cabinet Interface SafeRobot KCP Standard 10m Connection Cable Set Safe SafeRDW with Robot Installation Cable Set

Product Example SafeOperation Product Example SafeOperation SafeRDC Box Cabling 1.

SafeRobot X41 Connection for data cable X21.

4 Product Example SafeOperation Product Example SafeOperation SafeRDC Box Cabling 1. SafeRDC Box X02 Junction box on SafeRDC box X31 Connection for data cable X21 X31 X32 Connection for electronic measuring tool EMT X40 Connection for safe inputs and outputs of KUKA.SafeRobot X41 Connection for data cable X21.1 X41 X42 Connection for encoder cable X42 - XSRef 1 Robot controller 2 Robot 3 Reference switch 4 Encoder cable X42 XS Ref 5 Connecting cable X40 external safety logic 6 Data cable X21 X31 7 Data cable X21.1 X41 14 Product Example SafeOperation Product Example SafeOperation Interfaces As an interface to the cell / line the SafeRDC provides 8 dual channel failsafe inputs 4 dual channel failsafe outputs Enable Monitoring Inputs Workspace 2 Workspace 3 Workspace 4 Workspace 5 Workspace 6 Workspace 7 Operational Stop Reduced Velocity Workspace 8 Workspace 9 Workspace 10 Safety Staus SafeRDC X40 Outputs Messages Safety Parameters The safety parameters contain all the values and settings for the safe robot. The safety parameters are displayed as a tree structure in the configuration program. General information Monitored axis Reduced axis velocity and acceleration Cartesian velocity Axis range monitoring Monitoring of mastering Standstill monitoring Interfaces Machine data ($robcor.dat / $machine.dat) 15 16

Product Example SafeOperation Product Example SafeOperation Workspace Monitoring This function is used for the software setting and monitoring of the axis limits of each indi-vidual axis.

Workspace 1: Permanently monitored and always active. It can be modified, but not deactivated Workspaces 2 to 7: Activated using safe inputs Workspaces 8 to 10: Always activated and set safe outputs.

the risk for the worker Sets the axis velocity of an axis to a user-defined maximum value for the following operating modes Reduced Axis Acceleration Monitoring Is used to monitor the acceleration of

following operating modes Reduced axis acceleration can only be active if reduced velocity is active.

5 Product Example SafeOperation Product Example SafeOperation Workspace Monitoring This function is used for the software setting and monitoring of the axis limits of each indi-vidual axis. A range can be defined as a danger zone or a safety zone The individual axis ranges together make up the overall workspace, which may consist of up to 6 robot axes ranges and 2 external axes ranges Workspace 1: Permanently monitored and always active. It can be modified, but not deactivated Workspaces 2 to 7: Activated using safe inputs Workspaces 8 to 10: Always activated and set safe outputs. They do not trigger a stop of the robot Reduced Axis Velocity Monitoring Is used to monitor the velocity of a single axis when the risk analysis of the application requires a limitation to minimize the risk for the worker Sets the axis velocity of an axis to a user-defined maximum value for the following operating modes Reduced Axis Acceleration Monitoring Is used to monitor the acceleration of a single axis when the risk analysis of the application requires a limitation to minimize the risk for the worker Sets the axis acceleration of an axis to a user-defined maximum value for the following operating modes Reduced axis acceleration can only be active if reduced velocity is active Product Example SafeOperation Strategies for programming of robots Standstill monitoring (activated by a safe input) Drives remain in operation during the stop sequence and the brakes are not activated - the SafeRDC monitors the stop of the robot The robot is in a safe operational stop, but may nonetheless move within the axis angle tolerance despite the standstill monitoring The axis angle tolerance is specified separately for each individual axis Axis angle tolerance Todays robots are programmed using procedural programming! Trajectory following by jogging the system along a trajectory 19

Human Robot Interfaces Pen Based Interfaces Consideration of possible interfaces to make robots easier to use on the factory floor Simulation of process prior to deployment Intg of

Easy interfacing includes sound input and pen gestures Design of structured dialogs to drive process / example welding HRI using new types of interfaces New Interfaces Using structured

6 Human Robot Interfaces Pen Based Interfaces Consideration of possible interfaces to make robots easier to use on the factory floor Simulation of process prior to deployment Intg of process, CAD, and program to visualize Speech interface to talk to the robot Use of a smart pen to provide input (the ANOTO) Started to see smart pens with Bluetooth & USB interfaces Easy interfacing includes sound input and pen gestures Design of structured dialogs to drive process / example welding HRI using new types of interfaces New Interfaces Using structured dialogs combined with new interfaces such as the anoto pen How to structure the dialog to enable limited choice Identification of cut points in the foundry Organization of check box type dialogs

Knowledge based design of robot applications

for interaction with systems Using PDA, Speech,

between humans How can modern tools from knowledge

organize the different aspects of a process First

uses both speech and gesture to direct the robot

7 Knowledge based design of robot applications Multi-Modal Interfaces Providing natural interfaces for interaction with systems Using PDA, Speech, Gesturing, Body Pose Natural interaction as done between humans How can modern tools from knowledge engineering be utilized to structure processes and organize the different aspects of a process First person games Gestures for Task Learning Human teacher uses both speech and gesture to direct the robot Gesture glove allows the teacher to optionally control the robot for a short time 27

8 They are experts on Using Simulation in the Programming Process Simulation is a cheap way to evaluate systems before deployment of a full system Tremendous progress on simulation systems and virtual reality vs Primarily driven forward by gaming community and new hardware for game visualization (GPUs) Components based design for quick visualization of robot systems Simulation example Using simulation in design of systems

9 KUKA SIM Robot Simulation Simulation environments that matches operation on the real platform (including timing) Viewer KUKA SIM Layout Pro KUKA Sim Viewer KUKA Sim Layout KUKA Sim Pro 1 st Level: KUKA Sim - Viewer KUKA Sim CAD Import Layout KUKA Sim V1.1: Daten Import: Pro Viewer Daten Export:

10 2 nd Level: KUKA Sim - Layout 2 nd Level: KUKA Sim - Layout Layout Layout 2 nd Level: KUKA Sim - Layout 3 rd Level: KUKA Sim - Pro Layout Pro

3 rd Level: KUKA Sim - Pro Structuring the programming process There is a need

requirements Organize the acquisition of knowledge to be accessible by people

Can we structure the process adequately?

oriented architecture to enable easier deployment of applications Design of

11 3 rd Level: KUKA Sim - Pro Structuring the programming process There is a need to provide more structure to programming Pro Taking down the educational requirements Organize the acquisition of knowledge to be accessible by people on the shop floor What kind of knowledge is needed to make this a success? Can we structure the process adequately? Bringing it together Service architecture - Process Design of a service oriented architecture to enable easier deployment of applications Design of the structure of the process Design of templates for easy programming Enable easy integration with existing tools Example of welding

12 Structured CALIBRATION TASK FLOW Application packages Product Example CAMRob components Data transfer via Ethernet (CD-Rom) CAMRob 2.0 PC Module (installed on an external PC) - CAMRob Tech with Sim Tech - CAMRob Pro with Sim Pro CAMRob 2.0 KRC Module (installed on the KR C2 ed05) - CAMRob KRC - Robot - ApplicationModule Milling

Product Example - CAMRob CAMRob Process structure!

Automatic support of turntable or linear axis!

visualization of the milling process and milling

Singularity check (singularity avoidance will be

Support of tool change CAMRob Necessary Process

milling process planning (roughing / smoothing)

milling 49 Product Features - CAMRob CAMRob

13 Product Example - CAMRob CAMRob Process structure! Import of CNC files! Automatic support of turntable or linear axis! Embedded in KUKA-SIM for simulation and visualization of the milling process and milling cell! Collision detection! Singularity check (singularity avoidance will be available in the next version)! Support of tool change CAMRob Necessary Process structure Generation of CAD model (SolidWorks, Autodesk, CATIA..) Transfer of CAD models to a CAM system Tool and milling process planning (roughing / smoothing) KUKA part CAMRob: Conversion of CNC data Robotic milling 49 Product Features - CAMRob CAMRob training CAMRob - Operation Cell map Cell map 2 Componenten 1. Robot 2. Motor 3. Tool 4. Positioner 5. Clamps 6. Workpiece 7. Base Connections Father-Son Connection Logical Connection 4 52

14 CAMRob training CAMRob Base components 1. Dress package 2. Motor 3. Cone 4. Tool 5. TCP 1 2 The frame defined at (1) is a robot base The frame defined at (2) is the reference frame of the CNC file. All CNC positions are referred to this frame. The offsets (3) are the relative position of the CNC reference frame (2) as defined in the robot base (1). An offset in X,Y and Z can be defined. Tip: (1) is a frame which is always fixed and which has been measured once. (2) depends on the work piece at hand The base (1) must be attached to the workpiece or the world CAMRob Main applications for CAMRob Data flow 55 56

Spoken dialog interfaces Interface Architecture Tremendous progress on spoken dialog systems Natural language processing Definition of a dialog structure Analog range data occupied space geometric

15 Spoken dialog interfaces Interface Architecture Tremendous progress on spoken dialog systems Natural language processing Definition of a dialog structure Analog range data occupied space geometric mapping free space local route graph typed areas areas typed global global route r. graph grph Analog odometry data self-localization pose, state estimation people following dyn. object tracking motion & navigation obst. av., r. planning Open-ended vs closed vocabulary Semantic parsing Ontology-based BDI mediation Action Generation Lets do a small example Dial.ctxt meaning Dial. interpretation Ling. meaning CCG parsing Utterance speech recognition Analog speech input Communicative goal Dialogue planning (Multimodal) content Planning, CCG real. Utterance(s) text-to-speech Analog speech output We need an ontology! Definition of ontology Cross Modal contents association Meaning: propositional contents + intention + context Ontology for propositional truth and intention Multi-valued truth system Kruijff et al (2006) informative. attributive informative. factual polar ability command X endurant movement instance generic quality position motion thing region location size color causative subjective locationchange person in-situ motion transport perspective pose guidance destination spatial relative subjective direction X unknown true false known true false ambiguous

Spoken Language Processing Simple Example Dialogue context meaning structural dialogue analysis [ SDRT+paradigmatic choice ] Linguistic meaning natural language grammar analysis [ OpenCCG ] String

16 Spoken Language Processing Simple Example Dialogue context meaning structural dialogue analysis [ SDRT+paradigmatic choice ] Linguistic meaning natural language grammar analysis [ OpenCCG ] String (off-the-shelf) speech recognition [ Sphinx4, Nuance ] SF Networks CCG Grammar Speech Grammar (@d0:dvp(body ^ <AType>informative.attributive.endurant.perspective.spatial ^ <TurnT>continuous ^ <Mood>ind ^ <ContentBody>(b3:state ^ be ^ <Mood>ind ^ <Restr>(w3:person ^ we) ^ ^ <Mood>ind ^ <Restr>(w1:person ^ we) ^ <Scope>(i1:region ^ in ^! <Plane>horizontal ^! <Positioning>static ^! <Dir:Anchor>(o1:location ^ office ^! <Delimitation>unique ^! <Quantification>specific_singular))) We are in the office ComSys BDI SpLAM dyn. object tracking: found dyn. object command.movement. motion.locationchange. motion.guidance.person command ^ <Mood>imp ^ <Actor>(r1:hearer ^ robot) ^ <Patient>(i1:person ^ I)) people following: success start head-tracking response to command: success produce positive grounding feedback mapping:?known area get descr. current area informative.factual. person.perspective.spatial question ^ <Mood>int ^ <Restr>(w1:region ^ where ^..) <Scope>(w2:person ^ we)) mapping: unknown_f reply to question: mapping unknown_f negative reply-wh Audio signal Start guidance Come with me Acknowledge Okay. [Look] Inquiry Where are we? Reply Guidance Let me see. I am sorry. [follows] I don!t know this room Guidance [continues walking] Assembly of industrial objects such as a door Door Assembly - GT Platforms Utilized How can we simplify programming? Can we easily transfer to a running line? Can we combine speech, gesture,...? 64

17 Interactive Task Acquisition Interactive Task Acquisition Human teacher uses speech and gesture to direct the robot through a new task Robot generalizes task model/plan Human guides robot through task examples Infers task model from seen relations between parts Progress: focus on door assembly using actions and relational features in the taxonomy Definition of taxonomies or ontologies for control Learn & Execute Detect Align Pickup Insert Slide Retract Transport Feature Object Pose Contact Force/Torque Constraint Assembly Action Time Period Pose Motion Hold Time Period Environment Speed DOF Motion Type Fasten F Type Task Acquisition Execution Recog. Params Percept Control Alignment Dual Obj Rel. Pose Rec/Servo 3D Servo/ Pickup Obj/Action Obj Desc Rec/Servo Contact Servo Insert Obj/Action Pos / Const Vision/Force Approach Seq / Slide Action Start/End 1-2D Force Impedance Impedance Retract Action Space NA Law Path Plan F. Screw Action Torque Force Insert/Torque F. Snap Action Force/Event Force Force Disc

Gestures for Task Learning 69 70 Task Transfer to Execution Platform Learned model has general assembly task representation, independent of sensors/actuators Some features of the

18 Gestures for Task Learning Task Transfer to Execution Platform Learned model has general assembly task representation, independent of sensors/actuators Some features of the model need to be grounded in the new domain (e.g., object names/parts) Pointing recognition will allow for interaction about transfer between domains Example process for car assembly 72

Gestures for Task Transfer Goal Learning from Human Demonstration In another example, the human is in full control of the learning interaction.

Simon pays attention to the state changes throughout the demonstrations Task: learns object oriented tasks on the table-top Ex: books should be closed and

19 Gestures for Task Transfer Goal Learning from Human Demonstration In another example, the human is in full control of the learning interaction. Human demonstrates the task, showing the state changes to perform. Simon pays attention to the state changes throughout the demonstrations Task: learns object oriented tasks on the table-top Ex: books should be closed and put-away to the left 73 Task Learning Transparent Active Learning In a final example, we let the robot take more control of the learning process with Active Learning The human gives examples of the task, but the robot uses gestures to solicit particular examples from the human.

Transparent Active Learning Transparent Active Learning Task: learning tangram assembly tasks Ex: House has same size and red roof, snowman is small circle on top of big one After human

Transparent Active Learning Transparent Active Learning With active learning the robot is taking more control of the process, guiding the teacher s input since it knows what examples

20 Transparent Active Learning Transparent Active Learning Task: learning tangram assembly tasks Ex: House has same size and red roof, snowman is small circle on top of big one After human gives an example, Simon points to a piece it wants to replace, asking Can we also use this piece, is this still a house? Transparent Active Learning Transparent Active Learning With active learning the robot is taking more control of the process, guiding the teacher s input since it knows what examples would provide the most information However preliminary results indicate that people are not comfortable with the robot asking all the questions What we need is a smarter way to flexibly balance the control of the learning session from the human to the robot

Interactive Learning Key-Frame Based Learning by Demonstration Robots that can be easily re-tooled will be useful in dynamic areas of manufacturing, for example

people teach each other such skills Robots that learn in similar ways may be easier to use/ interact with 81 Embodied Interfaces Setup of the Woz Scenario for

21 Interactive Learning Key-Frame Based Learning by Demonstration Robots that can be easily re-tooled will be useful in dynamic areas of manufacturing, for example the general assembly process People working these lines often need to learn a new process or task in the assembly It is an intuitive and natural process by which people teach each other such skills Robots that learn in similar ways may be easier to use/ interact with 81 Embodied Interfaces Setup of the Woz Scenario for initial tests Navigation in a living scenario Specify the key objects / places in the environment Verify places as needed Gestures and spoken dialogue is available

22 Woz Setup Scenario Example Video CERO An Early Study Simple transportation Fetch cup of coffee mail delivery,... long-term tests

23 CERO CERO System Tasks Drive to position x (kitchen, office,...) Fetch coffee, deliver mail,... Interface Dialogue system PDA interface WWW interface User test over a period of 3 months Mobile platform Box for deliveries A simple user interface Design is crucial CERO Interface Example - Trilobite motion strategy Cleaning of rooms Autonomous coverage Efficient handling Inexpensive to implement

24 Trilobite motion strategy Second Example What are some of the long-term implications of interaction with robots? Can we trust short-term interaction What about adaptation to robots How does dynamic behavior influence the perception of a robot The italian driver experience? Example - Passage How to behave in presence of people Social Rules - Proxemics Human-Human Distance studied in psychology [Hall, 1966] Regions considered Intimate < 10 in Social rules of engagement Personal in Social in Embedding into a context Public space > 100 in There are cultural variations

Passage Behavior Control Design Robot Person Entering watch area Start detect y v x R x P v P x GOAL Social "avoid" behaviour Signal detection Passage/Park behaviour Passage completed x dy (x,y ) R 0

25 Passage Behavior Control Design Robot Person Entering watch area Start detect y v x R x P v P x GOAL Social "avoid" behaviour Signal detection Passage/Park behaviour Passage completed x dy (x,y ) R 0 R 0 G G (x,y ) resume (x,y ) R P R P dx Early Result Early Evaluation Y (m) 2 0 PERSON WALKING ALONG THE CORRIDOR Setup is too artificial to enable real evaluation Y (m) The variety of situations is large Useful to consider a more credible scenario Y (m) X (m) Long term evaluation? Y (m) X (m)

motion commands Simple GUI Architecture person position/velocity LASER PEOPLE TRACKING MODULE PERSON PASSING MODULE

26 Layout of Environment Embedding in a system User Interface mission A Offices Staircase B Elevator X Printer Recharging Station Localization robot pose Mission Planning Offices report target location People Tracking people Motion Control motion commands Simple GUI Architecture person position/velocity LASER PEOPLE TRACKING MODULE PERSON PASSING MODULE SONAR LOCAL MAP obstacle configuration CORRIDOR NAVIGATION MODULE COLLISION AVOIDANCE MODULE motion commands MOTION CONTROL MODULE

Embedding for evaluation Person Passage 12 10 elevator 8 6 robot person 1 person 2 home 4 2 office 0 5 10 15 20 25 30 Evaluation Results Lessons Significant personal variations Adaptation over time

27 Embedding for evaluation Person Passage elevator 8 6 robot person 1 person 2 home 4 2 office Evaluation Results Lessons Significant personal variations Adaptation over time must be considered Early evaluation using WoZ efficient easy to fool people The data material can be overwhelming Distance, Dialog, Sensor Data,... A careful design is required to make it manageable

28 HRI Summary Traditional interfaces based on CNC history Adoption of new types of interface modalities Gestures, Speech,... Simulation is also being adopted Extensive use of application packages to manage complexity Safety is essential to design of all systems HRI is one of the fastest growing areas in robotics Few HRI studies that are centered on real industry cases

KORE: Basic Course KUKA Official Robot Education

Training KUKAKA Robotics USA KORE: Basic Course KUKA Official Robot Education Target Group: School and College Students Issued: 19.09.2014 Version: KORE: Basic Course V1.1 Contents 1 Introduction to robotics...