2. Visually- Guided Grasping (3D)

Size: px

Start display at page:

Download "2. Visually- Guided Grasping (3D)"

Domenic Griffith
5 years ago
Views:

1 Autonomous Robotic Manipulation (3/4) Pedro J Sanz sanzp@uji.es 2. Visually- Guided Grasping (3D) April 2010 Fundamentals of Robotics (UdG) 2 1

2 Other approaches for finding 3D grasps Analyzing complete 3D object model 3D reconstruction CAD model Object Recognition Extensive training sessions Previously known! Off-line Time Consuming Complex and Non-reactive algorithms April 2010 Fundamentals of Robotics (UdG) 3 Towards 3D Visually-Guided Grasping with a Dexterous Hand Grasping of previously unknown 3D objects using visual information (camera in hand) 2D features from contour multiple views as few views as possible Model objects with a bounding box 3D Partial Reconstruction April 2010 Fundamentals of Robotics (UdG) 4 2

Our Method follow centroid Start Top1 front position Top2 distance dz dx height Center of Mass / Center of Bounding Box width Z Y Z-Offset length Work area

3 Our Method follow centroid Start Top1 front position Top2 distance dz dx height Center of Mass / Center of Bounding Box width Z Y Z-Offset length Work area Robot Base X (known) Coordinate System April 2010 Fundamentals of Robotics (UdG) 5 Grasping Procedure video-06 April 2010 Fundamentals of Robotics (UdG) 6 3

Height refinement through front view April 2010 Fundamentals of Robotics (UdG) 7 Object Model Generation Object Contour Contour Analysis Partial model per View next position e.g. centroid, curvature,

4 Height refinement through front view April 2010 Fundamentals of Robotics (UdG) 7 Object Model Generation Object Contour Contour Analysis Partial model per View next position e.g. centroid, curvature, Imin, Imax, bounding box, orientation, Internal 2-D Models (Views) 3-D Analysis 3-D Object Model based on geometric features Views +center of mass, +center of bounding box +length, width, height Grasp Planning April 2010 Fundamentals of Robotics (UdG) 8 4

Image Processing Grey Image Undistortion Binarisation/ Segmentation Contour Extraction Object Model Generation April 2010 Fundamentals of Robotics

C x where I x, y Centroid y M 10 01 x c, yc, M 00 M 00 M 1 if (x,y) O 0 else Normalized central moment of order p+q 1 µ pq, M 00 p q x xc y yc Ix y

5 Image Processing Grey Image Undistortion Binarisation/ Segmentation Contour Extraction Object Model Generation April 2010 Fundamentals of Robotics (UdG) 9 Centroid and Axis of Inertia y x Binarized Image (x c,y c ) Imin Imax Image I with N pixels Object O Moment of order p+q p q M x y I x, y pq C x where I x, y Centroid y M x c, yc, M 00 M 00 M 1 if (x,y) O 0 else Normalized central moment of order p+q 1 µ pq, M 00 p q x xc y yc Ix y x Orientation of Imin towards y-axes y 0.5*atan2 2* Axis of Inertia sin Imin cos I max 11, 02 cos sin 20 April 2010 Fundamentals of Robotics (UdG) 10 5

Following the Centroid video-05 April 2010 Fundamentals of Robotics (UdG) 11 Following the Centroid y Image Coordinate System x dx dy Center of Image Velocity Control with Visual

6 Following the Centroid video-05 April 2010 Fundamentals of Robotics (UdG) 11 Following the Centroid y Image Coordinate System x dx dy Center of Image Velocity Control with Visual Servoing 2-D v img t v tool t dx * dy 0 tool R img * v img t y x Tool Coordinate System tool R img cos π 2 sin 2 0 sin 2 cos Roll 1 2 April 2010 Fundamentals of Robotics (UdG) 12 6

7 3-D Reconstruction Corresponding Points y y Pose1 Pose2 x x ray 2 ray 1 Rays in general skewed! P Camera calibration matrix: f px C 0 0 tan f py 0 X ray Y C 1 f py 1 c x c y 1 From image points to rays x * y 1 April 2010 Fundamentals of Robotics (UdG) 13 Current Experimental Setup Mitsubishi Robot Arm (PA-10) Three-Fingered Hand (BarrettHand) Camera in Hand (Sony XC-333) Kinematics 14 7

8 Grasp Planning 3D-Object Model Grasp Characterization Feasible grasp regions Grasp Synthesis Grasping type (precision / power), desired Tool pose and hand pre-shape (spread) Grasp Control Grasp regions (GR) ->compatible GR ->feasible GR Filtering for executable grasps (size) Choosing best grasp (manually) Generating pose and spread of hand Approaching Grasping Grasp Synthesis from Top 2 Finger Grasps 3 Finger Grasps April 2010 Fundamentals of Robotics (UdG) 16 8

9 Grasp Synthesis from Front April 2010 Fundamentals of Robotics (UdG) 17 Grasp Synthesis from Front thresh April 2010 Fundamentals of Robotics (UdG) 18 9

10 Examples of top grasps April 2010 Fundamentals of Robotics (UdG) 19 Examples of front grasps 10

11 Characteristics of Top Grasp Vertical Position Orientated and Positioned with respect to Grasping Regions Precision Grasp Spread if 3 finger grasp selected Finger tips stay at the same z-level April 2010 Fundamentals of Robotics (UdG) 21 Grasp Execution From Front Horizontal Position Positioned always at Height of Center of Mass Distance of Length/2 to Center of Bounding Box Power Grasp No spread Palm stays at object for support April 2010 Fundamentals of Robotics (UdG) 22 11

12 Examples video-07 April 2010 Fundamentals of Robotics (UdG) 23 Problems? video-08 April 2010 Fundamentals of Robotics (UdG) 24 12

Towards Complex Tasks video-09 April 2010 Fundamentals of Robotics (UdG) 25 Conclusions and Future Lines A new approach for grasping previously unknown 3D objects has been implemented and tested on a

13 Towards Complex Tasks video-09 April 2010 Fundamentals of Robotics (UdG) 25 Conclusions and Future Lines A new approach for grasping previously unknown 3D objects has been implemented and tested on a real robotic system Force/torque and tactile sensors are starting to use now for compensating the inaccuracies of both, grasp planning and visual perception April 2010 Fundamentals of Robotics (UdG) 26 SET 2008 IROS'

14 3. Sensor-based Control Interaction 3.1 planning of physical interaction tasks [ICRA-07-Prats] 3.2 vision-force-tactile integration for robotic physical interaction [ICRA-09-Prats] PhD Thesis (UJI; June 21st 2009) European Doctorate Mario Prats Robotic Physical Interaction through the Combination of Vision, Tactile and Force Feedback Preliminary ideas April 2010 Fundamentals of Robotics (UdG) 28 14

and the Humanoid Robot April 2010 Fundamentals of

and collaborating with humans in human environments

Poorly defined tasks Inaccuracy in the information about

15 Definition-1 What is a Service Robot? It is an intermediate state between the Industrial Robot and the Humanoid Robot April 2010 Fundamentals of Robotics (UdG) 29 Definition-2 Properties Robots working and collaborating with humans in human environments Unstructured and changing environment Unknown objects Poorly defined tasks Inaccuracy in the information about the environment Uncertainty is a main concern April 2010 Fundamentals of Robotics (UdG) 30 15

A Brief History about AI Robots Shakey (SRI 1967) vs Herbert (MIT 1990) Hierarchical Paradigm Reactive Paradigm Hierarchical Paradigm Environment Shakey

16 A Brief History about AI Robots Shakey (SRI 1967) vs Herbert (MIT 1990) Hierarchical Paradigm Reactive Paradigm Hierarchical Paradigm Environment Shakey SENSE ACT PLAN Task-Planner STRIPS [Nilsson, Rosen et al., 1969] Stanford Research Institute Problem Solver April 2010 Fundamentals of Robotics (UdG) 32 16

17 Reactive Paradigm Intelligence! Herbert Environment [Connell, 1990] SENSE ACT April 2010 Fundamentals of Robotics (UdG) 33 Behavior-Based Robotics Visually-Guided Grasping 17

Robot control system spectrum Adapted from [Arkin & Grupen, 93] DELIBERATIVE Purely Symbolic Representation-dependent Slower response High-level intelligence (cognitive) REACTIVE Reflex Response

18 Robot control system spectrum Adapted from [Arkin & Grupen, 93] DELIBERATIVE Purely Symbolic Representation-dependent Slower response High-level intelligence (cognitive) REACTIVE Reflex Response Representation-free Real-time response Low-level intelligence SPEED OF RESPONSE PREDICTIVES CAPABILITIES DEPENDENCE ON ACCURATE, COMPLETE WORLD MODELS Which are our expectations for this kind of robots? April 2010 Fundamentals of Robotics (UdG) 36 18

Human attendance in museums, hospitals, etc.

University of Bremen, Germany ISAC project

19 Human attendance in museums, hospitals, etc. Maggie Robotics Lab UCIII, Madrid April 2010 Fundamentals of Robotics (UdG) 37 Attending elderly and disabled people FRIEND project University of Bremen, Germany ISAC project University of Vanderbilt, US April 2010 Fundamentals of Robotics (UdG) 38 19

20 Are Autonomous Actions Necessary? University of Karlsruhe, Germany 3.1 Planning of physical interaction tasks April 2010 Fundamentals of Robotics (UdG) 40 20

Preliminary concepts the contact-level approach vs the knowledge-based approach Task-oriented grasps and grasporiented tasks April 2010 Fundamentals of Robotics (UdG) 41 The

21 Preliminary concepts the contact-level approach vs the knowledge-based approach Task-oriented grasps and grasporiented tasks April 2010 Fundamentals of Robotics (UdG) 41 The concept of physical interaction is introduced to refer indistinctly to the grasp (prehensile and non-prehensile) and to the task April 2010 Fundamentals of Robotics (UdG) 42 21

22 The Questions to Answer How can everyday physical interaction be specified in a common framework, including both the grasp and the task, and supporting sensor-based control? How can a robot autonomously plan a physical interaction task, making use of this framework? What sensors are necessary, and how can a robot combine those sensors and control its motors for performing physical interaction tasks in a robust manner? April 2010 Fundamentals of Robotics (UdG) 43 Grasp preshapes and shape primitives The six basic prehensile patterns defined by Schlesinger (1919), adapted from (Taylor & Schwarz, 1955) April 2010 Fundamentals of Robotics (UdG) 44 22

23 The physical interaction frames (left): the object frame (O), the end-effector frame (E), the task frame (T), the hand frame (H) and the grasp frame (G). The task motion must be transformed into robot coordinates through the kinematics chain formed by T, G, H and E (right). The grasp link is the relative pose between frames H and G, and represents the bridge between the task and the grasp April 2010 Fundamentals of Robotics (UdG) 45 Sensor-based tracking of the physical interaction frames The use of sensor feedback is specially important for the estimation of the grasp link April 2010 Fundamentals of Robotics (UdG) 46 23

Example: Use of tools The use of tools consists of two

involving a grasp for a transport task, and another one,

Reaching for the object Using the tool April 2010

preshapes The ideal task-oriented hand preshapes considered

24 Example: Use of tools The use of tools consists of two different instances of physical interaction tasks: one, involving a grasp for a transport task, and another one, involving the actual use of the tool Grasping the tool Reaching for the object Using the tool April 2010 Fundamentals of Robotics (UdG) 47 Task-oriented hand preshapes The ideal task-oriented hand preshapes considered by the physical interaction task planner April 2010 Fundamentals of Robotics (UdG) 48 24

25 Mapping to the Barrett Hand The Barrett Hand task-oriented hand preshapes Mapping to a Parallel Jaw gripper The parallel jaw gripper task-oriented hand preshapes April 2010 Fundamentals of Robotics (UdG) 50 25

26 Object representation (e.g. Structural model of a cabinet) April 2010 Fundamentals of Robotics (UdG) 51 Classification of object parts April 2010 Fundamentals of Robotics (UdG) 52 26

specification according to the task description and

27 Task description April 2010 Fundamentals of Robotics (UdG) 53 Planning Some examples of the grasp frame specification according to the task description and selected preshapes April 2010 Fundamentals of Robotics (UdG) 54 27

3.2 vision-forcetactile integration for robotic physical interaction April 2010 Fundamentals of Robotics (UdG) 55 AN EXPERIMENT The experimental

28 3.2 vision-forcetactile integration for robotic physical interaction April 2010 Fundamentals of Robotics (UdG) 55 AN EXPERIMENT The experimental environment consists of a mobile manipulator, an external camera and a cabinet with a sliding door that must be pushed open to the left ICRA-09 ICRA-09 28

29 4. The UJI Service Robot: A Case Study April 2010 Fundamentals of Robotics (UdG) 57 International Cooperation The UMass Torso (USA) Prof. Grupen Prof. Melchiorri Italy LASMEA Le Laboratoire Sciences Matériaux l'electronique 'Automatique France Prof. Martinet April 2010 Fundamentals of Robotics (UdG) 58 29

30 International Cooperation Prof Dillman Germany Prof Färber TECHNISCHE UNIVERSITÄT MÜNCHEN Germany Looking for Books in a Library University of Tsukuba Remote book browsing system using a mobile manipulator Tomizawa ICRA 03 ICRA 02 Johns Hopkins University Comprehensive Access to Printed Materials (CAPM) Choudhury 30

Scanning Mechanism Labels based on Library of Congress Classification (USA) 3

31 Looking for Books in a Library The UJI Librarian Robot ICRA 05 IROS 05 IROS 06 April 2010 Fundamentals of Robotics (UdG) 61 Main Steps 1 2 User Input Scanning Mechanism Labels based on Library of Congress Classification (USA) 3 Success? not 4 yes Grasping module Label April 2010 Fundamentals of Robotics (UdG) 62 31

32 System Sketch 2 3 OCR 1 4 April 2010 Fundamentals of Robotics (UdG) 63 Identification, Localization and Extraction 32

33 The System in Action 1 2 video [ICRA 05] April 2010 Fundamentals of Robotics (UdG) 65 Grasping Sketch Force Feedback April 2010 Fundamentals of Robotics (UdG) 66 33

34 Evolution of the Project ICRA 05 IROS 06 ICRA 07 April 2010 Fundamentals of Robotics (UdG) 67 From Jaume to Jaume-2 April 2010 Fundamentals of Robotics (UdG) 68 34

35 The Software Architecture April 2010 Fundamentals of Robotics (UdG) 69 The Task Frame Formalism (TFF) April 2010 Fundamentals of Robotics (UdG) 70 35

36 The Task Frame Formalism (TFF) Example 1: the robot turning the door handle April 2010 Fundamentals of Robotics (UdG) 71 Example 2: the UJI Librarian Robot April 2010 Fundamentals of Robotics (UdG) 72 36

37 Jaume-2 in Act Jaume-2 in Act 5/9/2011 April 2010 Fundamentals of Robotics (UdG) 73 April 2010 Fundamentals of Robotics (UdG) 74 37

38 Jaume-2 in Act 5/9/2011 April 2010 Fundamentals of Robotics (UdG) 75 Our new platform April 2010 Fundamentals of Robotics (UdG) 76 38

Multisensory Based Manipulation Architecture

Marine Robot and Dexterous Manipulatin for Enabling Multipurpose Intevention Missions WP7 Multisensory Based Manipulation Architecture GIRONA 2012 Y2 Review Meeting Pedro J Sanz IRS Lab http://www.irs.uji.es/