Push Path Improvement with Policy based Reinforcement Learning

Transcription

1 1 Push Path Improvement with Policy based Reinforcement Learning Junhu He TAMS Department of Informatics University of Hamburg Cross-modal Interaction In Natural and Artificial Cognitive Systems (CINACS)

2 2 Outline Motivation Research Concept Previous Works System Architecture Policy Based Reinforcement learning Simulator Training Manipulation learning Learning Result

3 3 Motivation To complete real world tasks intelligently (in-hand manipulation/grasping) Shadow Hand ASIMO NEXTAGE

4 4 Motivation In-hand manipulation An ability to move and position objects within one hand Fingers 'push' an object to generate motions. Challenges A large number of joints (shadow: 19/24 DOFs) Complex interaction model (sensitive to errors) Limited perception capability (visual & tactile sensors)

5 5 Research Concept In-hand interaction system is a black box Fingers push in the black box in different directions Perceive from trial and error

6 6 Research Concept Push and Support Models Push finger Push finger Push finger Object P e ξ c Object ξ P e c Object ξ P e c Elastic Surface Elastic Surface Elastic Surface To roll the object on an elastic surface Trade off down and forward motions

7 Research Concept Push finger Model Evolution Different support fingers Number (Different views) Type: Fixed support finger Spring support finger Object ξ M P e c Os Push finger Spring support finger Object ξ e P Elastic surface c (a) Object Mf Of ξ P c Push finger Oe Ms Os Object Support fingers (b) Support fingers (c) Support fingers A C Push finger B View A View B Object ξ M P e c Os Push finger Fix support finger Object Kn ξ M P e c Os Push finger Spring support finger Object Mf Of ξ P c Fix support finger Push finger Oe Ms Kns Os Spring support finger 7 Object Mf Of ξ P c Fix support finger (d) (e) (f) (g) Push finger Oe Ms Kns Os

8 8 Manipulation Model Enhanced Manipulation Model Hybrid support model for yaw manipulation Opposite Velocity Enhanced manipulation model

9 9 System Architecture Robot hand: Shadow hand (Anthropomorphic, 19 DOFs, tenden dirven) Haptic sensing: BioTac (force, vibration and temperature, etc.) Visual tacking: AprilTags (2D barcode)

10 10 Experiment Experiment Setup

11 11 Experiments Initial Grasping Configuration

12 12 Experiments Rotational Manipulation (Yaw) Yaw O z x y Pitch Index finger pushes θ from -60 to 60 α from -60 to 60

13 13 Experiments Haptic feature Haptic reward Visual feature Object s Rotation: V T r Visual reward

14 14 Experiments O z x y Snapshots (rotational manipulation) Yaw Pitch

15 15 Experiments Enhanced Manipulation Rigid object

16 16 Policy Based Reinforcement Learning Reinforcement Learning Markov Decision Process (MDP) State: x Action: u Reward: r

17 17 Policy Based Reinforcement Learning Cost function (cost function): J The gradient of the cost function: Stationary distribution of the state Q function Baseline

18 18 Policy Based Reinforcement Learning Williams' Episodic REINFORCE algorithm Peters' Episodic Actor-Critic algorithm With compatible function:

19 19 Policy Based Reinforcement Learning

20 20 Policy Based Reinforcement Learning

21 21 Policy Based Reinforcement Learning Learning Frame

22 22 Simulator Training Density Push experiment: Push action: 380 push actions

23 23 Simulator Training RBFNs: Radial Basis Function Networks

24 24 Simulator Training Visual regression with RBFN

25 25 Simulator Training Visual: Approximate visual result with RBFNs Conduct as a simulator for the learning agents Haptic:

26 26 Manipulation learning with simulators Manipulation learning with simulators Learn to push object in a right direction Interact with visual and haptic simulators

27 27 Manipulation learning with simulators Reward Visual only Visual-haptic Final reward

28 28 Episodic REINFORCE Algorithm Learning Parameters Williams Episodic REINFORCE Algorithm Peter s Episodic Natural Actor-Critic

29 29 Learning Results Episodic REINFORCE Algorithm Visual-Only Visual-Haptic

30 30 Learning Results Episodic Natural Actor-Critic Visual-Only Visual-Haptic

31 31 Learning Results Episode Number before Learned NAC is little faster than REINFORCE Multimodal(Visual- Haptic) speeds up learning speed than unimodal (Visual- Only)

32 32 Thank You! Junhu He TAMS Department of Informatics University of Hamburg Cross-modal Interaction In Natural and Artificial Cognitive Systems(CINACS)