Jane Li Assistant Professor Mechanical Engineering Department, Robotic Engineering Program Worcester Polytechnic Institute
State one reason for investigating and building humanoid robot (4 pts) List two kinds of morphological communication (3 pts) Describe an example of in-hand manipulation (3 pts) 9/6/2018 2
Human surpasses current robots for overall performance Dexterity, energy-efficiency, versatility, reasoning, learning, selfadaption, compensation and recovery Build robotic embodiment to imitate human characteristics Appearance, voice, motion, intelligence Use humanoids as a tool for better understanding of human 9/6/2018 3
Expressive morphology and behavior Interpreting human expression Natural response in physical human-robot interaction 9/6/2018 4
9/6/2018 5
Platform Course projects 9/6/2018 7
Major components Baxter robot Mobile base Compliant grippers 3D cameras 9/6/2018 8
Simulation mode Physical mode GUI Various input devices Multi-perspective camera views 9/6/2018 9
Hardware 10 cameras Vero 2.2 Max frame rate = 330 Hz Resolution 2048 x 1088 = 2.2 MP Covered space = 17 ft X 21 ft 8 on railing + 2 on ground (for closer view) Software Nexus human motion capture and analysis Tracker moving object tracking 9/6/2018 10
9/6/2018 11
9/6/2018 12
9/6/2018 13
9/6/2018 14
Standard projects Shared-autonomous motion coordination of nursing robot Contact TA Team 1 Hand-arm coordination Team 2 Perception-action coordination Team 3 Loco-manipulation Human-robot collaboration Contact Heramb Nemlekar Team 4 Robot-mediated handover Team 5 High-level learning of human-robot motion coordination Small projects 1-student: Soft robot hand (me), Kinect teleoperation interface (Sihui) 2-student: Physical fatigue assessment (me) 9/6/2018 16
Functional robot platform Debugging hardware is a pain! May be stuck by technical details of a software Experienced members in the team Starting from sketch takes much longer time Commitment of every team members Your project grade may be ruined if the team fall apart Healthy team dynamics Your leader is knowledgeable, reasonable, and helpful Your partners are trustworthy 9/6/2018 17
A good project team means Successful project outcome (35%) High-quality group literature review (10%) Continue current project in RBE 550 and direct research for thesis 9/6/2018 18
Find the team right for you Project survey form allow you to fill in preferred teammates Talk to project contact to see whether you like and project and have the right skill set Talk to your classmates and find the ones you like to work with It may be a good idea to choose a small project You can have full control of the project progress You can work at your comfortable pace 9/6/2018 19
HUMAN MOTION COORDINATION Human motion study Analyze human motion coordination Develop novel models for human motion prediction Learn human motion strategies Model validation Compare model prediction with more human motion data SHARED AUTONOMOUS NURSING ROBOT User study Evaluate task performance Identify user challenges Propose shared-autonomous robot control design Refer to human motion strategies User study for performance evaluation Compare with/without autonomy
Autonomous reach-to-grasp, perception-action coordination, loco-manipulation Assessment of physical fatigue in robot teleoperation
Risk-sensitive task, unstructured environments Nursing, in-home caring, surgery, rescue Needs human in the loop Teleoperation interface for robot learning Can also be used for robot teaching human 9/6/2018 22
Much slower than human performance Perception issues, robot s physical capability, interface design Large training efforts Need one hour or more for the demonstrated task Hard to control motion coordination Subjects can only focus on controlling one robot components Some interfaces does not support simultaneous control of many DOFs 9/6/2018 25
Reduce novice users effort Mental/physical, training and using Improve task performance Efficiency, robustness, safety Enable complex robot behaviors 9/6/2018 26
Teleoperation Interface Evaluation (A-term) Propose sharedautonomous robot control (B-term) Team 1 Team 2 Team 3 Teleoperate the nursing robot yourself to find: (1) How teleoperation is difficult? (2) How teleoperation is different for different interfaces? (3) What part of task can be shared-autonomous? (4) Desirable level of automation? Reach-to-grasp Perception-action coordination Loco-manipulation Leaders Sihui Li Alexandra Kene
Sihui Li sli16@wpi.edu 2 nd year PhD Research focus Shared autonomous tele-nursing robots Teleoperation interfaces, Learning from demonstration 9/6/2018 28
Kenechukwu C. Mbanisi kcmbanisi@wpi.edu 3rd year PhD Research focus Human motion modeling and learning Human performance assessment Human-vehicle interaction 9/6/2018 29
Alexandra Valiton arvaliton@wpi.edu 2nd year PhD Research focus Shared autonomous tele-nursing robot Interactive perception 9/6/2018 30
Distorted perspective Lack of depth perception Keyhole Effect: narrow FOV 9/6/2018 31
We conduct human experiment to examine How does behavior evolve with expertise? How do head and clavicle camera behavior differ? Is there any way to be successful with an active wrist camera? How does the subject use the perception camera? Which camera(s) does the subject choose, when given a choice? 9/6/2018 32
9/6/2018 33
Understanding human perception-action coordination to inspire development of robotic teleoperation interfaces How can a robot help the operator perceive a remote environment? How can remote perception be improved? How are vision and action coupled in human motion? 9/6/201834
Collect data in human and robot teleoperation experiments Model and Compare the camera usage in human motion coordination and teleoperated motion coordination Propose autonomous perception control based on human preference Implement on TRINA platform demonstration Sept Oct Nov Dec 9/6/2018 35
Heramb Nemlekar hsnemlekar@wpi.edu 2rd year Master Research focus Human-robot handover High-level motion planning
RBE 550 Motion Planning Instructor: Jane Li, Mechanical Engineering Department & Robotic Engineering Program - WPI 9/6/2018 38
9/6/2018 39
9/6/2018 40
9/6/2018 41
Learning parameters from human-human handovers Giver and receiver are not face-to-face Giver is Sitting vs Standing Giver is Moving Receiver using one hand vs both hands Generalizing prediction and robot response across variants of handovers 9/6/2018 42
Human motion analysis Conduct motion study for handovers Pre-process and segment human demonstrations Algorithm development Formulate regression models for OTP prediction Improve object detection for grasping Modify probabilistic estimation in ProMPs 9/6/2018 43
Parastegari et al, Modeling human reaching phase in human-human object handover with application in robot-human handover, in IROS 2017. D. Vogt et al, One-shot learning of human-robot handovers with triadic interaction meshes, Autonomous Robots, vol. 42, pp. 1053 1065, 2018. M. Prada et al, Implementation and experimental validation of Dynamic Movement Primitives for object handover, in 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, sep 2014, pp. 2146 2153. G. Maeda et al, Phase estimation for fast action recognition and trajectory generation in human robot collaboration, The International Journal of Robotics Research, vol. 36, (13-14), pp. 1579-1594, 2017. 9/6/2018 44
Heramb Nemlekar hsnemlekar@wpi.edu 2rd year Master Research focus Human-robot handover High-level motion planning
Motivation High-level task plan can be built upon low-level motor skills Learning and planning can be more efficient at high-level 1. Leg1 Joint2 40 2. Leg3 Joint1 110 3. 4. Arm Joint5 25 5. 6. Hand Preshape 0.5 7.... 1. Move to door 2. Grasp handle 3. Pull door open 4. Move ahead 5....
For robot to plan actions on a high level - Learn high-level actions (options) and states (symbols) Training data from human demonstrations Understand reward for each high-level state (symbol) Generate an optimal task plan for a daily activity Objectives Learn symbols & rewards for a loco-manipulation task Perform table-top organization task involving handovers 9/6/2018 48
Human motion analysis Conduct motion study Pre-process and segment human demonstrations Algorithm development Implement low-level options as autonomous functions Improve SVM classification Generate symbols for loco-manipulation task 9/6/2018 49
References: G.D. Konidaris, L.P. Kaelbling, and T. Lozano-Perez. From Skills to Symbols: Learning Symbolic Representations for Abstract High-Level Planning. Journal of Artificial Intelligence Research 61, pages 215-289, January 2018. Literature Keywords: High-level planning, Symbolic MDP, PDDL, STRIPS, Hierarchical reinforcement learning 9/6/2018 50
Tasks Integrate hand with tele-nursing robot Test hands with various grasping Compare to ReFlex SF hand for performance Outcome Interface for hand motion control Demonstration videos Fits for ECE/ME background Learning ROS 9/6/2018 52
Problem with current Kinect teleoperation: Hand orientation detection is problematic with Kinect only. Needs other hardwares (e.g. IMU) to be integrated. Goal - Full body teleoperation with Kinect Implement proposed methods and integrate into the Trina system. Compare Kinect teleoperation with Vicon teleoperation. Fits for: 1 student, CS background, familiar with Kinect 9/6/2018 53
What is fatigue? A feeling of tiredness and being unable to perform tasks effectively. Change in muscle and brain electrical activity Affects heart rate and oxygen levels Alters movement patterns and coordination, including posture and perception Fatigue Performance 9/6/2018 54
Delsys Trigno EMG/IMU Sensors 9/6/2018 55
When comparing interfaces, which causes the most/least fatigue? How does level of fatigue compare with speed or ease of use? How does fatigue develop differently in the different interfaces? Which teleoperation tasks cause the most/least fatigue? How can these results be combined to suggest an ideal teleoperation interface? 9/6/2018 56
Assist Kene and Alexandra in gathering pilot data Based on preliminary data, compare teleoperation interfaces Examine the data for trends in fatigue development Propose a teleoperation interface design that limits fatigue and maximizes effectiveness. 9/6/2018 57
Fill in the project choice survey https://goo.gl/forms/6xdctwa8xjaamvnh2 9/6/2018 58
Goal Integrate PPU with off-the-shelf parts Reference F. Wang, G. Chen, and K. Hauser. Robot Button Pressing In Human Environments. IEEE Intl Conf. on Robotics and Automation (ICRA), 2018. 9/6/2018 59