Perception and task guidance Perceived world model & intent Prospective Teleautonomy For EOD Operations Prof. Seth Teller Electrical Engineering and Computer Science Department Computer Science and Artificial Intelligence Laboratory MIT
Goal: robots that join human teams Our lab develops machines that: Have a rich, live awareness of surroundings Understand natural speech and gesture Act autonomously, under human supervision Self-driving car (DARPA urban challenge) MIT vehicle one of only 6 to complete final course, of 89 original entrants Unmanned, voice-operated operated forklift Developed for DDR&E and U.S. Army Now part of fielding effort at ARDEC Self-driving di i wheelchair h & robot arm Learns from human-narrated motions, actions Prospective teleautonomous manipulator Effort funded by ONR starting 1/1/12 Subject of today s presentation
What problem are we tackling? ONR BAA 11-019 asked the research community to develop and demonstrate emerging technologies for dismounted missions to detect/locate, access, diagnose/identify, and render safe/neutralize explosive hazards, including IEDs and UXO. From EOD personnel at SOCPAC, SPAWAR: Degree of autonomy provided by existing platforms is very low Existing teleoperation interfaces are clumsy, even for experts Even simple missions can take tens of minutes via teleoperation Significant fraction of missions can t be completed via teleoperation; in these cases, personnel must dismount and approach manually We re pursuing a new approach that, t if successful: Will significantly reduce the need for manual approach Will significantly reduce average mission duration, error rate Could mitigate issues of cost, weight, comms req ts, s/w complexity
Traditional Task Decomposition H U M A N All levels of planning All scene interpretation Uses OCU to control robot DOFs Tracking of workspace dynamics Reacting to unexpected events R O B O T Sends video feed to operator Follows commands of OCU Limited Cartesian motions Motor velocities/forces/torques Raw sensor data Key limitation: operator must control high-dof arm with low-dof OCU
Novel Proposed Decomposition (SRI/Sarnoff Video-Trek?) Perception and task guidance Perceived world model & intent H U M A N New component on human side: Multi-touch human-robot interface R O B O T Changes human-robot division of labor: New components on robot side: Lidar (captures RGB point cloud) Prospective autonomy algorithms High-level mission, task planning Sends video, lidar feed to operator Scene interpretation, motion guidance Scene perception (human-assisted) Cues about objects, desired actions Low-level motion planning via multi-touch gestures on lidar data Scene-adaptive motion control Rather than make robot smart, make it able to follow detailed instructions
Video (but please keep in mind ) Proof-of-concept demonstration Research robot (WG PR2), touchscreen Actions shown: pulling a lever, opening a box flap Imagine, next, what we re working toward: Typical EOD robot (Talon, Packbot), rugged tablet Actual EOD tasks (dig, break window, open car door, detach circuit board, clip wire, use disruptor)
Prospective Teleautonomy
Novel system aspects (1/7) Mouse/touchscreen interaction No teleop knobs/joysticks/gloves/controllers/armatures* Operator no longer forced to do inverse kinematics, i.e., move joysticks to make arm & gripper move as desired Comment: elimination of complex OCU mechanism could reduce both system weight and cost *But system could retain standard OCU as backup manual control method
Novel system aspects (2/7) Operator s free view of workspace View rendered from bot-pov lidar point cloud Operator can change view without moving robot or arm Comment: running change detection locally, and sending new HD/scans to operator only when needed, could significantly reduce bandwidth needs
Novel system aspects (3/7) Operator selection of object of interest Can add/subtract lidar points while changing view System responds by highlighting its idea of object
Novel system aspects (4/7) Operator has free 3D view of highlighted object Comment: could integrate give me a better look at that or give this fancy sensor a better look at that
Novel system aspects (5/7) Operator indicates object s degrees of freedom then specifies object motion based on indicated DOFs Comment: in future, system could infer DOFs automatically by matching selected point cloud to object library
Novel system aspects (6/7) System animates its idea of what operator wants Waits for operator confirmation before executing plan
Novel system aspects (7/7) Robot executes the operator-specified motion Uses local sensing, closed-loop control to track object Comment 1: no longer need low-latency comms; could even tolerate intermittent link Comment 2: in future, operator could provide guidance continuously, during execution Comment 3: motion planner is platform-agnostic, reducing software complexity
Prospective Teleautonomy
Technical Challenges / Risks Which sensors should the system use? Size/weight/power/cost vs. utility outdoors Point clouds (esp. dynamic) don t compress well How should the human-robot interface work? Rich enough to convey operator mental model Intuitive enough for training, predictability, trust Robust enough to generalize across manipulands How to achieve robust motion planning? Must incorporate operator-indicated constraints (+/-) Must hide details of robot morphology from operator How to achieve robust motion control? Bot must track non-rigid changes in workspace, and adapt to moving manipuland geometry in real time
Milestones (5 parallel thrusts over 48 mos.) Bidirectional human-robot interface Touchscreen to hand gestures; don t touch constraints; articulation/manipulation gestures; complex task sequences Workspace perception Terrain classification; object segmentation; manipuland categories; fine-resolution perception; clutter; night operations Multi-modal guidance and control Simple route planning; pre-grasp stance planning; execution of don t touch plans; macro capability for task sequences End-to-end capability Simple mobility; 1-DOF actions; single platform; path adjustment; multi-dof actions; multiple platforms; autonomous approach and pre-grasp p stance; bimanual manipulation with stance adjustment Frequent engagement with EOD personnel Observation of EOD training; feedback about prototypes
Conclusions New approach to conduct of EOD missions Treats robot as junior partner to human operator Operator conveys interpretation, action plan to robot Robot reflects its understanding, then executes plan Requires advances along multiple fronts Human-robot interaction: assisted robot perception Sensing and perception: objects from point clouds Platform-independent motion planning & control Proposed several concrete milestones From actions to complex action sequences From single platform to multiple platforms
Questions / Comments? Reactions from EOD folks? Highest-priority capability gaps? Stats on mission durations, success rates? Access to training materials? Opportunities to observe training? Opportunities for EOD test/evaluation?