Cognitively Compatible and Collaboratively Balanced Human-Robot Teaming in Urban Military Domains

Cognitively Compatible and Collaboratively Balanced Human-Robot Teaming in Urban Military Domains Cynthia Breazeal (P.I., MIT) Deb Roy (MIT), Nick Roy (MIT), John How (MIT) Julie Adams (Vanderbilt), Rod Grupen (UMASS Amherst), Dieter Fox (Univ. Washington), Pam Hinds (Stanford)

Today s Outline Project Overview (30 mins) Objectives Application Domain for Testing Technical Challenges Team Organization Facilities Milestones/Demonstrations

Today s Outline Technical Briefs by MURI Team Local Peer-to-Peer HRI (40 mins) Remote HRI (30 mins) Evaluation (15 mins) Wrap-up (5 mins)

The Objective We want flexible, robust & scalable human-robot teams in dynamic and uncertain environments Current mission control approach for 1 human: N robot teams suffers from the fan-out problem (Olsen & Wood, 2004). Robot monitoring and selection, Context switching, Problem solving, Command expression Fundamentally limited by cognitive, attentive, and communication burden placed on the human operator.

Our Proposal Develop human-centric robot teammates that collaborate with humans peer-to-peer as well as remote-teams in a manner that is cognitively compatible with human cognitive, attentive, and communication capabilities and limitations.

Two Central Technical Themes Integration of cognitive models, action schemas and statistical estimation. We propose that a strong coupling is required in order to move human-robot collaboration and interaction into realworld scenarios where uncertainty is the norm. Integration of behavior models and distributed control. We propose that a strong coupling between distributed control algorithms and human behavior models is essential to allow for effective and robust tasking across mixed human-robot teams.

Considerations for Flexible, Robust, Scalable H-R Teaming Support multi-layer chain of human command

Considerations for Flexible, Robust, Scalable H-R Teaming Support multi-layer chain of human command Humans collaborate with a remote heterogeneous robot team Task-able autonomous robot teams

Considerations for Flexible, Robust, Scalable H-R Teaming Support multi-layer chain of human command Humans collaborate with a remote heterogeneous robot team Task-able autonomous robot teams Co-located humans and robots collaborating side-by-side as peers Humans and robots teammates train together before deployment More robots than humans

Potential Benefit of H-R Teams Improves with Training Training Human-Robot teams together shall improve robustness, flexibility, fluency, and efficiency of teamwork. Training together should also improves trust, learn preferences, sets appropriate expectations of responsibility, etc.

Scalability Demonstrate scalability to Increasingly complex tasks requiring significant cooperation between team members, Increasingly complex teams where a single human cooperates with a diverse team of robots on a shared mission. Human-robot teaming Robot-robot teaming Human-human teaming Choose a suitable domain

Civilian Response to Mass Casualty Event Chemical, Biological, Nuclear, Radiological, and Explosive Incidents (CBRNE)

CBRNE Cognitive Task Analysis Results in one or more models of the world that describe the world and how work is performed. Analyze knowledge, thought processes, and goal structures. MIT-Vanderbilt-Stanford UW-UMASS Amherst

CBRNE Cognitive Task Analysis Goal Directed Task Analysis (Endsley, Bolté, and Jones) used to determine which data people need to be aware of, how that data needs to be understood relative to operator goals, and what projections need to be made to reach those goals. Provides a basis for understanding how situation awareness maps to goals. Cognitive Work Analysis (Rasmussen and Vicente) is a constraint-based approach that provides an understanding of the socio-technical context in which workers perform tasks and yields insight into unanticipated scenarios. Provides a basis for understand how work should be completed. MIT-Vanderbilt-Stanford UW-UMASS Amherst

CBRNE Cognitive Task Analysis Information sources Local, state, and federal government documentation. Direct interaction with personnel Nashville Mayor s Office of Emergency Management Nashville Metro Fire and Police Departments FBI Field Office TEMA 45 th Civil Support Team Exercise observation MIT-Vanderbilt-Stanford UW-UMASS Amherst

Opportunities for Robots in Civilian Response to Mass Casualty CBRNE Event Once event has occurred, human responders cannot enter hot zone until suit up, adequately characterize the site (hazards, exits, etc.) and decontamination & medical triage areas set up. All humans are at risk in hot zone HAZMAT suits encumber vision, dexterity, limited oxygen, etc. Human victims in hot zone may wait a long time before help arrives

Our CBRNE-inspired HR- Teams Scenario

Opportunities for Robots: First Response Robots Enter Hot Zone & Coordinate with Humans in Cold Zone Cold Zone Incident Commander Hot Zone Remotely Supervised Robot Team Mission Assignment & Global Coordination Aerial Robot Team ID Hazards Find Ingress, Egress Dispersion Patterns Locate Victims Mapping EMS Triage Officer HAZMAT Chief Global Situational Awareness Heath Assessment Triage Direct Ambulatory Victims Ground Robot Team Waiting OK for entering Kickoff Hot Zone Meeting 2007 Human Victims

Opportunities for Robots: Second Response HAMZAT and EMS Humans Enter Hot Zone EMS Triage Officer Cold Zone HAZMAT Chief Mission Assignment & Global Coordination Global Situational Awareness Peer-to-Peer Human-Robot Team Distributed H-R Teams in Hot Zone Task Assignment & Coordination Remotely Supervised Robot Team Incident Commander HAZMAT H-R Team Situational Awareness Global Situational Awareness EMS Medic H-R Team Reduce physical & cognitive demands Improve task efficiency Human Victims

Many Technical Challenges EMS Triage Officer Cold Zone HAZMAT Chief Incident Commander Mission Assignment & Global Coordination Global Situational Awareness Global Situational Awareness Objectives & Task Requirements Aerial Robot Teammate Peer-to-Peer Human-Robot Team Human Teammate Ground Robot Teammate Distributed H-R Teams in Hot Zone Human-Robot Model Strategic Planning Shared Representation and Collaboration Human-Robot Team Estimation & Modeling Human Human-Robot Communication Cognitive & Model Collaboration Situational Human-Robot Joint Task Planning Awareness & Decision Making Autonomy Task Assignment & Coordination Situational Awareness Collaborative Decisions & Actions Multi-Modal Communication Learning & Training with Humans Remotely Supervised Robot Team Aerial Robot Teammate Team Estimating and Team Modeling Estimation & Modeling Task Planning Env. State Est., Represention & SA Team Task Allocation & Perf Human Victims

Research Program Organized as Three Technical Thrusts Thrust I: Peer-to-peer HRI for Communication, Collaboration and Training A. Cognitive Models for Common Ground and Joint Action B. Human-Centric Multi-Modal Communication C. Decision Making for Joint Action D. Training and Learning with Humans.

Research Program Organized as Three Technical Thrusts Thrust II: Remote Multi-Robot HRI for Situational Awareness and Coordination A. H-R Collaborative Environmental State Estimation and Mapping B. Dynamic Task Allocation for Distributed H-R Teams C. Human Compatible Visualization and Interfaces for Situational Awareness

Research Program Organized as Three Technical Thrusts Thrust III: HRI Studies for Human Factors and Performance Evaluation A. Simulation-Based Evaluations B. Real Robot Scenario Analysis and Evaluations

Team Organization & Management Tight Knit Task Forces THRUST I Peer-Peer HRI Thrust II Remote Multirobot HRI Thrust III HRI Studies & Evaluati o n A: Cognitive Models B: Multi-Modal Communication C. Breazeal (lead) D. Roy (lead) D. Roy C. Breazeal A: Environmental State B: Dynamic Task Est. and Map. D. Fox (lead) A: Simulation-Based Evaluations J. Adams (lead) J. How Allocation J. How (lead) C: Joint action D: Training & Learning (lead) R. Grupen (lead) C. Breazeal C. Breazeal R. Grupen D. Roy C: Visualization and Interface for Situational Awareness J. Adams (lead) J. How D. Fox B: Real Robot Scenarios for Analysis and Evaluations P. Hinds (lead) J. Adams C. Breazeal Greg Trafton, NRL -- human cognitive models to support collaboration, IA&C Rick Granger, Dartmouth -- Training and Learning, ID

Cognitive Models for Common Ground & Joint Action C. Breazeal MIT THRUST I Peer-Peer HRI Thrust II Remote Multirobot HRI Thrust III HRI Studies & Evaluati o n A: Cognitive Models B: Multi-Modal Communication C. Breazeal (lead) D. Roy (lead) D. Roy C. Breazeal (Greg Trafton, NRL) A: Environmental State B: Dynamic Task Est. and Map. Allocation D. Fox (lead) J. How (lead) A: Simulation-Based Evaluations J. Adams (lead) J. How C: Joint action D: Training & Learning (lead) R. Grupen (lead) C. Breazeal C. Breazeal R. Grupen D. Roy C: Visualization and Interface for Situational Awareness J. Adams (lead) J. How D. Fox B: Real Robot Scenarios for Analysis and Evaluations P. Hinds (lead) J. Adams C. Breazeal

Multi-Modal Communication Kai-Yuh Hsaio Deb Roy MIT THRUST I Peer-Peer HRI Thrust II Remote Multirobot HRI Thrust III HRI Studies & Evaluati o n A: Cognitive Models B: Multi-Modal Communication C. Breazeal (lead) D. Roy (lead) D. Roy C. Breazeal A: Environmental State B: Dynamic Task Est. and Map. D. Fox (lead) A: Simulation-Based Evaluations J. Adams (lead) J. How Allocation J. How (lead) C: Joint action D: Training & Learning (lead) R. Grupen (lead) C. Breazeal C. Breazeal R. Grupen D. Roy C: Visualization and Interface for Situational Awareness J. Adams (lead) J. How D. Fox B: Real Robot Scenarios for Analysis and Evaluations P. Hinds (lead) J. Adams C. Breazeal

Joint Action Under Uncertainty Nick Roy MIT THRUST I Peer-Peer HRI Thrust II Remote Multirobot HRI Thrust III HRI Studies & Evaluati o n A: Cognitive Models B: Multi-Modal Communication C. Breazeal (lead) D. Roy (lead) D. Roy C. Breazeal A: Environmental State B: Dynamic Task Est. and Map. D. Fox (lead) A: Simulation-Based Evaluations J. Adams (lead) J. How Allocation J. How (lead) C: Joint action D: Training & Learning (lead) R. Grupen (lead) C. Breazeal C. Breazeal R. Grupen D. Roy C: Visualization and Interface for Situational Awareness J. Adams (lead) J. How D. Fox B: Real Robot Scenarios for Analysis and Evaluations P. Hinds (lead) J. Adams C. Breazeal

Training & Learning Rod Grupen UM Amherst THRUST I Peer-Peer HRI Thrust II Remote Multirobot HRI Thrust III HRI Studies & Evaluati o n A: Cognitive Models B: Multi-Modal Communication C. Breazeal (lead) D. Roy (lead) D. Roy C. Breazeal A: Environmental State B: Dynamic Task Est. and Map. D. Fox (lead) A: Simulation-Based Evaluations J. Adams (lead) J. How Allocation J. How (lead) C: Joint action D: Training & Learning (lead) R. Grupen (lead) C. Breazeal C. Breazeal R. Grupen D. Roy C: Visualization and Interface for Situational Awareness J. Adams (lead) J. How D. Fox B: Real Robot Scenarios for Analysis and Evaluations P. Hinds (lead) J. Adams C. Breazeal

Environmental State Estimation & Mapping Dieter Fox Univ. Wash. THRUST I Peer-Peer HRI Thrust II Remote Multirobot HRI Thrust III HRI Studies & Evaluati o n A: Cognitive Models B: Multi-Modal Communication C. Breazeal (lead) D. Roy (lead) D. Roy C. Breazeal A: Environmental State B: Dynamic Task Est. and Map. D. Fox (lead) A: Simulation-Based Evaluations J. Adams (lead) J. How Allocation J. How (lead) C: Joint action D: Training & Learning (lead) R. Grupen (lead) C. Breazeal C. Breazeal R. Grupen D. Roy C: Visualization and Interface for Situational Awareness J. Adams (lead) J. How D. Fox B: Real Robot Scenarios for Analysis and Evaluations P. Hinds (lead) J. Adams C. Breazeal

Distributed Task Allocation John How MIT THRUST I Peer-Peer HRI Thrust II Remote Multirobot HRI Thrust III HRI Studies & Evaluati o n A: Cognitive Models B: Multi-Modal Communication C. Breazeal (lead) D. Roy (lead) D. Roy C. Breazeal A: Environmental State B: Dynamic Task Est. and Map. D. Fox (lead) A: Simulation-Based Evaluations J. Adams (lead) J. How Allocation J. How (lead) C: Joint action D: Training & Learning (lead) R. Grupen (lead) C. Breazeal C. Breazeal R. Grupen D. Roy C: Visualization and Interface for Situational Awareness J. Adams (lead) J. How D. Fox B: Real Robot Scenarios for Analysis and Evaluations P. Hinds (lead) J. Adams C. Breazeal

Visualization & Interface for Situational Awareness Julie Adams Vanderbilt THRUST I Peer-Peer HRI Thrust II Remote Multirobot HRI Thrust III HRI Studies & Evaluati o n A: Cognitive Models B: Multi-Modal Communication C. Breazeal (lead) D. Roy (lead) D. Roy C. Breazeal A: Environmental State B: Dynamic Task Est. and Map. D. Fox (lead) A: Simulation-Based Evaluations J. Adams (lead) J. How Allocation J. How (lead) C: Joint action D: Training & Learning (lead) R. Grupen (lead) C. Breazeal C. Breazeal R. Grupen D. Roy C: Visualization and Interface for Situational Awareness J. Adams (lead) J. How D. Fox B: Real Robot Scenarios for Analysis and Evaluations P. Hinds (lead) J. Adams C. Breazeal

Simulation-Based & Real-World Evaluation Julie Adams Vanderbilt Pam Hinds Stanford THRUST I Peer-Peer HRI Thrust II Remote Multirobot HRI Thrust III HRI Studies & Evaluati o n A: Cognitive Models B: Multi-Modal Communication C. Breazeal (lead) D. Roy (lead) D. Roy C. Breazeal A: Environmental State B: Dynamic Task Est. and Map. D. Fox (lead) A: Simulation-Based Evaluations J. Adams (lead) J. How Allocation J. How (lead) C: Joint action D: Training & Learning (lead) R. Grupen (lead) C. Breazeal C. Breazeal R. Grupen D. Roy C: Visualization and Interface for Situational Awareness J. Adams (lead) J. How D. Fox B: Real Robot Scenarios for Analysis and Evaluations P. Hinds (lead) J. Adams C. Breazeal

The Experimental Testbed 4 autonomous indoor ground robots 4 autonomous indoor aerial robots Indoor GPS 3 wearable sensor network systems

Autonomous Ground Robots MDS: Mobile, Dexterous, Social 4 MDS Robots. Development funded by ONR DURIP (MIT, UMASS Amherst, & Xitome, Inc.) Novel platform that combines mobile manipulation & Human-Centered Communication Small, safe indoor operation On & Off-board computing with 802.11g

UAV Testbed Modeled after RAVEN which simulates outdoors in a controlled setting allowing rapid prototyping of Planning/control algorithms Operator interfaces Avoids logistic and legal issues of operating outside Enabled by Vicon motion capture sensors Global, very accurate, real-time, minimal logistics Like having GPS indoors Routine operation of 1 operator to 4 UAVs Fully (and semi-) autonomous operations Low risk approach: vehicle is cheapest par Heterogeneous vehicles (UAVs, UGVs)

Wearable Sensor Network Wearable data capture system (developed by UW as part of DARPA ASSIST) Multi-sensor board: camera, microphone, 3-axis accelerometer, phototransistors, temperature, compass, etc. Intel imote2 processor Holux GPS with SIRF-III chipset ipaq handheld computer with 802.11 and bluetooth

Proposed Demonstrations

YEAR 1 Set up infrastructure, cognitive and system architectures Integration of existing abilities, with some extension for new abilities

YEAR 1 Theme First Response Small demonstrations of specific tasks UAV generate map of hot zone with regions of interest labeled MDS use map enter hot zone and characterize labeled areas in more detail. Update map. MDS perform simple medical triage tasks with human victims (both autonomous and support communication between victim and remote EMS)

YEAR 2 Theme HR Teams in Hot Zone Focus on Peer-to-Peer HRI Negotiation & dynamic task allocation among HR team: 1 human, 2 MDS, 2 UAV. Perform a collaborative task that requires some face-to-face and some distributed teaming (HAZMAT) Select candidate task procedure often performed by two people (e.g., hazardous sample collection). Have robot learn the task with a human teammate, and then improve fluency of teamwork through practice.

YEAR 3 Integrated EMS Demo HOT ZONE COLD ZONE Triage Officer EMS Unit 1 remote human 1 local human 4 MDS 4 UAV > 5 bystanders

YEAR 5 Integrated HAZMAT & EMS COLD ZONE Incident Commander Triage Officer HAZMAT Chief HOT ZONE EMS Unit HAZMAT Unit > 10 1 EMS 1 HAZMAT 4 MDS 4 UAV

Kai-Yuh Hsaio Rod Grupen UM Amherst John How MIT Julie Adams Vanderbilt Nick Roy MIT C. Breazeal MIT Technical Briefs Thrusts I-III

Thrust IA: Peer-to-Peer HRI Cognitive Models for Common Ground and Joint Action in Uncertain Environments Cynthia Breazeal (lead) Deb Roy External collaborator: Greg Trafton, NRL

Robust and flexible collaboration in dynamic and uncertain environments requires Mental Models Joint action: teammates share same goal and shared plan to achieve common purpose In dynamic, uncertain multi-agent worlds, teammates have [Cohen & Leveaque] diverging beliefs, changeable goals, fallible actions, etc. Communication is required to maintain a set of mutual beliefs and goals (Common Ground) to coordinate joint action [B. Grosz] Teammates must coordinate mental states to maintain common ground and coordinate joint action.

Thrust IA Objectives Develop cognitive architecture that endows robot teammates with mindreading skills (mental state inference) to support peer-to-peer collaboration Robot s mental models (of self and human partners) must be cognitively compatible with those of humans. Mental state inferences are useful for understanding and predicting human behavior Humans can apply their mental model to understand and predict the behavior of the robot Inferences of mental states must be robust in the face of uncertainly

Simulation Theory Neural Mechanisms of Mindreading TOM+SELF- TOM-SELF- TOM+SELF+ TOM-SELF+ Evidence of overlapping brain regions involved in SELF and TOM SELF: meta-representational cognitive capacity to apply a self perspective TOM: mindreading capacity to model someone else s state of mind Simulation Theory: certain parts of the brain have dual use - they are used to not only generate behavior and mental states but also to predict and infer the same in others Vogeley et al, Neuro Image 14, 170-181 (2001)

Robot s Cooperative Behavior despite human s False Beliefs and Invalid Plans Use own belief maintenance mechanisms to model and relate beliefs of others to its own beliefs

State of the Art Plan recognition in symbolic (language-based) domains Ambiguity resolution & plan recognition in dialog systems: [Grosz & Sidner 1990; Gorniak 2005] Perspective taking in adversarial decision making in computer gaming: [Laird 2001] Perspective taking in robots (verbal & non-verbal) for colocated teams Visual perspective taking for situated collaboration (disambiguate utterance) and action recognition: [Trafton et al. 2005; Johnson & Demiris 2005] Mental perspective taking for situated collaboration (handle false beliefs and invalid plans) for multiple human partners [Breazeal, Gray, Berlin, 2007]

Proposed Extensions Incorporate language as well as non-verbal behavior in estimating mental models (with Deb Roy) Scale to distributed teams (with Greg Trafton) Keep track of mental states for collaborative tasks with multiple people/partners in different spatial locations Integrate perspective taking with models of spatial cognition Scale up level of uncertainly (with Nick Roy) Robot is not witness to all events. Robot may have false beliefs, too. Needs to estimate mental states of self an others in the face of increasing levels of uncertainly.

Year 1 Demonstrations Integrate language-based interactions and observation of non-verbal behavior (Thrust IB) to inform mental model robot has for human Integrate statistical inference methods for belief construction (Thrust IC) Apply to initial triage of human victims Interview/dialog management Triage card decision

Year 2+ Demonstrations Integrate mental inference with decision making processes under uncertainty (Thrust IC) Robot anticipates human needs and responds Team is increasingly distributed spatially Integrate with trained tasks (Thrust ID) How does preferences, style, etc. factor into mental models? Integrate with remote humans (Thrust II) Integrated maps, situational awareness of remote coordinators how does this factor into mental models, and what to share with local human teammate & when? Apply to collaborative peer-to-peer task

Wrap Up

Where we are today Held Kickoff Meeting on November 7th Planning: Year 1 milestones, early integration efforts (will adjust ) Equipment: update & acquisition Common simulator tools (MDS, RAVEN) Collaboration mechanisms (meetings, etc.) Initial system architecture discussions Shared MURI lab space MIT-Vanderbilt-Stanford UW-UMASS Amherst

Initial System Integration Plan

Dedicated MURI Shared Lab In negotiation with Draper Labs for use of their high bay space. ~40 L x 20 W x 15 H Located 2-min walk to MIT