Multiplayer Computer Games: A Team Performance Assessment Research and Development Tool Elizabeth Biddle, Ph.D. Michael Keller The Boeing Company Training Systems and Services
Outline Objective Background Game-Based Testbed Development Performance Assessment Study Lessons Learned Future Plans 2
Objective Tasked to develop tools for evaluating collective training performance Needed a testbed environment to observe and evaluate collective training performance and develop assessment tools/technologies Rapidly develop a testbed for investigating team performance that is... Provides a realistic task context Easily reconfigured to support a new domain/task Cost effective Employ multiplayer COTS PC games to provide the simulated, operational environment 3
Collective Training Assessment Involves a combination of objective and subjective metrics Objective metrics: Compare observable behavior to a pre-determined standard Team establishes a timeline for managing the planning process (yes/no) Team member confirmed receipt of information (yes/no) Subjective metrics: Human observer uses predetermined criteria and experience to infer cognitive processes based on observable behaviors Team members avoided using excess chatter Team members demonstrated a willingness to change their plans in response to the evolving situation 4
Automated Collective Training Assessment Collection and translation of raw data through predetermined algorithms to provide meaningful performance assessment feedback that enhances traditional performance measurement methods Campbell et al., 2003: Recommended inclusion of APA capabilities in Future Combat Systems training devices Cardinal et al., 2004: Recommended data requirements for a performance assessment Base Object Model (BOM) to enable automated mission assessment as part of the Naval Aviation Simulation Master Plan (NASMP) Schreiber, Watz, Bennett, & Portrey, 2003: Developing a prototype APA technology Performance Effectiveness Tracking System (PETS) Throne & Holden, 2003: Developed and evaluated a prototypical automated performance assessment capability for Brigade level training 5
Collective Performance Assessment Testbed Requirements A simulated/virtual environment with... Multiplayer mode to implement operationally plausible collective or team military training scenarios Simulation data logging capability to: Specify data collection criteria, Record data to evaluate individual processes (e.g., button presses), and Record data to evaluate collective processes (e.g., maintaining formation, providing situational updates) Scenario authoring to create training scenarios Environment/behavior editing tools to develop training scenarios for various domain tasks 6
PC Games in Military Training commercial computer games provide the user with a sense of realism and an engaging story (scenario) Synthetic characters can serve as interactive, computerized allies or enemies Massive, multi-player (MMP) games provide an infrastructure to support distributed large- or smallscale collective training exercises Examples: U.S. Marine Corps Tactical Decision Making Simulation Family U.S. Army RDECOM Massively Multiplayer Simulation for Asymmetric Warfare DARPA DARWARS 7
PC Games in Research Game features attractive to R&D Game editing and scripting tools Develop synthetic world for context of interest Import/export data Modify/control game entities Visual and/or functional realism 3-D graphic display tools Human behavior representation Soar agent development (John Laird s Quakebots ) MOUT synthetic opposing forces development 3-D Viewers for military simulations 8
Team Training Testbed Validation Study Employ a commercial PC to develop a collective training testbed Unreal Tournament 2004 Extensive editing and scripting tools Collect data from the training environment Modify/create tailored training environment Multiplayer mode Multiplayer vehicles Internet voice communication 9
Approach Developed collective training scenario Specific performance measures to evaluate collective performance Identify data collection requirements Develop scenario with Unreal Tournament 2004 Conduct data collection experiment by having participants perform the scenario Analyze and evaluate data 10
Scenario Development Friendly forces are 2 helicopters (or comparable aircraft) 1 lead 1 wingman The mission objective is to destroy an enemy HQ camp, which is comprised of 5 towers. There will be a few land-based threats that the helicopters will encounter on their mission. Ground forces using rifles and surface-to-air missiles The heliopters will have planned waypoints that they are supposed to arrive at on the way to the initial point (the point from which they engage the enemy). After destroying the target, the helicopters will return to their Point of Departure. 11
Scenario 12
Performance Measures Time-on-target Accuracy of weapon fire Number of targets destroyed Other behaviors of interest: Ownship status (fuel consumption, damage) Communications 13
Data Collection Requirements Position of ownship (altitude and coordinates) for both helos Coordinates of: waypoints, refueling station, Point of Departure, Holding Area, all threats Speed of ownship Ability to time-stamp data (e.g., want to know what time a helo arrives at a waypoint) Player keyboard input (when they fire a weapon, steering input for the aircrafts, communications sent) Knowledge of weapon success/failure (e.g., when they shoot a missile, knowledge of whether the missile killed the threat or not or if it killed a friendly) Status of ownship (both helos) e.g., fuel consumption, damage 14
Testbed Implementation Frag Ops (Unreal Tournament 2004 mod ) used to implement scenario Set in the very near future (2006-2007) Simulates real-world soldiers and vehicle platforms (Comanche helicopter, jeep, MIG style aircraft) http://www.frag-ops.com/ Two players - each assumed the role of a Comanche pilot Mission Board and fly their own Comanche Destroy five lookout towers that were being defended by enemy ground forces 15
Testbed Two player-stations served as the helicopter cockpits One player station acted as the IOS station Three Dell Precision 670 desktop computer stations Altec Lansing headset with microphone 16
Performance Evaluation Time-to-complete-scenario 17
Track History 18
Conclusions Feasibility of using Unreal Tournament 2004 as a collective training testbed validated Refinements necessary to FragOps testbed in order to use for collective performance assessment R&D Re-directed to address fixed wing domain 19