EVALUATION OF DIGITAL MAPS AND RADIO COMMUNICATION IN DISMOUNTED INFANTRY OPERATIONS

Transcription

1 DRDC Toronto CR EVALUATION OF DIGITAL MAPS AND RADIO COMMUNICATION IN DISMOUNTED INFANTRY OPERATIONS by: Barbara D. Adams, David W. Tack, and Jessica A. Sartori Humansystems Incorporated 111 Farquhar St., 2 nd Floor Guelph, ON N1H 3N4 Project Director: David W. Tack (519) PWGSC Contract No. W /001/TOR Call-up: HSI SIREQ Item #58 On behalf of DEPARTMENT OF NATIONAL DEFENCE as represented by Defence Research and Development Canada - Toronto 1133 Sheppard Avenue West Toronto, Ontario, Canada M3M 3B9 DRDC Toronto Scientific Authorities LCol Linda Bossi (416) Capt Jameel Adam (416) May 2005 This document contains information that may not be passed or shared, even in confidence, with foreign military, research and development representatives or civilian contractors of any nationality without the expressed prior permission of the Exploitation Manager of SIREQ TD. The scientific or technical validity of this Contract Report is entirely the responsibility of the contractor and the contents do not necessarily have the approval or endorsement of Defence R&D Canada Her Majesty the Queen as represented by the Minister of National Defence, 2005 Sa Majesté la Reine, représentée par le ministre de la Défense nationale, 2005

2 Abstract SIREQ-TD (Soldier Information Requirements Technology Demonstration) has determined that information transfer and communication between members in a dismounted infantry section is critical for mission success, and most soldier modernization programs in other countries have adopted an intra-section radio as part of their standard equipment. The purpose of the present study was to determine whether the use of a digital map coupled with a radio would improve the communication and performance of soldiers in a computer-simulated 1 st person shooter game. Specifically, we were interested in determining which aspects of communication and performance would or would not be enhanced with and without a digital map, and whether there would be differences in information exchanged with or without a radio. Thirty-two (32) soldiers were organized into four 8-person sections. Each section participated in 16 virtual-reality multi-player missions (for a total of 64 missions) while seated at computer workstations in a laboratory over the course of 8 days, to evaluate the effects of adding digital maps and radios to simulated dismounted infantry operations. Participants had the ability to activate a digital map by pressing a key to turn the map on and off. There were 4 different conditions, which included a paper map and no radio (Paper/No Radio), a paper map and a radio (Paper/Radio), a digital map and no radio (Digital/No Radio), and a digital map and a radio (Digital/Radio). In the no radio missions, as in previous work, soldiers were required to whisper and use hand signals to communicate. All communications were recorded and later content analyzed for type, function, friendlies vs. enemies, duration, pathways (role and network), and anticipation ratios. Data relevant to digital map usage, team performance (ammunition use, rounds taken) and team process (status awareness, teamwork, and workload) were also taken throughout the study. Results indicated that the addition of digital maps and radios neither improved nor diminished performance. However, with respect to team process, soldiers experienced the least amount of workload and the greatest amount of teamwork when they had radio communication. Exit questionnaires and focus groups provided very strong support for having both a digital map and radio for dismounted infantry operations, and suggest that having a digital map may improve assault group and section coordination. Having a radio was also seen as likely to improve information transfer as well as coordination within and between assault groups. Implications for future research are discussed. Humansystems Intrasection Comms Lab II Page i

3 Résumé Le projet SIREQ TD (démonstration technologique des besoins des soldats en matière d'information) a déterminé que le transfert d information et la communication entre les membres d une section d infanterie débarquée est critique au succès des missions, et la plupart des programmes de modernisation du soldat d autres pays ont adopté une radio intrasection faisant partie de l équipement standard. La présente étude avait pour but de déterminer si l utilisation d une carte numérique combinée à une radio améliorerait la communication et le rendement des soldats dans un jeu de tir virtuel à la 1 re personne. Ce qui nous intéressait particulièrement, c était de déterminer quels aspects de la communication et du rendement seraient ou non améliorés avec l utilisation ou l absence d une carte numérique et s il y aurait des différences dans l information échangée avec ou sans radio. Trente-deux (32) soldats étaient organisés dans quatre sections de huit personnes. Chaque section participait à seize missions virtuelles (pour un total de 64 missions) à joueurs multiples qui étaient assis à des postes de travail informatisés dans un laboratoire pendant une période répartie sur huit jours, afin de permettre d évaluer les effets de l ajout de cartes numériques et de radios à des opérations simulées d infanterie débarquée. Les participants avaient la possibilité d activer et de désactiver une carte numérique en appuyant sur une touche. Il y avait quatre conditions différentes, à savoir une carte papier sans radio (papier/sans radio), une carte papier avec radio (papier/radio), une carte numérique sans radio (numérique/sans radio) et une carte numérique avec radio (numérique/radio). Pendant les missions sans radio, comme dans les études antérieures, les soldats étaient obligés à chuchoter et à utiliser des signaux manuels pour communiquer. Toutes les communications étaient enregistrées et, par après, le contenu a été analysé à l égard du type, de la fonction, de l origine (amie/ennemie), de la durée, des trajets (rôle et réseau) et des taux d anticipation. Les données pertinentes pour l usage d une carte numérique, le rendement de l équipe (utilisation des munitions, obus encaissés) et le processus de l équipe (conscience du statut, travail en équipe et charge de travail) ont également été collectées tout au long de l étude. Les résultats indiquent que l ajout de cartes numériques et de radios n a ni amélioré ni réduit le rendement. Cependant, en ce qui concerne le processus de l équipe, les soldats connaissaient une charge de travail moindre et la plus grande quantité de travail en équipe lorsqu ils avaient recours à la radiocommunication. Des questionnaires de départ et des groupes de discussion soutenaient fortement la disponibilité de cartes numériques et de radios pour les opérations d infanterie débarquée et semblaient indiquer que la disponibilité d une carte numérique peut améliorer la coordination des groupes et sections d assaut. On pensait également que l utilisation d une radio améliorerait probablement le transfert d information ainsi que la coordination au sein des groupes d assaut et entre ces derniers. Les implications pour les recherches futures sont discutées. Humansystems Intrasection Comms Lab II Page ii

4 Executive Summary This study is the third in a series examining the impact of radio communication on small infantry teams for Soldier Information Requirements Technology Demonstration (SIREQ-TD). The SIREQ cognitive task analyses identified the transfer of information and communication between members in a dismounted infantry section as critical requirements for mission success. Previous research has indicated that soldiers overwhelmingly support the use of radios during missions (Adams, Tack & Sartori, 2004; Adams, Tack & Thomson, 2004). Recent technological developments have enabled the use of personal digital maps during missions. Investigations into the combination of these technologies, however, have not been conducted. Therefore, the main purpose of the current study was to determine the effects of a digital map (vs. a simple paper map) with and without radio to intra-section communication and performance. Some possible advantages of these apparatuses together might include increased individual and/or collective status awareness, reduced individual workload, and improved team coordination and mission execution. A further question to examine is whether or not these enhancements are sufficiently large enough to increase the likelihood of mission success. Of course, it may be that having only one or the other may be most advantageous. In the research investigation, the following aims were pursued. To investigate the types, functions, and pathways of communications within a dismounted infantry Section, with and without digital maps and radios. To determine differences in communication concerning location in missions with a digital map compared to a paper map. To explore how digital maps and radios compare to paper maps and no radios, in terms of team process and performance. To gather feedback from soldiers regarding the advantages and disadvantages of a digital vs. paper map, in combination with radios (and without radio) with respect to hardware and perceived impact on team process and performance. An eight-day laboratory experiment was undertaken at the Defence Research and Development Canada Toronto (DRDC Toronto) over six days in May 2002, and 2 days in June Thirty-two (n = 32) reserve force infantry soldiers participated in 64 virtual-reality multi-player missions, while seated at a computer workstation, to evaluate the effects of adding a digital map and radio communications to dismounted infantry operations. About half of all conditions involved the use of a digital map, and half of all missions involved the use of a radio. Paper Map/No Radio (the current state of the CF); Paper Map/Radio; Digital Map/No Radio; and Digital Map/Radio. For any one mission, eight soldiers worked together as an organic infantry section comprised of two Assault Groups, and engaged human enemy in the pursuit of a goal-based mission in a highly realistic, virtual urban environment. Human factors (HF) measures included status awareness, Humansystems Intrasection Comms Lab II Page iii

5 workload, communication measures, and teamwork factors. Data collection included questionnaires, focus groups, and performance measures. Results of the study indicated that the digital map was more frequently used when participants did not have a radio than when they did, but was highly used in both cases. In terms of communication, there appeared to be few differences attributable to the presence of the digital map. As expected, communication was much more frequent in the radio conditions. Frequency of communication type followed similar patterns to previous two studies in that orders showed a decrease in frequency, while transfers and acknowledgements showed an increase when soldiers had a radio. For the purposes of the current study then, communication type does not appear to vary with the introduction of the digital map. Moreover, frequency of function related communications also revealed a similar pattern to the previous studies in that location was the most frequent, followed by communication related to status. This varied little across conditions. Overall, the amount of communication across roles (section commanders (SCs), second-in-command (2IC), and assault group (AG) members was evenly distributed. Within conditions, however, the SC s increased their communication when they had a radio whereas the AG members decreased theirs. The 2IC s showed little change. Like the previous studies, the SC s and 2IC s most common type of communication was the order, followed by transfers of information. AG members most common type was the transfer of information. With respect to outcome measures, team performance appeared relatively unaffected by the introduction of both digital maps and radios. Moreover, there were no differences in status awareness evidenced with the addition of a digital map and radio. Participants perception of teamwork was significantly higher in missions that included a radio compared to those that did not, and perceived workload was significantly lower with a radio than without. In focus group discussions and in exit questionnaires, however, soldiers strongly supported the use of both digital maps and radios, rating digital maps as significantly improving location awareness, and assault group and section coordination over paper maps alone. Radio communications were seen to facilitate information transfer, and coordination both in and between assault groups. In addition, acceptability ratings for both digital maps and radios were significantly higher than the current inservice capabilities. Humansystems Intrasection Comms Lab II Page iv

6 Sommaire La présente étude est la troisième d une série expliquant l incidence de la radiocommunication sur les petites équipes d infanterie pour le projet de démonstration technologique des besoins des soldats en matière d'information (SIREQ TD). Les analyses des tâches cognitives du projet SIREQ ont identifié le transfert d information et la communication entre les membres d une section d infanterie débarquée comme des exigences critiques au succès d une mission. Les recherches antérieures (Adams, Tack & Sartori, 2004; Adams, Tack & Thomson, 2004) indiquaient que les soldats sont grandement en faveur de l utilisation de radios pendant les missions. Des développements technologiques récents ont permis l utilisation de cartes numériques personnelles pendant les missions. Toutefois, des enquêtes sur la combinaison de ces technologies n avaient pas encore été menées. Par conséquent, le principal objectif de la présente étude était de déterminer les effets de l utilisation d une carte numérique (au lieu d une simple carte papier) avec et sans radio sur les communications intrasection et le rendement. Certains avantages possibles de la combinaison de ces appareils pourraient comprendre l augmentation de la conscience individuelle et/ou collective du statut, la réduction de la charge de travail individuelle et l amélioration de la coordination de l équipe et de l exécution de la mission. Un autre facteur à étudier consiste à déterminer si ces améliorations sont de portée suffisante pour augmenter la probabilité du succès de la mission. Évidement, il se peut que l utilisation d une carte numérique ou celle d une radio soit la plus avantageuse. L enquête de recherche avait les objectifs suivants : Étudier les types, fonctions et trajets de communication au sein d une section d infanterie débarquée, avec ou sans cartes numériques et radios. Déterminer les différences dans les communications portant sur la position lors des missions avec une carte numérique comparativement à celles faisant appel à une carte papier. Déterminer comment l utilisation de cartes numériques et de radios se compare à l utilisation de cartes papier sans radios, à l égard du processus et du rendement de l équipe. Obtenir la rétroaction des soldats à l égard des avantages et des inconvénients d une carte numérique comparée à une carte papier, combinée à des radios (et sans radio) en termes de matériel et de l incidence perçue sur le processus et le rendement de l équipe. Une expérience en laboratoire de huit jours menée à Recherche et développement pour la défense Canada Toronto (RDDC Toronto) à été répartie sur six jours en mai 2002 et deux jours en juin Trente-deux (32) fantassins de la Force de réserve ont participé à 64 missions virtuelles à joueurs multiples, tout en étant assis à un poste de travail informatisé, afin de permettre d évaluer les effets de l ajout d une carte numérique et des radiocommunications aux opérations d infanterie débarquée. À peu près la moitié de toutes les conditions comprenait l utilisation d une carte numérique, et la moitié de toutes les missions comprenait l utilisation d une radio. Carte papier/sans radio (statut actuel des FC); Carte papier/radio; Humansystems Intrasection Comms Lab II Page v

7 Carte numérique/sans radio; Carte numérique/radio. Pendant chaque mission, huit soldats travaillaient ensemble en tant que section d infanterie intégrée constituée de deux groupes d assaut et ont engagé un ennemi humain dans l exécution d une mission orientée-but dans un environnement urbain virtuel extrêmement réaliste. Les critères de mesure des facteurs humains comprenaient la conscience du statut, la charge de travail, les mesures des communications et le travail en équipe. La collecte des données s est faite au moyen de questionnaires, de groupes de discussion et de mesures du rendement. Les résultats de l étude indiquent que la carte numérique était utilisée plus fréquemment lorsque les participants n avaient pas de radio qu autrement, mais qu elle était utilisée intensivement dans les deux cas. Au chapitre des communications, il semblait y avoir peu de différences attribuables à la présence de la carte numérique. Comme on s y attendait, les communications étaient beaucoup plus fréquentes dans les conditions avec radio. La fréquence des types de communications suivait des profils semblables à ceux des deux études antérieures : les ordres présentaient une fréquence réduite, tandis que les transferts et les accusés de réception présentaient une hausse lorsque les soldats avaient une radio. Donc, aux fins de la présente étude, le type de communication ne semble pas être fonction de l introduction de la carte numérique. En outre, la fréquence des communications en rapport avec les fonctions présentait aussi un profil semblable à celui des études antérieures : les communications en rapport avec la position étaient les plus fréquentes, suivies de celles ayant trait au statut. Cela variait peu en fonction des conditions. Globalement, la quantité de communications pour l ensemble des rôles (commandants de section (CmdtS), commandants adjoints (CmdtA) et membres de groupe d assaut (GA)) était répartie uniformément. À l intérieur des conditions, toutefois, les CmdtS augmentaient leurs communications lorsqu ils avaient une radio, alors que les membres de GA réduisaient les leurs. La quantité des communications des CmdtA variait peu. Comme dans le cas des études antérieures, le type de communication le plus courant des CmdtS et des CmdtA était l ordre, suivi des transferts d information. Le type de communication le plus courant des membres de GA était le transfert d information. Pour ce qui est des résultats des mesures, le rendement de l équipe variait relativement peu en fonction de l introduction des cartes numériques et des radios. De plus, les résultats n indiquaient pas de différence en termes de conscience du statut avec l ajout d une carte numérique et d une radio. La perception qu avaient les participants du travail en équipe était nettement plus grande pendant les missions comprenant une radio comparativement à celles qui n en avaient pas, et la charge de travail perçue était nettement inférieure avec l utilisation d une radio qu en son absence. Dans les discussions de groupe et les questionnaires de départ, toutefois, les soldats appuyaient fortement l utilisation de cartes numériques et de radios, estimant que les cartes numériques augmentaient considérablement la conscience de la position et amélioraient la coordination des groupes et de la section d assaut comparativement aux cartes papier. Les radiocommunications étaient perçues comme un moyen de faciliter le transfert de l information et la coordination tant au sein des groupes d assaut qu entre ces derniers. En outre, les cotes d acceptabilité des cartes numériques et des radios étaient nettement supérieures à celles de la capacité en service. Humansystems Intrasection Comms Lab II Page vi

8 Table of Contents ABSTRACT...I RÉSUMÉ... II EXECUTIVE SUMMARY...III SOMMAIRE... V TABLE OF CONTENTS... VII LIST OF TABLES...IX LIST OF FIGURES... X 1. BACKGROUND AIMS METHOD OVERVIEW EQUIPMENT AND FACILITIES Communications Network st Person Gaming Network Laboratory Layout PARTICIPANTS EXPERIMENTAL CONDITIONS: PAPER/DIGITAL MAP AND NO RADIO/RADIO MISSION DESCRIPTIONS Outline Approach Mission Maps Enemy Casualty Control DEPENDENT MEASURES Start of the Experiment Mission Freeze Throughout each Mission End of each Mission End of the Experiment Experiment Design Overview Experiment Schedule BRIEFINGS / TRAINING DAILY EXPERIMENTATION WRAP-UP SESSION RESULTS PARTICIPANT CHARACTERISTICS DIGITAL MAP USAGE OVERVIEW OF COMMUNICATIONS Humansystems Intrasection Comms Lab II Page vii

9 4.3.1 Type of Communication Function of Communication Combined Type and Function Friendlies vs. Enemies Pathways of Communication NETWORK RADIO COMMUNICATION ANTICIPATION RATIOS MEASURES OF OUTCOME Team Performance TEAM PROCESS EXIT QUESTIONNAIRE AND FOCUS GROUP Radio versus No Radio Paper Map versus Digital Map LIMITATIONS DISCUSSION REFERENCES ANNEX A: MISSION MAP...A-1 ANNEX B: QUESTIONNAIRES... B-1 ANNEX C: FREQUENCIES IN THE OTHER CATEGORY...C-1 Humansystems Intrasection Comms Lab II Page viii

10 List of Tables TABLE 1: ASSAULT GROUP MISSIONS TABLE 2: TRIAL SCHEDULE TABLE 4: MILITARY EXPERIENCE RATINGS TABLE 5: COMPUTER EXPERIENCE RATINGS TABLE 6: FREQUENCY OF DIGITAL MAP USAGE TABLE 7: WAV FILES AND TRANSMISSIONS TABLE 8: COMBINED TYPE BY FUNCTION FREQUENCIES TABLE 9: FREQUENCY OF COMMUNICATION BY ROLE TABLE 10: TRANSMISSIONS BY NETWORK TABLE 11: ANTICIPATION RATIOS BY CONDITION TABLE 12: ANTICIPATION RATIOS BY RADIO NETWORK TABLE 13: TEAM PERFORMANCE INDICATORS TABLE 14: ASSAULT GROUP DEATHS TABLE 15: STATUS AWARENESS RATINGS TABLE 16: TEAMWORK TABLE 17: WORKLOAD TABLE 18: INFORMATION TRANSFER TABLE 19: COORDINATION WITHIN AGS TABLE 20: COORDINATION BETWEEN AGS TABLE 21: OVERALL ACCEPTANCE OF THE RADIO CONDITIONS TABLE 22: LOCATION AWARENESS TABLE 23: COORDINATION WITHIN AGS TABLE 24: COORDINATION BETWEEN AGS TABLE 25: OVERALL ACCEPTANCE OF MAPS TABLE 26: SOLDIERS LEVEL OF AGREEMENT WITH RADIO COMMS AND DIGITAL MAPS Humansystems Intrasection Comms Lab II Page ix

11 List of Figures FIGURE 1: COMMUNICATIONS NETWORK... 5 FIGURE 2: ROGUE SPEAR SCREENSHOTS... 6 FIGURE 3: LABORATORY LAYOUT... 7 FIGURE 4: DIGITAL MAP USAGE BY ROLE AND CONDITION FIGURE 5: BREAKDOWN OF TOTAL COMMUNICATION BY CONDITION FIGURE 6: TYPE OF COMMUNICATION BY CONDITION FIGURE 7: FUNCTION OF COMMUNICATION BY CONDITION FIGURE 8: FRIENDLY/ENEMY REFERENCES BY CONDITION FIGURE 9: COMMUNICATION BY ROLE FIGURE 10: SCS - TYPE OF COMMUNICATION FIGURE 11: 2IC S TYPE OF COMMUNICATION FIGURE 12: AG MEMBERS TYPE OF COMMUNICATION FIGURE 13: BREAKDOWN OF RADIO NETWORK COMMUNICATION FIGURE 14: TEAMWORK MEANS FIGURE 15. WORKLOAD Humansystems Intrasection Comms Lab II Page x

12 1. Background The SIREQ cognitive task analyses identified the ability to transfer information and to communicate between members in a dismounted infantry section as critical requirements for mission success. A previous study conducted by the current authors (Adams, Tack, & Sartori, 2005a), in which soldiers participated in computer simulated dismounted infantry missions, found that radios were highly desired by participants. These attitudes expressed in focus group discussions were strong and unanimous despite the fact that the data indicated no significant improvement in team performance (i.e. rounds fired, rounds taken) or in team process (i.e. status awareness, teamwork, and workload). Their communications data did indicate the utility of radios; they communicated about 10 times more often with a radio than without, and that their patterns of communication were slightly more evenly distributed across roles (Section Commander (SC), second-in-command (2IC), assault group (AG) when they had radios. In current operations, dismounted infantry have neither radios nor digital maps. Currently, a Section Commander would have only a paper map and compass, and would likely be the only person with access to important information concerning location. Recent technological developments have made the provision of digital maps possible. It is, therefore, important to explore the implications of providing a digital map to dismounted infantry soldiers in order to answer questions, such as whether the information provided by a digital map helps soldiers to perform more effectively as a team? Moreover, previous work has shown that orders regarding location were the most frequent form of communication in simulated infantry assaults, both with a radio and without (Adams, Tack and Sartori, 2005a). If this is the case, providing each member of an assault group (AG) with the ability to access information about the location of other teammates may lessen the need for location-related communication and may ultimately improve accuracy and performance, while at the same time witness a drop in the number of overall orders. Of course, these questions need to be explored in more detail in order to establish the benefits of digital maps and radios for infantry operations. The present study seeks to determine the effect of providing a digital rather than a paper map and a radio vs. no radio in terms of the communication evidenced, as well as team performance and process in dismounted infantry units executing simulated tactical assault missions. Why digital maps and radios? While it seems intuitive that digital maps might improve communication concerning location and tactical planning and movement, SIREQ must test whether or not this is actually so. It is also important to consider the impact of providing both digital maps and radios. Possible advantages of the provision of this equipment might include increased individual and/or collective status awareness, reduced individual workload, and improved team coordination and execution. Then, if some or all of these enhancements are realized by the addition of this equipment, it will be important to determine whether they increase performance (e.g. fewer rounds taken and expended). It is expected that soldiers will transmit fewer and/or shorter messages pertaining to their location when they have maps due to the ad hoc shared information. Rigorous empirical assessment is necessary, however, because it is also possible that soldiers will report that the addition of another piece of equipment increases their workload. What configurations? It will be important to determine whether a particular combination of radio/digital map configurations provides the greatest improvements in performance. A number of configurations were used in this study: Humansystems Intrasection Comms Lab II Page 1

13 Paper Map/No Radio; Paper Map/Radio; Digital Map/No Radio; and Digital Map/Radio. The experiment will help to understand the impact of simultaneously providing digital maps and radios to infantry operations. Testing each of the possible configurations allows us to explore how digital maps and radios compare to paper maps and no radios, in terms of measurable outcomes, such as team performance and process. Moreover, this study will also investigate the types, functions, and pathways of radio communications within a dismounted infantry section, with and without digital maps and radios. It is also expected that there may be differences in communication concerning location in missions with/without a digital map. This study will also elicit feedback from soldiers regarding the advantages and disadvantages of a digital vs. paper map, in combination with radios (and without radios) regarding hardware and perceived impact on team process and performance. Humansystems Intrasection Comms Lab II Page 2

14 2. Aims In this investigation, the following aims were pursued: To investigate the types, functions, and pathways of communications within a dismounted infantry section, with and without digital maps and radios. To determine differences in communication concerning location in missions with a digital map compared to a paper map. To explore how digital maps and radios compare to paper maps and no radios, in terms of team performance and process. To gather feedback from soldiers regarding the advantages and disadvantages of a digital vs. paper map, in combination with radios (and without radio) with respect to perceived impact on team process and performance, and hardware. Humansystems Intrasection Comms Lab II Page 3

15 3. Method This section gives an overview of the experimental method and explains the approach, data collection method and equipment used. 3.1 Overview An eight-day laboratory experiment was undertaken at Defence Research and Development Canada Toronto (DRDC Toronto) across four weeks in May and June Thirty-two (n = 32) reserve force infantry soldiers participated in 64 virtual-reality multi-player missions, while seated at a computer workstation, to evaluate the effects of adding a radio communications network and/or a personal digital map to dismounted infantry operations. Half of the missions included the use of digital maps, while the other mimicked current operations in which only the SC has a paper map and a compass. The same was true of radios. As such, there were four conditions, including both digital map and radio (Digital/Radio), paper map and a radio (Paper/Radio), digital map but no radio (Digital/No Radio) and paper map and no radio (Paper/No Radio). For any one mission, eight soldiers worked together as an organic infantry section comprised of two AGs, and engaged human enemy in the pursuit of a goalbased mission in a highly realistic, virtual urban environment. Human factors (HF) measures included task performance, mission outcome, status awareness, workload, communication measures, and teamwork factors. Data collection included questionnaires, focus groups, and performance measures. 3.2 Equipment and Facilities The 1 st Person Gaming Laboratory included a series of thirteen PC workstations networked together by two distinct computing networks: the communications network and the gaming network Communications Network A computerized communications LAN was developed to track and log all radio voice communications activity in each experimentation session. The communications network is comprised of a central PC server connected to a micro-processor driven switchboard unit, which manages the voice communication traffic for four voice networks for ten intercom units (see Figure 1 below). Each gaming workstation includes one intercom unit with four network push-to-talk buttons. Humansystems Intrasection Comms Lab II Page 4

16 Switching control Gaming Station Gaming Station Data logging PC Switchboard and amplifiers four channel Gaming Station Gaming Station Gaming Station Ethernet audio Recorder Recorder Gaming Station Gaming Station Gaming Station Gaming Station Gaming Station Data recording PC Alternative with audio recording on hard disk Figure 1: Communications network Prior to the start of any experiment, soldiers were assigned membership to any or all of the four voice networks. To talk to other members of a network, a participant depressed the appropriate button and spoke into the boom microphone integrated into the gaming headphones. The switchboard detected and transferred the voice communication to one of four soundcards resident in the PC server. The server then logged the time, sender identification, and network (thereby identifying the list of listeners), and stored a digital record of the communication in a WAV file. The ambient noise from the gaming network (e.g. footsteps, rain, rifle fire, and explosions) was ported from the gaming stations to the gaming headphones so soldiers could simultaneously hear the gaming audio track and any radio voice communications st Person Gaming Network The 1 st Person Gaming network was developed to achieve a virtual mission environment where goalbased collective infantry tasks can be undertaken by a team of soldiers in real time. The gaming network is comprised of thirteen PC computer workstations connected by a local area network (LAN) to a PC server. This network configuration links the workstations together in a multi-player virtual mission environment. Each workstation can be assigned to any number of friendly or enemy teams within a given mission depending on the software gaming environment chosen for the experiment. Humansystems Intrasection Comms Lab II Page 5

17 For this experiment, Rogue Spear (Urban Operations package) was selected as the software gaming environment (see example screenshots in Figure 2). Figure 2: Rogue Spear screenshots Rogue Spear provides a highly realistic, and richly rendered, simulation environment with a wide variety of mission maps in a range of operational conditions (e.g. rain, snow, sun, indoors, night, etc.). Soldiers in the game view the simulation environment from a first-person perspective, through the eyes of their computer avatar, and can observe the actions of other soldiers in real time. The clothing, weapons, body armour protection, and characteristics of each participant in the game can be standardized or customized as necessary. Rogue Spear can also track and record certain individual and mission variables: number of shots fired, number of hits, number of casualties, etc. As well, the gaming network includes one workstation dedicated to capturing a real time record of participant movements and engagements on the mission map from a bird s eye perspective Laboratory Layout The laboratory layout (Figure 3) was configured to achieve a number of goals. 1. Separate the two AGs as much as possible; 2. Minimize the distance between members of a AG to facilitate the use of hand signals; 3. Situate the two computer networks in the middle of the workstations for ease of cable management; and 4. Position the controller and enemy workstations centrally, facing out towards the two AGs for monitoring purposes. Humansystems Intrasection Comms Lab II Page 6

18 Assault Group 1 Enemy Spare Controller Servers Assault Group 2 Figure 3: Laboratory layout To minimize distractions and facilitate immersion in the game, a whisper track of unintelligible voices was played in the lab during each mission, and the lights in the lab were turned off while each mission was active. 3.3 Participants Thirty-two (n = 32) infantry reservists were organized into four 8-person sections, comprised of two 4- person AG. 3.4 Experimental Conditions: Paper/Digital Map and No Radio/Radio This experiment varied presentation of a digital and paper map and a radio and no radio communications. Each of these conditions is described in more detail below. Paper Map Condition: The paper map condition paralleled current operations. In the paper map condition, the leader of each AG (SC and 2IC) received a map of the area of operations that they were able to use throughout the mission. This map was the same as that projected in the Humansystems Intrasection Comms Lab II Page 7

19 digital map, but was obviously not dynamic and did not provide location information about oneself or one s teammates. No other soldiers had access to a map, and leaders were instructed not to share the map with the rest of the section. Digital Map Condition: In this condition, leaders received the same paper map provided during the briefing phase (as in the paper map condition). In addition, in the digital map conditions, soldiers could also access a blueprint-like map of the area of operations. This map showed relevant structures in the area of operations, as well as showing the location of all the other members of the section. Enemies, however, were not displayed until they were encountered in the game. This digital map was activated by depressing the T button on their keyboards when they wanted to view the map. A 200 pixel x 150 pixel map of the area of their mission would appear in the lower right corner of their monitors. To remove the map, participants depressed the T button. This on/off method let the researchers determine exactly when the digital map was activated and how long it was used. a) No Radio Condition: In the no radio option, the radio voice communications network was partially disabled, by eliminating the communication audio to the headphones. Soldiers were instructed to use hand signals to indicate a need to communicate with other members of their individual AG, using the motions of their computer avatar. As a result of the problems that emerged during initial proof of concept testing using hand signals in the previous lab study (Adams, Tack & Sartori, 2005a) and because limiting communications to only hand signals would have been more restrictive than would be the case in real life infantry situations, we allowed soldiers to communicate verbally even in the no radio conditions. Thus, soldiers were asked to use a hand signal when possible to indicate a desire to communicate, but were also allowed to communicate the same kinds of messages that they might use in a real life situation when extremely close. Within the Rogue Spear game (Urban Operations mission pack), there were three main signals available, including On me, a request for other members of the AG to gather around the SC, Halt, and pointing in a direction. Just prior to employing a hand signal, soldiers were instructed to depress their AG intercom button and voice the hand signal instruction. This logged the time and type of hand signal sent (e.g. signal HALT ). Soldiers were instructed to abide by two pre-requisites for employing hand signals: Unobstructed line of sight to the receiving soldiers in the virtual environment; and Suitable visibility (i.e. sufficient light and no obscuration such as fog, rain, etc.). Each communication was logged by the initiator of the communication depressing the AG intercom button, and by voicing the communication given to the other soldier(s). Experiment staff monitored the AGs to ensure these conditions were met. b) Radio Communication Condition: In this condition, all soldiers had access to radios. When a soldier wanted to communicate, he depressed a button on the comms control and spoke into the boom mike attached to his uniform, positioned at about mouth level. Soldiers had to press the button for as long they spoke in order to be heard. No two soldiers in a section could speak at the same time; soldiers would hear a busy tone if another soldier was already speaking, and would have to wait until the network became free. Within this condition, there were 4 networks: Humansystems Intrasection Comms Lab II Page 8

20 Network 1: SC and 2IC only (2 speakers); Network 2: SC s and Assault Group A (4 speakers); Network 3: 2IC and Assault Group B (4 speakers); Network 4: SC and 2IC had speaking capability, Assault Groups A and B members had only listen capability (2 speakers, 6 listeners). 3.5 Mission Descriptions The missions used in this experiment provided a goal-oriented, team-based scenario in a contextually relevant small Unit, infantry context, with functionally realistic environmental and terrain conditions. A sample mission map is presented in Annex A. The experimental approach for each mission is outlined below, and aspects of the mission setup are described Outline Approach The experimental approach is described below for each of the briefing, execution, and endex phases of the mission. a) Briefing Phase: Prior to each mission, the section was briefed via videotape by an Army reserve platoon leader using an overhead map of the mission terrain. Each briefing included: A statement of the mission goal; An indication of the mission objective and the approach routes for each AG; Actions on contact, snipers, obstacles, etc; Timings necessary to coordinate the assault; and Outline of known enemy locations and strength. The SC and the 2IC were given the opportunity to study the paper map and form a plan, which they communicated to the AG as a whole before breaking up into separate AGs and discussing the specifics of their route and mission. When the leaders of both AGs indicated readiness to proceed, the mission scenario began. b) Execution Phase: In preparation for the start of each mission, the enemy (played by trained confederates) assumed their predefined positions in the mission map. Both AGs were positioned at their respective insertion points in the map and the mission clock was activated. Each AG was required to move tactically with stealth along their assigned approach route. Enroute the AGs encountered physical obstacles, which required them to modify their route and to coordinate their movement. In addition, AGs also had to dispatch single enemy soldiers they encountered along the way. Each mission required one AG to provide coordinated fire support to the other AG in the crossing of open ground, the engagement of enemy, or the breaching of the objective building. At a pre-selected point in the progress of each mission, an experimenter froze the mission to collect status awareness data. Having manoeuvred to the objective building or structure, the two AGs then began a coordinated assault on the objective. Each mission required the AGs to enter and clear the Humansystems Intrasection Comms Lab II Page 9

21 objective building from different approaches (i.e. without line of sight between Groups). In most cases, this required the section to clear several floors of the objective building by engaging and dispatching enemy soldiers as they were encountered, while being mindful of the likely location and status of the other AG. Throughout each mission, mission outcome and communications data (voice or hand signals) were logged and archived for subsequent analyses. c) Endex Phase: Following the completion of each mission, soldiers completed two questionnaires (i.e. teamwork questionnaire and workload questionnaire). At the end of this study, all soldiers were involved in a 2-hour focus group, in which they were given the opportunity to describe their experiences in this study Mission Maps The ten mission maps selected for this experiment included a mixture of open country (typically a limited area for insertion, approach, withdrawal, and extraction), and urban terrain with a combination of urban streets and in-building activities. Mission maps were selected for the following features: A network of approaches or streets, to be navigated to reach a range of buildings for an assault. It was necessary to be able to assign each AG a different approach route to the objective. Each route also had to be visually obstructed to ensure that the AGs could not observe each other during the critical part of the mission. Sufficient complexity of terrain in both the streets and the buildings to ensure that soldier movement required monitoring, control, and coordination. Adequate structures and obstacles to enable the effective use of enemy forces during both the approach and the assault. A number of possible insertion points in the map to ensure that a mission could be repeated using a different approach and assault route for each AG Enemy Enemy was used as a means of delaying and obstructing the AGs. This role was played by experimental staff. By placing enemy in key locations, enemy presence would require the two AGs to use communication to coordinate their movements and their fire. Enemy movement and weapons were limited, however, so that they could not kill the AGs, but they could be killed by them. There were 4 active enemies within each mission, each positioned at predefined points which provided cover within the mission map, or that challenged the AGs at critical points within the mission (e.g. at the rendezvous points). Three enemies appeared during the course of the mission, and one final enemy appeared at the mission objective. When this enemy was killed, the mission ended. Enemies were asked to engage the appropriate AGs, and to provide opposition for between 15 to 30 seconds Casualty Control For this experiment, enemy soldiers were able to shoot at and hit AG soldiers, but the game software was configured to ensure that members of the AG could not be killed by enemy forces. The number of hits sustained by each participant was analyzed as an indication of survivability. Humansystems Intrasection Comms Lab II Page 10

22 3.6 Dependent Measures Data were collected for the following dependent measures at various points over the course the day Start of the Experiment Participant data were collected prior to the start of the experiment to characterize the subject sample and to provide a basis for matching AGs based on the profiles of soldiers in each group (i.e. to make the groups as similar as possible). Participant Characteristics: Soldiers completed a questionnaire detailing their military and computer gaming experience, which were used to organize the membership of each AG: Military Experience: years in the infantry, rank, field experience in small Unit tactics, particularly urban operations; and Computer Gaming Experience: familiarity with computer use, experience with 1 st Person shooter games, experience with Rogue Spear Mission Freeze During the course of a mission the game was paused to query soldiers for the following information. Status Awareness: At a critical stage of the mission (e.g. immediately following an engagement with the enemy) the game server was paused and soldiers rated their knowledge about casualties and timings of the other assault group, and the timings of their own group Throughout each Mission Outcome, communications, and map use data were collected throughout the course of applicable missions. a) Digital Map Use: In the digital map conditions, a designated PC server logged each time each soldier pressed the T button on the keyboard in order to bring up the digital map. The frequency with which soldiers used the map was assessed according the number of T button presses. b) Communication Measures: The radio voice communication measurement approach has been adapted from Entin, Entin, MacMillan, and Serfaty (1993). The process of content analysis began with a member of the experimental team listening to a large sample of the radio communication and the no radio communication missions. This process yielded a number of categories needed to describe the communications evidenced. Once a tentative categorization framework was created, two members of the experimental team worked together to evaluate and revise the framework. This involved two individuals listening to more transmissions, working individually to categorize the type, function and referent of the transmission, and then meeting to discuss their decisions, and further refining their understanding of each category until they reached agreement on the categorization process. Another important component of this process was deciding on a unit of information. As many of the transmissions contain more than one discrete thought, it was important to use a common strategy for delineating units of communication. A unit of communication was defined as one discrete thought. Humansystems Intrasection Comms Lab II Page 11

23 Using the data capture capabilities of the radio voice communications LAN and a review of mission activities, each radio voice message was classified for the following variables, in chronological order. Type: Messages were first classified as either a Transfer of information, a Request for information, an Order to perform some task, an Acknowledgement of receipt of any of the latter message types, Communications Check, an Informal Order, and Getting Attention. Function: Each message was then classified by its information transfer function. Functions describe the purpose for transferring the information and include Status, Location, Task Assignment, and Planning/Problem Solving. Functions related to Game Keys and Repeat Message were also included. Enemy/Friendly: For functions involving Status or Location, the message was further classified as relating to Friendly forces, Enemy forces or Undefined Enemy Forces. For example, while in contact with an enemy sniper the AG leader may ask Fire Team Alpha do you have a fix on the sniper location from your position?. This message would be classified as a Request for information pertaining to Location of Enemy. The reply by Fire Team Alpha, Sniper observed in 2 nd story window of brown office building to my front, would be classified as a Transfer of information pertaining to Location of Enemy. The leader s response, Fire Team Alpha prepare to provide suppressive fire on my signal, would be classified as an Order for a Task Assignment, and so on. The following measures were then determined for each of the message classifications (i.e., type and function). Message Duration: To provide some insights into the size of the messages sent, the duration of each message was calculated. Anticipation Ratios: The ability of team members to anticipate the information needs of other members is an indication of teamwork and coordination. Anticipation ratios provide some insight into this anticipation behaviour by relating the number of information transfers to the number of requests (i.e., if transfers exceed requests then anticipation behaviour exists). c) Measures of Outcome: For each AG, mission outcome data were collected throughout the mission for the following variables. Rounds Taken: While friendly forces could not be killed, data on the number of hits sustained by each member and the AG as a whole were recorded (i.e. rounds taken). Ammunition Use (Rounds Fired): The number of rounds expended by the AGs, and the section as a whole, were recorded. AG Deaths: The number of deaths incurred by all members of the AG s were recorded End of each Mission Having completed a mission, soldiers completed questionnaires concerning the following data: Humansystems Intrasection Comms Lab II Page 12

24 a) Teamwork Measures: Soldiers completed a teamwork questionnaire to assess the performance of each AG, and the section as a whole, for a variety of teamwork dimensions: e.g. anticipation of information needs, coordination, communication behaviour, monitoring behaviour, feedback behaviour, back-up/covering behaviour, and team orientation. b) Workload: Soldiers also completed a NASA TLX questionnaire, indicating the levels of subjective workload experienced during the mission. NASA TLX requires the participant to rate six measures of workload which mostly refer to mental workload: mental demand, physical demand, temporal demand, task performance, effort, and frustration End of the Experiment After groups completed all missions in each communication condition (i.e. no radio/map vs. radio/map), a final focus group was held to discuss the soldiers opinions about and experiences with the following issues: The experimental approach and the use of the 1 st Person gaming environment; The benefits and drawbacks of the radio voice communications conditions; Lessons learned about the use of a radio voice communications network for information transfer, navigation, and team co-ordination; The opportunities for exchanging different types of information in different modalities (hand or radio); and The implications of an information exchange network on small team tactics Experiment Design Overview Table 1 outlines the missions for each of the four conditions in this experiment. The Rogue Spear mission maps selected for the experiment were randomly assigned a mission order. Each mission map had two separate insertion points and approach routes per mission. Each section was designated to complete a total of 16 different missions; each of the 4 mission types was completed 4 times. Due to technical difficulties, section 4 received 3 no map/no radio missions and 5 map/radio missions. Table 1: Assault Group missions Section Paper Map /No Paper Map/ Digital Map/ Digital Total Radio Radio No Radio Map/Radio In all, then, this study involved 31 missions with a paper map, 33 missions with a digital map and 31 missions with No Radio and 33 missions with Radio. As each participant performed in each of the 4 Humansystems Intrasection Comms Lab II Page 13

25 conditions, unless otherwise noted, all statistical analyses were performed using a repeated measures design with map (digital or paper) and radio (radio or no radio) as within-subject factors Experiment Schedule Table 3 outlines the eight-day experiment schedule. The trial was organized into three blocks: briefings and training, daily experimentation, and a wrap-up session. Table 2: Trial schedule Day 1 Day 2 Days 3 8 Briefings / Training Daily Experimentation Morning Experiment Briefing Rogue Spear Briefing Group Training 1 st Mission 2 nd Mission 3 rd Mission 4 th Mission 5 th Mission 6 th Mission 7 th Mission 8 th Mission Afternoon Individual Training Lunch Group Training 9 th Mission 10 th Mission 11 th Mission 12 th Mission 13 th Mission 14 th Mission 15 th Mission 16 th Mission Exit Questionnaires Focus Group Discussion 3.7 Briefings / Training All soldiers were briefed on the goals and outline approach to the experiment. Soldiers were required to complete the background questionnaire so that AGs could be formed with similar levels of military and computing experience. Soldiers with extensive experience with the Rogue Spear software were randomly assigned to AGs. The features and gaming environment of Rogue Spear were demonstrated to the soldiers. Training was comprised of individual and group training. Individual training included instruction and practice in movement, posture, and target engagement skills. The Rogue Spear gallery range and obstacle course were used to train fundamental skills, enabling the measurement of each soldier s 1 As participants did each mission 4 times, this within-subject variable was also used for analyses, but is not discussed, as it is not a variable of interest. Humansystems Intrasection Comms Lab II Page 14

26 training progress. Individual skills were then combined in a mission context by training soldiers in the single and multiple kill-house training scenarios in the Rogue Spear game. Each participant was required to pass through the kill-house training before advancing to the team training. Group training comprised experience with both team interaction and coordination in the Rogue Spear environment, and the use of communications. Group training (i.e. four person AG) involved practice missions against computer-generated enemy, using mission maps that were different from the experiment maps. AGs were required to practice at the computer game and the equipment controls, being encouraged to use the systems liberally to gain insight into the best means of employing the radio network and map. In addition, as part of their training, soldiers also learned about the use of hand signals within the game, how to activate them, the conditions under which hand signals could be used to initiate conversation, how to coordinate the use of hand signals with initiating face to face voice communication, and how to log their communication. 3.8 Daily Experimentation During each experimentation day, soldiers engaged in eight missions. Thus, each section participated in the experiment for two days. The breakdown of the timing of each phase of the missions was as follows: Mission Brief Section Planning Mission Execution Endex Measures 10 minutes 15 minutes 30 minutes 15 minutes The mission brief was provided by a reserve force officer who detailed mission goals, timings, AG routes, an outline plan of execution, and enemy status. Next, fifteen minutes were allotted for the section to review the mission map and for the SC to create his plan. When this process was complete, the mission was started. The mission continued until the predefined point at which one of the enemy forces was killed. At this point, the mission froze and measures of status awareness were completed. The mission then resumed until the objective was reached or thirty minutes had elapsed. Following the mission, soldiers completed the Endex questionnaires and all mission data were logged and archived. 3.9 Wrap-up Session The final day included a half-day set aside for completing missed missions or repeating any missions. The final focus group discussion was held at the completion of all testing. Humansystems Intrasection Comms Lab II Page 15

27 4. Results 4.1 Participant Characteristics Soldiers completed a background questionnaire detailing the following information, which was used to organize the membership of each four-person AG within each eight-person section. Military Experience: The thirty-two soldiers in this study were infantry reservists from the Toronto, Ontario area. Participants consisted of 15 Ptes., 14 Cpls., and 3 Sgts. The soldiers who participated in this trial had no less than one year of land force experience, but no more than 5 years experience, with an average of 2.69 years experience. Twelve of the 32 soldiers had operational experience, including one or more of the following theatres: Croatia, Bosnia, Rwanda, Cyprus, Somalia, Cambodia, Kosovo, and another unspecified location. Using a 5-point scale ranging from terrible (1) to excellent (5), soldiers rated their own abilities on several skills relevant to the military domain (e.g. reading maps, using hand signals, and engaging targets), as well as their military experience and/or training in planning missions and conducting section assaults. These results are depicted in the table below. Table 3: Military skill ratings Skills Mean (StDev) Planning skills 3.8 ± 0.7 Map reading skills 3.8 ± 0.8 Hand signal communication skills 3.9 ± 1.0 Tactical movement skill 3.9 ± 0.7 Target engagement skills 3.9 ± 0.8 As the table above shows, all soldiers rated their military skills slightly above average on the skill items. Soldiers also rated their actual training and/or experience planning missions, indicating no experience (1), some (2), moderate (3), or extensive experience (4). On average, soldiers indicated that they had some experience planning missions and in conducting section assaults, and moderate experience conducting urban operations, as depicted in the table below. Table 4: Military experience ratings Experience Mean (StDev) Training and/or experience planning missions 2.6 ± 1.3 Training and/or experience conducting section assaults 3.0 ± 1.5 Training and/or experience in urban operations 3.0 ± 1.3 Humansystems Intrasection Comms Lab II Page 16

28 Computer Gaming Experience: Using the same scale, soldiers general experience using computers and their experience with 1 st person shooter games, such as Rogue Spear, was explored. As would be expected, there was a range of experience in general computer proficiency. Table 5: Computer experience ratings Computer Experience Mean (StDev) Experience with computers 3.4 ± 1.1 Experience with 1st person shooter games 3.0 ± 1.1 On average, soldiers indicated a moderate level of experience with computers and with the Rogue Spear game. 4.2 Digital Map Usage It is important to explore how digital map usage varied within the radio vs. no radio missions. As Table 6 suggests, use of the digital map was somewhat higher in missions without a radio. Table 6: Frequency of digital map usage # of uses Digital/No Radio (n=16) 732 Digital/Radio (n = 17) 600 Total 1332 Figure 4 shows the frequency of digital map usage by role and condition. Humansystems Intrasection Comms Lab II Page 17

29 140 Digital/No Radio Digital/Radio Frequency of Map Usage by Role and Condition SC 2IC AG members Mean Figure 4: Digital map usage by role and condition 2IC s were the most frequent users of the digital map, particularly when without a radio (136 uses). Assault group members each used the digital map a total of 87 times without a radio and 69 times with a radio. Section commanders used the map 85 times when they had a radio and 71 times when they did not. As a whole, then, digital map usage was higher in missions that had no radio than those that did. This suggests that the digital map may have provided an opportunity to maintain contact with other team members especially when radio communications were no longer available. 4.3 Overview of Communications Each of the four sections completed 16 missions, for a total of 64 missions. Content analyses were conducted on all 64 missions. Across all conditions, there were a total of 9933 transmissions across all missions. The total number and mean number of transmissions for each of the four conditions appear in Table 8. Humansystems Intrasection Comms Lab II Page 18

30 Table 7: Wav files and transmissions N # of wav files # of transmissions Mean # of transmissions per mission Paper/No Radio ± 60.8 Paper/Radio ± 99.2 Digital/No Radio ± 61.2 Digital/Radio ± Total Clearly, communication was much more frequent when the AGs had radios than when they did not, at a ratio of about 2 transmissions in the radio sessions to every 1 transmission in the no radio sessions. Whether or not they had a map did not seem to make a difference in the frequency of their transmissions, although it is noteworthy that the Paper/Radio conditions averaged the highest number of transmissions across missions. Figure 5 shows the proportion of total number of transmissions in the two conditions. 35% 30% Proportion of Communication by Mission 25% 20% 15% 10% 5% 0% Paper/No Radio Paper/Radio Digital/No Radio Digital/Radio Figure 5: Breakdown of total communication by condition Humansystems Intrasection Comms Lab II Page 19

31 As the chart depicts, the amount of communication within a section appears to be contingent upon having a radio and decreases only slightly with a digital (vs. a paper) map Type of Communication 2 The important question, however, is whether the content of communications changes when section members have a digital map and/or radio in comparison to current operations in which they have neither. Figure 6 compares the types of communication in all 4 conditions as a proportion of the total communication within each. 45% Proportion of Communication Type by Condition 40% 35% 30% 25% 20% 15% 10% Paper/No Radio Paper/Radio Digital/No Radio Digital/Radio 5% 0% Transfer Request Order Acknowledgement Informal Order Getting Attentn Other Figure 6: Type of communication by condition Overall, the most frequent form of communication was orders, followed closely by transfers of information, then requests and acknowledgements which were relatively equal. When AGs did not have radio communications, orders were the most frequent form of communication, regardless of whether or not they had a digital map. Transfers were most frequent when they had radios and a paper map. There appeared to be little difference in transfers over the remaining three conditions. Requests were relatively constant over conditions, whereas acknowledgements seemed to vary somewhat depending on whether soldiers had radios, but not on whether they had a digital map. In general, these findings suggest that the type of communication in small military teams changes more as a result of the introduction of radios and not digital maps. For example, the frequency of orders and acknowledgements appear to be unaffected by the presence of maps, but are more variable depending on whether or not they have radios. Transfers appear to be affected by whether or not they have either piece 2 The other category includes items that were recorded wav files, but are not true communication (e.g. comms checks before missions started, or button presses produced because of participants clicking the comms button). A full breakdown of these items is presented in Annex A. Humansystems Intrasection Comms Lab II Page 20

32 of equipment, being particularly high in the presence of radios and paper map, but relatively equal across the other three conditions. Requests are relatively consistent whether they have digital map or radios Function of Communication It is important to note that not every type of transmission was associated with a specific function. This is the case, for example, with button presses, whose function was indeterminate within this experiment, and were therefore eliminated from this discussion (see Annex B, however, for the frequencies of other categories seen as indeterminate). Within the 4 conditions, 85% of transmissions were seen as having a functional aspect. Figure 7 shows the breakdown of the various functions of communication in the 4 relevant conditions. 60% Proportion of Communication Function by Condition 50% 40% 30% 20% 10% Paper/No Radio Paper/Radio Digital/No Radio Digital/Radio 0% Status Location Task Assign Plan/Probsolv Repeat Message Figure 7: Function of communication by condition As can be seen in the chart above, there was relatively little variation in function across the four conditions. Location was the most common function in all conditions, followed by status, task assignment, planning, and requests to repeat a message. Contrary to what was expected, there was no sharp decline in the number of times soldiers communicated about their location when they carried digital maps. In fact, in spite of having a common understanding of the geographic parameters of the mission when they had digital maps, soldiers communicated more frequently in the Digital/No Radio condition compared to the three other conditions. Status was the next most frequent function, being highest, and relatively equal when they had neither or both piece of equipment, and lowest and relatively equal when they had either the map or the radio. Humansystems Intrasection Comms Lab II Page 21

33 4.3.3 Combined Type and Function Most transmissions have information relevant to both type and function, for example, combining the type of communication order and the function of the communication location. It was of particular interest in this experiment to see if the introduction of a digital map would show a decrease in orders for location. Table 7 below lists the eight most frequent occurrences of forms of communication noted during the missions and their relative frequencies across the 4 conditions. Table 8: Combined type by function frequencies 3 Form of Communication Paper/No Paper/Radio Digital/No Digital/Radio Radio Radio Order regarding location 25% 17% 25% 20% Transfer of location 17% 23% 20% 21% Order regarding status 19% 13% 17% 15% Transfer of status 14% 16% 16% 15% Order regarding task assignment 4% 4% 6% 5% Requesting location 4% 4% 4% 4% Transfer task assignment 4% 5% 3% 4% Acknowledging request for location 2% 6% 3% 6% As the table shows, there was not a decrease in the orders for location as was expected with the introduction of the digital map. In fact, the decrease was more a result of whether or not the soldiers carried a radio. This is evidenced in the difference in percentage from the Paper/No Radio condition to the Paper/Radio condition, as well as the Digital/No Radio condition to the Digital/Radio condition Friendlies vs. Enemies Each item of communication was also categorized as to whether it described friendly forces or enemy forces across the four conditions, as shown in Figure 8. 3 This chart does not sum to 100%, due to small frequencies of other communication combinations. Humansystems Intrasection Comms Lab II Page 22

34 80% Proportion of Friendly/Enemy References by Condition 70% 60% 50% 40% 30% 20% 10% Paper/No Radio Paper/Radio Digital/No Radio Digital/Radio 0% Friend Enemy Undefined Figure 8: Friendly/enemy references by condition There was relatively little variation in references to friendlies versus enemies whether soldiers had radios/maps or not. Clearly, most references across all conditions were to friendlies, followed by references that were unclear as to whether they referred friend or foe, and last, references to enemies. Overall, soldiers were most concerned about information concerning their own team as opposed to the enemy, across all conditions. There were slightly more references to friendlies in the no radio conditions than in the radio conditions, and slightly more references to enemies in the radio conditions Pathways of Communication In order to understand the pathways of communication, communications were also analyzed in terms of the frequency with which speakers of different roles transmitted messages in both the radio and no radio conditions. This analysis is noted in Table 9: Table 9: Frequency of communication by role AG SC 2IC Members Total Paper/No Radio Paper/Radio Digital/No Radio Digital/Radio Total Humansystems Intrasection Comms Lab II Page 23

35 Overall, soldiers in all roles communicated with relatively equal frequency across conditions, with SC s having the most transmissions, followed by AG members as a group, and finally 2IC s. All roles communicated far more often in the missions in which they had radios. However, although SC s and 2IC s had similar numbers of transmissions when they had radios, AG members communicated far more often when they did not have a digital map (i.e., 1149) than when they did (i.e., 882). It is also possible to consider the relative proportion of communications within each role or position across conditions (Figure 9 below). Proportion of Communication by Role and Condition 40% 35% 30% 25% 20% 15% 10% 5% Paper/No Radio Paper/Radio Digital/No Radio Digital/Radio 0% SC 2IC AG Members Figure 9: Communication by role The graph above indicates that 2IC s communication across conditions did not vary greatly, However, they communicated most often in the Digital/No Radio condition, and least often in the Paper/No radio condition. SC s communication rates were higher with radios, regardless of map condition. The graph shows little difference in SC s transmissions across both no radio conditions, and these were lower than when they did have radios. SC s communicated most often in the Digital/Radio condition. AG members evidenced the most variation in their communication rates across conditions, and, in fact, their pattern of communication was different than their AG leaders. That is to say, AG members communicated most often when they had neither the digital map and radio, and least often when they had both. This suggests that AG members might use the digital maps and radios because they have the least awareness (in terms of both mission awareness and location information) of anyone within the section. Having equipment may help them to be more aware. Considering the types of communication that people in different roles used in varying conditions may shed some light on these findings. Figure 10 below shows this comparison. Humansystems Intrasection Comms Lab II Page 24

36 70% Proportion of Communication Type by Condition 60% 50% 40% 30% 20% Paper/No Radio Paper/Radio Digital/No Radio Digital/Radio 10% 0% Transfer Request Order Acknowledgement Informal Order Getting Attentn Other Figure 10: SCs - Type of communication As the graph above shows, for SCs, the frequency of orders comprised the most transmissions and exhibited the most marked change, and depended more on whether they had radios than maps. SCs clearly issued more orders in the two no radio conditions, and less with a radio. Conversely, they transferred more information when they had radios than when they did not, independent of the map. A similar pattern was evidenced for requests and acknowledgements, although there were far fewer transmissions concerning these types of information. There were very few informal orders and transmissions concerning getting attention, thus comparison was not possible. The pattern for 2IC s was similar to that of SC s, as shown in Figure 11. Humansystems Intrasection Comms Lab II Page 25

37 80% Proportion of Communication Type by Condition 70% 60% 50% 40% 30% 20% Paper/No Radio Paper/Radio Digital/No Radio Digital/Radio 10% 0% Transfer Request Order Acknowledgement Informal Order Getting Attentn Other Figure 11: 2IC s Type of communication While the most prominent form of communication for 2IC s was orders, this finding was particularly strong when they did not have a radio, but they had a digital map. Whether or not they had a digital map appeared to make little difference. They made more transfers in the Paper/Radio condition than any of the others. Requests were relatively few and equal, comprising less than 10% for any given condition. There were relatively few acknowledgements as well, but these were more common when soldiers had radios, regardless of the map condition. The 2IC s made more informal orders when they had both the radio and digital map than in any other condition. The same analysis was conducted for AG Members. Figure 12 shows this comparison. Humansystems Intrasection Comms Lab II Page 26

38 60% Proportion of Communication Type by Condition 50% 40% 30% 20% 10% Paper/No Radio Paper/Radio Digital/No Radio Digital/Radio 0% Transfer Request Order Acknowledgement Informal Order Getting Attentn Other Figure 12: AG members Type of communication AG members made more transfers of information than other types of communication, followed by acknowledgements and the other category, then orders and requests, but with a lesser frequency. Transfers were relatively consistent across conditions. Acknowledgements exhibited an increase in both the Paper/Radio and Digital/Radio conditions. 4.4 Network Radio Communication Four different radio networks were created, which allowed soldiers in different roles to choose the network on which they would communicate with others. Several questions emerged from this configuration. How frequently would each of the networks be used? What kinds of communication would dominate each of the networks? Table 10 shows the frequency of radio transmissions over each of the 4 networks. Table 10: Transmissions by network Network # Composition (# speaking members) # Transmissions 1 SC and 2IC (2) SC s Assault Group (4) IC s Assault Group (4) SC and 2IC with AG listening (2) 1630 Total 9933 Figure 13 shows the proportion of communications over each of the four networks. Humansystems Intrasection Comms Lab II Page 27

39 45% 40% Proportion of Communication by Network 35% 30% 25% 20% 15% 10% 5% 0% Network 1 - SC and 2IC Network 2 - SC's Assault Group Network 3-2IC's Assault Group Network 4 - SC and 2IC w/ AG listen Figure 13: Breakdown of radio network communication As shown in Figure 10, the individual AG networks were the most used, with the 2IC s used more frequently than the SC s. The network available to only the SC and 2IC was the least frequently used. It is noteworthy that this graph suggests leaders prefer to keep their communication within their own networks, using the two leader networks much less often. However, when they did choose the leader networks, they clearly preferred the condition in which the rest of the section could hear them talking to each other, as opposed to when they could not. This finding suggests that leaders felt little need for a private network on which members could neither receive nor send messages. In comparison to the previous laboratory study (Adams, Tack, & Sartori, 2005a), Network 1 saw a large decrease in overall transmissions. In Network 4, when AG members were restricted to listening only, it seems that AG leaders chose this network to communicate with each other instead of the private leader net. Perhaps the intent was to limit the overall number of communications among section members, thereby impacting overall efficiency. In the first laboratory study (Adams, Tack, & Sartori, 2005a), Network 4 was an all section members net, meaning that everyone could listen as well as speak. In this case, the AG members may have been reluctant to speak when all were listening or they might have been ordered by the leaders to communicate over AG member nets, while the SC and 2IC spoke between themselves privately. Both experiments point to the control that the leaders wanted to maintain over radio communication. 4.5 Anticipation Ratios The ability of team members to anticipate the information needs of other members is an indication of teamwork and coordination (Sperry, 1995). Anticipation ratios provide some insight into this anticipation behaviour by relating the number of information transfers to the number of requests (i.e. if Humansystems Intrasection Comms Lab II Page 28

40 transfers exceed requests then anticipation behaviour exists, Sperry, 1995). Anticipation ratios are calculated using the ratio of transfers to requests. Sperry (1995) represents this quotient with a numerator only; the denominator is understood to be 1. Thus, for example, an anticipation ratio of 1.63 indicates that a team s transfers exceeded their requests, 1.63 to 1. A ratio of 1.00 or more is indicative of anticipation behaviour, and, possibly, of shared mental models. Similarly, a ratio of less than one indicates that team members have requested more information than has been given. This may be a sign of less effective team communication. It is important to compare whether AGs show different patterns of anticipatory communication as a result of having radio communication. Anticipation ratios were calculated for each of the 4 conditions, and are presented in the table below. As can be seen from Table 11, the ratio of transfers to requests was highest when soldiers had radios, but not maps (5.7) and was lowest when they had neither piece of equipment (4.1). Table 11: Anticipation ratios by condition Frequency Paper/No Radio Paper/ Radio Digital/No Radio Digital/ Radio Transfers Requests Anticipation Ratio These results suggest that teams anticipated the needs of the section better when they had radios. It is possible that having both pieces of equipment precludes the need for this type of communication, due to the existence of information that teams might otherwise feel a need to communicate about. Moreover, while the latter may true, it was clear that having at least one piece of additional equipment, be it a map or a radio, facilitated anticipatory communication. Moreover, as Table 12 shows, there was diversity in anticipation ratios across the communication networks, with network 4 (SC and 2IC network in which members could listen to their transmissions), being the richest in anticipation, at a ratio of 5.4. Table 12: Anticipation ratios by radio network Frequency SC and 2IC SC Assault Group 2IC Assault Group SC/2IC with members Transfers Requests Anticipation Ratio The 2IC s AG exhibited the lowest anticipation ratio at 4.5. Nonetheless, these ratios indicate fairly similar levels of anticipatory communication within the various network configurations. Humansystems Intrasection Comms Lab II Page 29

41 4.6 Measures of Outcome Outcome measures included team performance and team process. The former includes measures such as rounds taken, rounds fired, and deaths. Team process includes measures of status awareness, teamwork, and workload Team Performance One of the possible advantages of having radio communications is improvement in team performance. One might expect missions in which section members had radio communication to be more effectively and/or efficiently performed. Within the context of tactical assault missions, analyses were conducted to explore section performance with/without radios and with/without maps. These indicators were rounds taken and rounds fired as well as the numbers of AG member deaths. 4 Data on the number of rounds taken sustained by AG members was recorded (see Table 13). Table 13: Team performance indicators Mean (StDev) Paper/No Radio Paper/ Radio Digital/No Radio Digital/ Radio Sig. Differences Rounds Taken 73.8 ± ± ± ± 35.9 ns Rounds Fired ± ± ± ± ns In terms of rounds taken, there were no significant differences among conditions. However, teams took the least rounds in the Paper/ Radio condition, and the most rounds in the Paper/No Radio condition. Rounds taken in the digital map conditions were in between these extremes. The number of rounds expended by the section as a whole was recorded (also see Table 13). Soldiers fired the least rounds in the Paper/Radio conditions, and fired the most rounds in the Paper/No Radio condition. Again, rounds fired in the Digital/ Radio condition, and the Digital/No Radio conditions were in between these values. However, these differences were not statistically significant. These results also show no significant impact (either positive or negative) with the addition of digital maps and radios. AG member death sometimes resulted from unintentional fratricide (e.g. being shot by a member of one s assault group). An analysis was also conducted on the number of AG member deaths within each of the 4 conditions. Frequencies are shown in Table 14 (coded as dichotomous data for this analysis). 4 Due to a recording malfunction, performance data for two missions was not collected. Humansystems Intrasection Comms Lab II Page 30

42 Table 14: Assault group deaths Frequency Paper/No Radio Paper/ Radio Digital/No Radio Digital/ Radio Sig. Differences Deaths Cochran Q (15 df) = 37.3, p =.001). This test showed a significant overall difference in deaths among the 4 conditions (Cochran Q (15 df) = 37.3, p =.001), with deaths highest in the Digital/No Radio condition, and lowest in the Paper/Radio condition. As a whole, however, these analyses indicated that (at least for these measures of performance), having radio communications and digital maps did not generally improve team performance. The lack of significant differences for the mission performance indicators was surprising. It seemed intuitive that having radios may improve the efficiency and quality of communications, and that this would be associated with improved performance Team Process Another possible advantage of having radio communications and maps is improvement in team processes. One might expect team processes, such as status awareness and teamwork, in which section members have radio communication and/or maps to be more effective and/or efficient. One also might expect, however, that soldiers might have been overloaded with two new pieces of equipment and that this might have masked any improvements in performance. Status Awareness: Radios may provide an easy conduit for information. Such information may increase awareness of one s own AG, and a better sense of one s own AG timings. Providing soldiers with radios and maps has the potential to enhance the status awareness of section members. At a critical stage of the mission (e.g. immediately following an engagement with the enemy), the game server was paused and soldiers indicated their awareness of the casualties and timings, with respect to their own AG and the other AG, using a 4 point scale ranging from 0 (not at all) to 4 (very) aware. As Table 15 shows, there were no significant differences in relative status awareness among the 4 conditions with respect to awareness of own or others timings, or awareness of casualties. Table 15: Status awareness ratings Paper/No Radio Paper/ Radio Digital/No Radio Digital/ Radio Sig. Differences Mean (StDev) Awareness of others timings 2.8 ± ± ± ±.7 ns Awareness of own timings 2.8 ± ± ± ±.7 ns Awareness of casualties 1.4 ± ± ± ±.5 ns This result suggests that at least in this study, a digital map and radio did not improve awareness of one s own status, or of the other members of the assault group. It is important to note out, however, that this finding may speak as much to the implementation of the digital map feature within the 1 st person gaming lab. The digital map provided was relatively small, and it was somewhat difficult to toggle to the map during intense mission periods. This might suggest that a different digital map system may show more accurate the impact of digital map capabilities on awareness. Humansystems Intrasection Comms Lab II Page 31

43 Teamwork: Soldiers completed a teamwork questionnaire to assess the performance of each AG, and the section as a whole, for a variety of teamwork dimensions including anticipation of information needs, coordination, communication behaviour, monitoring behaviour, feedback behaviour, backup behaviour, and team orientation. This questionnaire included 14 items, and soldiers used a 7-point scale, ranging from 1 (strongly disagree) to 7 (strongly agree) to rate these items (see Annex A). As seen in Table 16 and Figure 14 below, a repeated measures analysis on the teamwork index showed a main effect of radio, F (1,23)=11.1, p <.05. Table 16: Teamwork Paper/No Paper/ Digital/No Digital/ Sig. Mean (StDev) Radio Radio Radio Radio Differences Index 5.4 ± ± ± ± 1.0 R>NR Teamwork Score No Radio Radio Mean Mean±SE Mean±SD Figure 14: Teamwork Means Teamwork was rated significantly higher in the radio conditions than in those without a radio. The main effect of map and the interaction were not significant. It appears that soldiers were most impressed by their teamwork when they had radios, regardless of maps. In other words, having a radio seems to facilitate the perception of teamwork within military teams. It is important to note that all ratings were above the neutral point on the teamwork scale, suggesting that soldiers were overall more positive than negative about their levels of teamwork, irrespective of condition. Workload: Soldiers also completed a NASA TLX questionnaire, indicating the levels of subjective workload experienced during the mission. This version of the NASA TLX requires the participant to rate six measures of mental workload on an 18-point scale ranging from high (18 represents demanding, complex) to low (1 represents easy, simple). Items included mental, physical, and temporal demand, as well as task performance, effort, and frustration. Humansystems Intrasection Comms Lab II Page 32

44 Mean (StDev) Table 17: Workload Paper/No Radio Paper/ Radio Digital/No Radio Digital/ Radio Significant Differences Index 6.4 ± ± ± ± 3.3 R<NR A repeated measures analysis on the workload index showed only a main effect of radio, F (1, 23) = 16.7, p <.01, such that perceived workload was significantly lower when soldiers had a radio Workload Score No Radio Radio Mean Mean±SE Mean±SD Figure 15. Workload The main effect of map and the interaction were not significant. This result suggests that having a radio decreases the overall perceived workload. In a previous study (Adams, Tack & Sartori, 2005a) in which we compared having a radio to not having a radio, soldiers indicated that they experienced more workload when they had radios than when they did not. In this study, however, radios do seem to reduce soldiers workloads and energy expenditure. Summary: Overall, the findings here indicated that soldiers perceived increased teamwork and decreased workload with the addition of radios. The fact that they did not experience increased status awareness with either or both pieces of equipment is somewhat puzzling, however. One would expect greater status awareness with the introduction of a digital map given that it indicates each section member as a point. On the other hand, it might also be that tactical status awareness is more relevant at the section level than at the individual level. This might have been a problem as the digital map used in this experiment providing information at the individual level, raising the possibility that this cluttered section members status awareness. Humansystems Intrasection Comms Lab II Page 33

45 4.7 Exit Questionnaire and Focus Group At the end of the missions, participants completed the exit questionnaire and participated in a focus group designed to explore soldiers overall perceptions regarding information transfer, coordination between and within AGs when soldiers carried a radio. Soldiers overall perceptions regarding location awareness, AG coordination, section coordination, and overall acceptance when they carried a digital map compared to a paper map were also probed. For all questionnaire items, soldiers were asked to rate their opinions on 7-point scales from 1 Completely Unacceptable to 7 Completely Acceptable. All items were analyzed using dependent t- tests and significant differences are indicated. In addition, a final focus group was held to discuss these issues as well Radio versus No Radio Information Transfer: Soldiers completed two sets of rating scales assessing acceptability of the radio systems for transfer of information using either the radio communication system, or using current operational procedures without enhanced communication capabilities. The results for information transfer are shown in Table 18. Table 18: Information transfer Mean (StDev) No Radio Radio Significant Differences Amount of info that can be passed 3.6 ± ±.5 R > NR, t (31)=-14.6, p<.001 Time required to pass information 3.7 ± ±.7 R > NR, t (31)=-9.1, p<.001 Timeliness for initiating comms 4.1 ± ± 1.0 R > NR, t (31)=-5.5, p<.001 Ease of passing information 3.5 ± ±.6 R > NR, t (31)=-10.6, p<.001 Ease of requesting information 3.2 ± ±.7 R > NR, t (31)=-10.4, p<.001 Ease of receiving information 3.6 ± ±.5 R > NR, t (31)=-12.9, p<.001 Ease of getting acknowledgement of information received Ease of getting acknowledgement of information understood 3.6 ± ±.6 R > NR, t (31)=-9.9,p< ± ±.7 R > NR, t (31)=-10.8, p<.001 Accuracy of information passed 3.5 ± ±.8 R > NR, t (31)=-9.8, p<.001 Message detail possible 3.4 ± ± 1.0 R > NR, t (31)=-7.8, p<.001 Results show that soldiers rated information transfer significantly more acceptable on every item when they had radios compared to when they did not. In fact, soldiers rated information transfer without a radio as only barely unacceptable to borderline acceptable. Coordination and Awareness Within and Between Assault Groups: Having radio communications will also likely impact the ability of section members to coordinate their actions, and to have high status awareness given the potential increase in information. Moreover, whether section members are working to communicate within their specific AGs, or with AGs approaching the enemy objective from a different route, coordination and awareness are likely to be higher both within and between AGs. Humansystems Intrasection Comms Lab II Page 34

46 Soldiers ratings of overall coordination within their AGs with and without a radio are presented in Table 19. Table 19: Coordination within AGs Mean (StDev) No Radio Radio Significant Differences Coordination of movement 4.5 ± ±.9 R > NR, t (31)=-7.3, p<.001 Coordination of fire 4.2 ± ±.7 R > NR, t (31)=-7.6, p<.001 Coordination of action 4.3 ± ±.7 R > NR, t (31)=-8.4, p<.001 Issuing/receiving orders 4.1 ± ±.8 R > NR, t (31)=-9.4, p<.001 Designating targets 4.2 ± ±.9 R > NR, t (31)=-7.0, p<.001 Ease of distributing message with assault group 3.9 ± ±.6 R > NR, t (31)=-8.2, p<.001 Awareness of location of your own assault group Awareness of casualties among your assault group members Awareness of ammunition usage of your assault group 4.7 ± ± 1.0 R > NR, t (31)=-5.9, p< ± ±.8 R > NR, t (31)=-5.5, p< ± ± 1.3 R > NR, t (31)=-5.5, p<.001 Results show that soldiers rated coordination and awareness within their AGs significantly better on every item with a radio than without. Overall, soldiers indicated that coordination and awareness within their AG was borderline acceptable without a radio. Similar results were evidenced for items related to coordination between AGs, shown in Table 20. Table 20: Coordination between AGs Mean (StDev) No Radio Radio Significant Differences Coordination of movement 3.6 ± ±.7 R > NR, t (31)=-9.8, p<.001 Coordination of fire 3.5 ± ±.7 R > NR, t (31)=-11.1, p<.001 Coordination of action 3.7 ± ±.7 R > NR, t (31)=-10.7, p<.001 Issuing/receiving orders 3.4 ± ±.8 R > NR, t (31)=-10.5, p<.001 Designating targets 3.5 ± ± 1.2 R > NR, t (31)=-8.9, p<.001 Ease of distributing message with Section 3.2 ± ±.6 R > NR, t (31)=-10.6, p<.001 Awareness of location of the other assault group 2.8 ± ±.8 R > NR, t (31)=-12.4, p<.001 Awareness of casualties of the other assault group members 2.3 ± ± 1.4 R > NR, t (31)=-11.5, p<.001 Awareness of ammunition usage of other assault group 2.2 ± ± 1.9 R > NR, t (31)=-7.9, p<.001 Humansystems Intrasection Comms Lab II Page 35

47 Again, abilities to perform all of the necessary coordination and awareness activities without a radio were rated overall as significantly less acceptable. These differences were even greater in the between AGs questions than the within AGs items. That is, soldiers rated coordination and awareness between AGs as reasonably to barely unacceptable, but within their AGs as borderline acceptable. Soldiers were also asked to rate their overall acceptance of having a radio vs. not on the same scale as described above. As Table 21 shows, there was a clear preference for having a radio in infantry operations. Table 21: Overall acceptance of the radio conditions Mean (StDev) No Radio Radio Significant Differences Overall acceptance 3.4 ± ±.6 R > NR, t(31)=-11.0, p< Paper Map versus Digital Map Location Awareness: Having a digital map is likely to affect soldiers awareness of the location of various parties. Soldiers were asked to rate the acceptability of having a digital map compared to a paper map for awareness of their own location, their own AG members locations and the other AG s location. The means and standard deviations are presented in the table below. Table 22: Location awareness Mean (StDev) Paper Map Digital Map Significant Differences Awareness of own location 4.2 ± ±.5 Digital>Paper, t(31)=-8.2, p<.001 Awareness of location of your own assault group 4.6 ± ±.5 Digital>Paper, t(31)=-6.6, p<.001 Awareness of location of other assault group 3.2 ± ±.8 Digital>Paper, t(31)=-12.6, p<.001 The table above clearly indicates that soldiers found the digital map significantly better for their awareness of location in every way: personal, other teammates, and the other AG. They rated their awareness with a paper map as barely unacceptable to borderline. Coordination Intuitively speaking, the presence of a visual referent should also improve the coordination of various aspects of the missions. The results from these questionnaire items concerning aspects of their own AG operations are presented in the table below. Table 23: Coordination within AGs Mean (StDev) Paper Map Digital Map Significant Differences Coordination of movement 4.6 ± ±.8 Digital>Paper, t(31)=-6.0, p<.001 Coordination of fire 4.8 ± ± 1.3 Digital>Paper, t(31)=-3.3, p<.01 Coordination of action 4.4 ± ±.8 Digital>Paper, t(31)=-5.9, p<.001 Issuing/receiving orders 4.1 ± ± 1.5 Digital>Paper, t(31)=-5.0, p<.001 Designating targets 4.3 ± ±.9 Digital>Paper, t(31)=-6.2, p<.001 Sharing information 4.3 ± ± 1.0 Digital>Paper, t(31)=-5.7, p<.001 Humansystems Intrasection Comms Lab II Page 36

48 The table above shows that soldiers rated having a digital map significantly better than having a paper map for every aspect of the mission in their own AG. They rated having a paper map as borderline acceptable and having a digital map as reasonably acceptable for each item. Similar results were found for the coordination of the section as a whole. Table 24: Coordination between AGs Mean (StDev) Paper Map Digital Map Significant Differences Coordination of movement 4.2 ± ±.7 Digital>Paper, t(31)=-7.8, p<.001 Coordination of fire 4.2 ± ±.8 Digital>Paper, t(31)=-6.7, p<.01 Coordination of action 4.2 ± ±.7 Digital>Paper, t(31)=-6.9, p<.001 Sharing information 3.9 ± ± 1.1 Digital>Paper, t(31)=-2.1,, p<.001 Ability to maintain mission tempo 4.2 ± ± 1.3 Digital>Paper, t(31)=-6.1, p<.001 Ability to meet mission timings 3.9 ± ± 1.1 Digital>Paper, t(31)=-7.4, p<.001 Anticipating actions of other AG 3.6 ± ± 0.8 Digital>Paper, t(31)=-7.9, p<.001 Again, the table above shows that soldiers rated having a digital map significantly better than a paper map for every aspect of the mission measured at the section level. Specifically, they rated the digital map as barely to reasonably acceptable, but having a paper map as reasonably to barely unacceptable. The table below shows soldiers overall acceptance level for the digital map compared to the paper map. Table 25: Overall acceptance of maps Mean (StDev) Paper Map Digital Map Significant Differences Overall acceptance 3.8 ± ±.7 Digital>Paper, t(31)=-8.5, p<.001 Again, soldiers responses showed a very strong preference for having a digital map over the current inservice condition that provides only a paper map. In fact, the paper map on its own was rated as unacceptable. Finally, soldiers were also asked several general questions concerning their opinions of the radios and digital maps on a 7-point scale from 1 Strongly Disagree to 7 Strongly Agree. Results of the means and standard deviations for each item are presented below. Note that some of these items involve hypothetical opinions (e.g., comparing night versus day) given their experience in the laboratory. Humansystems Intrasection Comms Lab II Page 37

49 Table 26: Soldiers level of agreement with radio comms and digital maps Mean (StDev) We were more effective as a team with the radio communication system than with the no-radio system. All members of a dismounted infantry Section should be provided with radio comms during a mission. We were more aware of our situation with the no-radio system than with the radio comms system. Assault Group command and control were improved with the radio comms system, over the no-radio system. Section command and control were improved with the noradio system, over the radio comms system. Agreement rating 6.7 ± ± ± ± ± 1.5 The digital map was more useful than the paper map. 4.8 ± 1.4 We were more aware of our situation with the digital map system alone than with the radio comms system alone. Assault Group command and control were improved with the digital map alone, over the radio comms system alone. Section command and control were improved with the radio comms system alone, over the map system alone. With a digital map system, we did not need to use the radio comms system as often. All members of a dismounted infantry Section should be provided with a digital map during a mission. All members of a Section should be provided with both a radio and a digital map during a mission. 3.1 ± ± ± ± ± ± 1.9 It is clear from the table that soldiers supported the use of digital maps, but found them most helpful when used in conjunction with a radio. Nonetheless, the majority of participants agreed that all section members should have both a digital map and a radio in future dismounted infantry operations. 4.8 Limitations This study has some limitations. Key among these is that the provision of digital map capability does not parallel actual digital map usage in the field. As such, the results evidenced in terms of status awareness and team performance may not have been provided the best possible test of the utility of a digital map. This study is also limited by the relative inexperience of the section members in performing tactical assault missions. Lastly, as noted earlier, the implementation of the digital map feature within the 1 st person gaming lab appeared to be somewhat problematic. The digital map provided was relatively small, and it was somewhat difficult to toggle to the map during intense mission periods. This might suggest that a different digital map system may be necessary before performance improvements could be evidenced. Humansystems Intrasection Comms Lab II Page 38

50 5. Discussion Currently, soldiers other than commanders in the CF do not have the benefit of radios or maps, but soldier modernization programs in other nations have shown a trend toward using radios, and recent technological advancements have made it possible for all soldiers to carry a personal digital map. The purpose of the present study was to determine whether the addition of radios and digital maps improved various aspects of simulated tactical assault missions, including communication, team performance, and team processes. The first aim of this study was to explore the frequencies of types and functions of communication with and without radios and digital maps. We found that AGs communicated about twice as often with radios than without, regardless of map condition. In general, orders were the most frequent type of communication, followed by transfers of information. Differences in condition indicated that there were more orders in the no radio conditions than the radio conditions, regardless of the map condition. Like previous studies, the introduction of the radio also witnessed a rise in acknowledgements. Requests, on the other hand, remained fairly constant across the conditions. This suggests that radios and not necessarily digital maps are conducive to other forms of communication. It is also possible that orders are over-used in no radio conditions because they are easier to clearly communicate than other types of communication. In any event, the variation in communication type was a result of the radio and not the digital map. We were also interested in whether communications concerning location would change with the addition of radios and especially digital maps. This analysis was exploratory in that we were unsure as to whether the use of a digital map would reduce communications for location due to the common model of their geographic location. Overall, there was little difference in the frequencies of communication about location, except that they were slightly higher when everyone had a digital map. This was not unexpected as all members of a section might find they are better able to communicate about location when they have a digital map than when they do not. However, it could be argued that if a digital map made missions more efficient by showing section members location, fewer communications should be devoted to this function. Another area for exploration included how the introduction of the radio and digital map would impact the different roles. Overall, the amount of communication across roles (SCs, 2ICs, and AG members) was evenly distributed. Within conditions, however, the SCs increased their communication when they had a radio, whereas the AG members decreased theirs. The 2ICs showed little change. Like the previous studies, the SCs and 2ICs most common type of communication was the order, followed by transfers of information. However, where the frequency of orders decreased and transfers increased with the advent of the radio for SCs, this was not entirely true for 2ICs. The frequency of transfers remained the same in the Digital/No Radio and the Digital/Radio conditions. Moreover, the 2ICs number of orders exceeded 70% in the Digital/No Radio condition. When communicating, section members used the AG networks more often than the all-section members net and the private leader net. The second aim of this investigation was to compare the outcome measures, such as team performance and team process, on missions with or without a radio communications network and digital map. Like the previous laboratory study, team performance was measured by the number of rounds taken and expended by soldiers and the number of AG deaths in each of the four conditions. Results indicated no Humansystems Intrasection Comms Lab II Page 39

51 consistent performance impacts evidenced with the addition of a digital map and radio communication. Results exploring team processes (e.g. status awareness, teamwork, and workload) also showed no significant differences in status awareness among conditions. However, participants perception of teamwork was significantly higher in missions that included a radio compared to those that did not. Moreover, ratings of perceived workload were also significantly lower with a radio than without. It appears then that the introduction of the radio impacts soldiers perception of teamwork and workload in positive ways. The exit questionnaires and focus groups, however, strongly supported the use of both digital maps and radios in infantry operations, showing digital maps as significantly improving location awareness, and assault group and section coordination over paper maps alone. Radio communications were seen to facilitate information transfer, and coordination both in and between assault groups. In addition, acceptability ratings for both digital maps and radios were significantly higher than the current inservice capabilities (no digital maps or radios). As the exit questionnaire and focus groups findings run somewhat counter to the research findings related to team process and performance, it is important to explore this contradiction in more detail. Even though they clearly indicated their preference for it during the focus groups, teams did not seem to perform differently with a digital map and radio. One possible way to view this finding is that the impact of having a digital map and radio may actually impact on team performance, but not on the indicators available in this study. Moreover, the very nature of the simulation and the kind of mission types used in this research may also have resulted in reduced outcome variability. In addition, one might also argue that the experience of completing several missions using this new technology may not enable clear performance differences between conditions, because of the many other factors likely to impact on mission performance. These include the novelty of the equipment and adjusting to using it, and working together as a new adhoc team (and the time to actually get coordinated). In some sense, the missions that the sections performed may be seen as having incrementally contributed to their knowledge of and ability to work with the digital map and radio equipment at the cost of obvious performance differences among conditions while this experience was accumulating. It may be that clear performance differences may be seen with more experience and ability to learn how to both use the digital map and radio systems more adeptly. However, it is important to explore whether the value of having a digital map (both in terms of team performance and process) may be evident in a more realistic setting. The next step is clear. SIREQ must now focus on determining the effects of having digital maps and radios in the field. Humansystems Intrasection Comms Lab II Page 40

52 6. References A. Adams, B., Tack, D., and Sartori, J. (2005). Evaluation of Intra-Section Voice Communication in Dismounted Infantry Operations. DRDC T CR Toronto, ON: Defence Research and Development Canada Toronto. B. Adams, B.D, Tack, D.W., and Thomson, M.H. (2005). Field Evaluation of Radio Communication Network Configurations in Dismounted Infantry Operations. DRDC T CR Toronto, ON: Defence Research and Development Canada Toronto. C. Entin, E.B., Entin, E.E., MacMillan, J., and Serfaty, D. (1993). Structuring and Training High Reliability Teams. U.S. Army Research Institute (Technical Report-599). D. Sperry, D. L. (1995). Distinguishing the communication and coordination differences between superior and good teams in tactical scenarios. Master s Thesis. Naval Postgraduate School, Monterey, CA. Humansystems Intrasection Comms Lab II Page 41

53 Annex A: Mission Map Humansystems Intrasection Comms Lab II Page A-1

54 Mission Map Humansystems Intrasection Comms Lab II Page A-2

55 Annex B: Questionnaires Humansystems Intrasection Comms Lab II Page B-1

56 Personal Information Questionnaire Please provide the requested personal information in the spaces provided: 1. JOB DETAILS: MOC kkkk Unit Abbreviation Rank Pte Cpl/MCpl Sgt/WO MWO/CWO Jnr Offr Snr Offr $ $ $ $ $ $ Sub-Unit (e.g. B Coy) Job Title (e.g. C9 Gunner, Medic, Signaller ) jjjjj jjjjjjj jjjjjjjjj 2. LAND FORCE EXPERIENCE: Length of Service 0 1 Year 1 5 Years 5 10 Years Over 10 Years $ $ $ $ Operational Experience (By Theatre) (more than one selection is permitted) Middle East Golan Heights Sinai Croatia Bosnia Rwanda $ $ $ $ $ $ Cyprus Somalia Cambodia Haiti Kosovo Other $ $ $ $ $ $ Skills Rate your planning skills Terrible Poor Fair Good Excellent $ $ $ $ $ Rate your map reading skills Terrible Poor Fair Good Excellent $ $ $ $ $ Rate your hand signal communication skills Humansystems Intrasection Comms Lab II Page B-2

57 Terrible Poor Fair Good Excellent $ $ $ $ $ Rate your tactical movement skill Terrible Poor Fair Good Excellent $ $ $ $ $ Rate your target engagement skill Terrible Poor Fair Good Excellent $ $ $ $ $ Experience Rate your training and/or operational experience planning missions. None Some Moderate Extensive $ $ $ $ Rate your training and/or operational experience conducting Section assaults. None Some Moderate Extensive $ $ $ $ Rate your training and/or operational experience using maps. None Some Moderate Extensive $ $ $ $ Rate your training and/or operational experience in urban operations. None Some Moderate Extensive $ $ $ $ Humansystems Intrasection Comms Lab II Page B-3

58 Humansystems Intrasection Comms Lab II Page B-4

59 Teamwork PARTICIPANT NUMBER : MISSION NUMBER : ASSAULT GROUP : SECTION NUMBER : RADIO CONDITION: RADIO NO RADIO $ $ MAP CONDITION: MAP NO MAP $ $ ROLE: RIFLEMAN 2IC SECTION COMD $ $ $ SECTION A: Rate the degree to which you agree with the following statements using the scale provided. Please consider your responses to these scales carefully: Agreement with Statement Strongly Strongly Disagree Neutral Agree My assault group coordinates well in completing this mission. My assault group leader provides good direction in helping our assault team to meet this mission s goals. During this mission, my team members are able to anticipate my actions and I am able to anticipate theirs. Teamwork increases to our assault group s ability to complete this mission. During this mission, my assault team members have failed to give me the information that I need to make decisions. Our assault team shows a poor level of cooperation during this mission. In this mission, my assault group and I work well as a team. Humansystems Intrasection Comms Lab II Page B-5

60 SECTION A: Rate the degree to which you agree with the following statements using the scale provided. Please consider your responses to these scales carefully: Agreement with Statement Strongly Strongly Disagree Neutral Agree Our assault group is able to accomplish more as a team than as individuals. I am not confident in the abilities of my teammates during this mission. The members of my assault team have a common view of how to complete this mission. In general, my assault group works well as a team. SECTION B: COMMENTS Humansystems Intrasection Comms Lab II Page B-6

61 Workload Questionnaire NAME : MISSION NUMBER : ASSAULT GROUP : SECTION NUMBER : CONDITION: VOICE NO VOICE ROLE: RIFLEMAN 2IC SECTION COMD Section A: Rate the trial by marking each scale at the point which matches your experience. Each line has two endpoint descriptors to help describe the scale. Please consider your responses to these scales carefully. MENTAL DEMAND (thinking, deciding, searching, remembering) Low (easy, simple) High (demanding, complex) PHYSICAL DEMAND (controlling, operating, activating) Low (easy, restful) High (demanding, laborious) TEMPORAL DEMAND (time pressure) Low (leisurely) High (frantic) PERFORMANCE (how successful and how satisfied were you with performing this task?) Good Poor EFFORT (how hard did you have to work, both mentally and physically?) Low High FRUSTRATION Low (gratified, complacent) High (discouraged, annoyed) Humansystems Intrasection Comms Lab II Page B-7

62 Section B: Comments (Use back of page if required) Humansystems Intrasection Comms Lab II Page B-8

63 Humansystems Intrasection Comms Lab II Page B-9

64 Humansystems Intrasection Comms Lab II Page B-10

65 Humansystems Intrasection Comms Lab II Page B-11