Using VRML and Collaboration Tools to Enhance Feedback and Analysis of Distributed Interactive Simulation (DIS) Exercises

Using VRML and Collaboration Tools to Enhance Feedback and Analysis of Distributed Interactive Simulation (DIS) Exercises Julia J. Loughran, ThoughtLink, Inc. Marchelle Stahl, ThoughtLink, Inc. ABSTRACT: Military training exercises using Distributed Interactive Simulation (DIS) are often recorded and played back so trainees can visualize the 3D exercise and receive feedback in the form of an After Action Review (AAR). This paper discusses the capability of converting DIS exercises into virtual worlds accessible over the Internet using the Virtual Reality Modeling Language (VRML). The Internet accessibility of VRML, combined with other network-based collaboration tools (e.g., video-teleconferencing and whiteboarding) supports the implementation of a remote expert where geographically dispersed teams of trainers can provide feedback to trainees in other locations. The VRML exercise file can be annotated with multi-media collaboration input and analytical information. This file can later be used to validate a training package, train new recruits, and contribute to a lessons-learned repository. 1. INTRODUCTION This paper discusses the remote expert concept, in which a team of experts and one or more teams of students interact with each other using various kinds of collaboration technology: audio, video, whiteboarding, and shared visualization of the three-dimensional (3D) simulation replay by converting distributed interactive simulation (DIS) exercises into the Virtual Reality Modeling Language (VRML). Multi-media annotations can be inserted into the 3D environment. These annotations are inserted by participants during the collaboration and can be video, audio, or text files. In addition, analytical information generated by intelligent agents and computer generated algorithms, can be inserted and visualized in the 3D world. These human and computer generated annotations can later be used to gain a better understand of the training event, provide a learning tool for new trainees, and also contribute to a pattern repository of training data. This training data repository can be used for generating lessons learned and for tracking individual student and group performance statistics. 1 Distributed Interactive Simulation (DIS) is a complex, simulation training environment that is commonly used by the U.S. military. It is particularly well suited to training collective tasks, such as command and control or movement of forces. Studies have shown that effective learning in this environment comes when students view a 3D replay of the training exercise and simultaneously receive structured feedback in the form of an After Action Review (AAR). This feedback generally comes from the trainees commander, or a military training expert, referred to by the U.S. Army as an Observer/Controller (O/C). Although training with O/Cs is very effective, it s also costly. The remote expert concept can be applied to DIS training with O/C teams. This may help reduce costs, by allowing O/Cs and unit to remain at different sites. There are fewer travel expenses, increased utilization of fewer O/Cs, and richer feedback to units. 1.1 U.S. Army s Use of Structured Training in Virtual Environments The U.S. Army has known for some time that structured training, with a trainer providing feedback to the units, is more effective than allowing units to do freeform training. In free-form simulation training, the units pick what they want to train, develop their own scenario, get in the simulators, and execute the training. In structured training, scenarios are developed by mapping training objectives to the tasks to be trained in the simulation environment. These tasks are monitored by the commander or O/C and feedback is provided during the AAR. The key role of the O/C during the AAR is to help the unit identify what they did well, what they did poorly, and what skills they need to focus on in the next scenario they will train. Successful examples of structured training using virtual environments include the Virtual Training Program (VTP) for training Company and Platoon collective tasks and the Unit Conduct of Fire Trainer (UCOFT) for two-person tank training. Structured training, like that implemented in the programs mentioned above, will become increasingly difficult to conduct in the upcoming decade as experienced mentor and financial resources in the U.S. Army become less available. For example, the U.S. Army s Close Combat Tactical Trainer (CCTT) program -- supplying the next generation of man-in-the-loop simulators -- does not have program funding for O/Cs. In addition to the AAR, another form of feedback provided in the structured training environment is the Take Home Package (THP). Although THPs may vary across training programs, they are generally a textual document that details the training event, describes the

exercises and tasks trained, and identifies those tasks that need further training. Anecdotal evidence suggests that units rarely review the information contained in them once the training event is concluded. This paper will discuss the generation of multi-media THPs that could potentially be much more effective. It would allow units to replay their previous training in 3D, replay the O/C s comments in audio, and, with some additional work, might even allow the unit s training events to serve as a starting point for what-if analyses with a constructive model to explore alternative outcomes. 2.0 THE REMOTE EXPERT CONCEPT The remote expert concept envisions a rich 3D space in which a geographically dispersed team of experts (O/Cs in this military training example) interact with students (the unit being trained). DIS sites typically have a two-dimensional (2D) and 3D replay capability, and these are currently extensively used during AARs. In the remote expert concept, these virtual environments would be extended to directly incorporate collaboration technology and would allow participants (O/Cs or the unit training) to annotate the virtual environment. Analytic information could also be overlaid into the virtual environment. The remote expert concept allows for a highly skilled core of O/Cs to remain at one or a few locations and interact virtually with units performing training at their home stations, CCTT or other simulation sites. This interaction includes the creation, storage, and access of multi-media annotations and the analytical augmentation of 2D and 3D environments. Figure 1 at the end of this document shows how the remote expert concept links units training with an O/C, using a local or wide-area network. In addition, other units can observe the AAR and simulation replay from their home stations, providing a valuable tool for them to learn from others experiences. 2.1 An Example As an example, assume a unit has been told they are being deployed to some crisis in a foreign country. They schedule their training and contact a remote O/C. The unit trains at the nearest training site, which may be their home station, while the O/C remains at his or her home station. The remote O/C reviews their training either in real-time, or after it has been completed, using a variety of tools, including: the training pattern repository (described below), analytic visualization tools, and 2D and 3D exercise replay. 2 After the training, the unit and the remote O/C meet via VTC for the AAR. Important decisions and events are reviewed by both the O/C and the trainees using the 2D and 3D replay capability. Both the O/C and the trainees can stop the replay at any time to insert an annotation. The annotation may be graphical, similar to a whiteboard over the 3D environment. For example, the O/C could mark the phase lines of the battle, routes that should have been taken, points at which particular events occurred, etc. Voice and text can also be used to annotate the exercise replay. An example of the use of voice recordings might be to have the O/C narrate parts of the battle, describing the situation, or the unit may record voice annotations describing their perspective of the battle, e.g., why they made certain decisions and what their information/ situational awareness was at the time. See Figure 2 for an example of the 3D environment with a textual annotation. These annotations can be inserted while the O/Cs and the unit are communicating synchronously however, asynchronous annotation could also be supported. While the unit is observing the 3D replay, the intelligent agent and computer based algorithms are evaluating different pre-defined Measures of Performance (MOPs). These MOPs can be embedded in the visualization, showing information such as the number of shots and hits for each tank or the evaluation of the unit s formation at any point in the battle. One of the benefits to this environment is that it runs over a network (either the Internet or a secure network) and a variety of commercial off-the-shelf (COTS) tools can be used. Figure 3 shows a hypothetical implementation of the remote expert concept using a Netscape browser, COTS VTC tool, a text chat window, and whiteboarding technologies over the 2D display. In addition, the exercise is being replayed in VRML. Various portions of the original exercise and AAR can be recorded and provided to the training unit as a multimedia THP. Both the audio and visual portions of the interactive session can be recorded. Selected parts of this, or even all of it, could be given to the unit for their future review. This kind of multimedia feedback, that units could review in a vivid 3D world after their initial training, could lead to greater retention of the subject matter. A natural extension of this feedback would be the ability to use the unit s training experience as a starting point for what-if analyses. Software could be written to translate the recorded remote expert session into data to be fed into another simulation, probably constructive or at least semi-automated. The data would create the initial conditions for the simulation. Then other choices and outcomes could be explored using the simulation. Given many instances of a training exercise, a pattern repository could be created for the O/C team. O/Cs recognize that there are certain areas where units

encounter the same set of problems. By creating a repository of patterns, these problem areas can be automatically identified by the system itself and, potentially, alternative training can be provided to help the unit in that particular area. The repository can also be used by curriculum developers so that course materials are focused in the areas where units seemed to encounter the greatest number of problems. If a pattern repository were created, the role of the expert could augmented or replaced to some extent by a virtual agent, who could guide units through the most kinds of problems and solutions. This could substantially lower the cost of providing the remote expert service. 2.2 Multi-media Annotations VTC, text chat, and whiteboarding allow the O/Cs and trainees to create annotations which can be inserted at any time or location in the virtual environment. These annotations can clarify a point, capture teacher feedback, or elaborate on a student s perspective or understanding of the experience. The annotations are captured during the simulation replay in the AAR, and can be accessed again when replaying an archived, annotated file. 2.3 Analytical Augmentation Analytic or symbolic information can be presented as bar charts, text, graphical overlays or with the use of color. Analytic results can be computed in the virtual environment at run-time or more typically, while replaying a simulation. These computations represent pre-defined MOPs associated with the training goals of the simulation. 3 Interesting work in this area has been conducted at the University of Southern California s Information Sciences Institute. 4 Under the PROBES project, intelligent agents help in monitoring overall team behavior. Based on a description of the training objectives, an intelligent agent (the Puppet Master) collects information, analyzes the data, and presents the data (in real-time) to the instructor in an augmented 3D display. An example would be the use of colored bars overlaid in the 3D world to show when and if tanks in a platoon were travelling in the correct formation given a particular command or situation. This technology would contribute significantly to the success of the remote expert concept. 3.0 THE TECHNOLOGY Advances in networking technologies, graphics software, collaboration tools, and software standards are enhancing the use of the Internet for education and training applications. Current technologies for webbased 3D virtual environments and collaboration are described below. 3.1 Virtual Reality Modeling Language (VRML) In the early years of the World Wide Web, content consisted mostly of text and static images. Later, sound, video, and software applications could be shared over the Web with the adoption of standards for sharing these file types. VRML is the next evolutionary stage in Web standards. It provides the capability to build, transport, and display 3D interactive virtual worlds. VRML worlds are most commonly accessed through HTML pages. Using a VRML browser as a plug-in to an Internet browser, a user can click on an HTML page, causing a VRML file to be downloaded and displayed. The user can typically navigate to any point in the scene - above, below, in between the graphical elements (however, VRML creators can choose to specify predefined viewing positions that do not allow users to freely navigate the scene). User interaction with the world is supported. Under VRML 2.0, the user may be able to click on an object and cause something to happen, e.g., the user can move an object, the viewpoint changes, or an audio or video recording plays. By translating DIS exercises to VRML, 3D replays can be viewed by anyone with an Internet connection and a browser. In contrast, most 3D DIS replays currently require fairly expensive workstations with fast graphics, running special-purpose and fairly expensive software. The first VRML standard (VRML 1.0) was approved in the Spring of 1995. By the Fall of 1996, VRML 2.0 was approved and added many features, including the capability to play sounds and movies in virtual worlds, interact with objects, and control and enhance worlds with scripts. For more information on VRML, see the VRML Repository at http://www.sdsc.edu/vrml. The VRML Repository is an impartial, comprehensive, community resource for the dissemination of information relating to VRML. It is maintained by the San Diego Supercomputer Center (SDSC). 3.2 Collaboration Tools Collaboration technologies such as video teleconferencing (VTC), shared documents and applications, text chat, and whiteboarding are becoming more affordable and available for use over the World Wide Web. Tribal Voice s PowWow (http//www.tribal.com) and Microsoft s Netmeeting (http://www.microsoft.com) are two examples of free multi-user conferencing software that support text chat (PowWow) and VTC (NetMeeting). In addition, the most current releases of Microsoft s Internet Explorer and Netscape s Communicator support collaboration. Various standards have been or are being defined, including H.263, H.323, H.324, T.120, and MPEG-4, for

transmission of various media types over networks. As these standards become more widely used, the variety and quality of collaboration tools will likely increase. 3.3 Technology Challenge The process of integrating virtual worlds, simulation, and collaboration technologies has not yet been realized in any commercially available products. By combining these technologies, voice, video and textual annotations can be captured in the virtual environment and later visualized when the virtual world data is replayed. 4. WORK TO DATE Under work conducted at the Institute for Defense Analyses, the authors, with others, researched and implemented tools for converting DIS exercises into VRML for 3D replay. 5 As part of an earlier task, this team developed data collection, reduction, and analysis techniques to assist in the analysis of one year s worth of SIMNET training exercises. SIMNET is the precursor to DIS. Binary SIMNET data was reduced and stored in a standard ASCII file format called the Logged Event Analysis Format (Leaf). The Leaf data was then converted into a set of VRML files. The VRML files describe the terrain, the vehicle types and their positions over time, and the time and location of significant events (e.g., shots and their results). Using a VRML browser, the files can be replayed and the user can adopt a variety of viewpoints over the battlefield. Analytic information has been embedded in the replay with colored bars over each tank showing the number of shots, hits, and misses for each tank. In order to implement the remote expert concept, collaboration technologies would need to be integrated with the 3D and 2D exercise replay. The authors have created a conceptual prototype of these capabilities at http://www.thoughtlink.com/explorer.html. We are currently researching how currently available COTS tools might be used to generate and capture these multimedia annotations. There are still some technical issues to be resolved, including how these annotations will be stored and replayed. Users would presumably wish to search on these annotations or search for a particular exercise based on a participant and/or his or her comments. Experience working with VRML demonstrates that the current state of the technology makes the implementation of this concept still a research effort. As new versions of browsers are issued, VRML files that used to work no longer work. In addition, the VRML files show different results with different types of VRML browsers. It s difficult to find and maintain a stable combination of VRML files and browsers. This makes the current development environment one that is rapidly changing and fraught with errors. 5. BENEFITS Despite the technical challenges facing the implementation of the remote expert concept, the benefits are many. These are described in more detail. 5.1 Enhanced Training Feedback The remote expert concept combines analytical and collaborative annotations in the 2D and 3D environment to provide a more detailed description of the training event: what occurred, what was done well, different perspectives. This detailed replay can used to create multi-media THPs. If THPs were saved as HTML documents, links could be made to parts of the VRML replay. For instance, the THP might read to see a visualization of the unit s wedge formation, click here. If selected, this would open a VRML window with replay of that part of the exercise. In addition, the multi-media THP might include video of the O/Cs feedback, an audio recording from the Platoon leader, or a textual reference to the task being trained. 5.2 Richer Learning Environment The remote expert concept combines traditional 3D replay currently used in an AAR with other types of persistent annotations. By providing information in multiple forms (audio, video, text, and graphical images) there is more sensory stimulation, providing more avenues for interpretation and understanding. 5.3 Savings of Time and Money The remote expert employs distance learning technologies that help reduce manpower and travel requirements. Having a team of experts co-located at one or a few sites reduces the need for experts at each training site. In addition, by saving and archiving previous training exercises, training sessions can be reused for learning by others. The annotations in particular enhance the understanding of the actions that occurred in previous exercises. In the U.S. Army s Distance Learning Plan (ADLP) 6, two of the desired initiatives include providing virtual reality applications to users at multiple locations and also to support AAR capabilities from remote sites. The remote expert concept addresses these needs through the effective use and integration of VRML and collaboration technologies.

6. ACKNOWLEDGEMENTS Work on the DIS to VRML research was initiated at the Institute for Defense Analyses under a DARPA contract for the Computer Aided Education and Training Initiative (CAETI). 7. REFERENCES [1] J. Loughran, Evaluating Aggregated Performance Data from Distributed Simulation Training Exercises, Proceedings of SCS Western Multiconference, Virtual Worlds and Simulation Conference (VWSIM 98), January 1997 [2] E. Johnson, M. Kappel, J. Loughran, Battle Explorer: VRML Techniques for Replaying and Analyzing DIS Exercises, Proceedings of the Fall Simulation Interoperability Workshop, September 1997 [3] J. Loughran, M. Stahl, Analytical Augmentation of 3D Simulation Environments, Proceedings of SPIE Vol. 3298, Visual Data Exploration and Analysis V, January 1998 [4] W.L. Lewis et al., Pedagogical Agents in Virtual Team Training, Proceedings of SCS Western Multiconference, Virtual Worlds and Simulation Conference (VWSIM 98), January 1997 Virtual Worlds and Simulation Conference (VWSIM 98), January 1998 [5] ThoughtLink, Inc. website: http://www.thoughtlink.com/remote.html, July 1997 [6] Army Distance Learning Plan, http://wwwdcst.monroe.army.mil/adlp/adlp.htm, April 1996 Units training Shared 3D and 2D Representations Trainees Observing with multi-media annotations Remote O/Cs Pattern Repository Figure 1. The Remote Expert Concept

Platoon Leader: We had mistaken the distant rocks to be enemy tanks. Figure 2. Textual explanation embedded in VE tells why an event occurred Video Teleconferencing Text Chat 2D Battle Replay 3D Battle Replay Figure 3. Remote Expert Concept Showing Collaboration and Simulation Replay