Facilitating Human System Integration Methods within the Acquisition Process

Facilitating Human System Integration Methods within the Acquisition Process Emily M. Stelzer 1, Emily E. Wiese 1, Heather A. Stoner 2, Michael Paley 1, Rebecca Grier 1, Edward A. Martin 3 1 Aptima, Inc., Woburn, MA 2 Realtime Technologies, Inc., Royal Oak, MI 3 Air force Research Laboratory, Wright Patterson AFB, OH Warfighters are performing their duties in complex systems that increasingly rely upon interactions among information, operators and warfighters, and advanced technologies. To enhance total system performance, systems must be designed to account for the cognitive strengths and limitations of the warfighter. Traditional systems engineering (SE) practice seldom considers these cognitive factors, or considers them at inopportune times in the design process. The Department of Defense mandates that SE consider general Human Systems Integration (HSI) issues such as human factors, training, manpower, and personnel. Applying cognitive engineering (CE) methods to these traditional HSI areas throughout the system development cycle could result in a reduction in the time and money needed to address these traditional areas. CE contributes to the interdisciplinary CE-SE gap as well as SE. Cognitive engineers lack the contextual, system-specific knowledge needed to integrate human factors practices into the design cycle in ways that are helpful to systems engineers and warfighters. The Resource for Applied Cognitive Engineering and Systems Engineering (www.trace-se.com) is a prototype web-based tool that addresses this problem by providing information needed to simultaneously support both cognitive and systems engineers during the design of usercentered systems. Using TRACE-SE, both cognitive and systems engineers can better understand how to employ their respective processes in order to develop systems that support superior decision making, improved safety, and greater operator productivity. With advances in technology, a number of new tools are being developed to support the warfighter in encountering the challenges that they face each day. To provide this support, it is essential that the design of these technologies takes into account the cognitive and perceptual strengths of the warfighter, as well as the inherent limitations of any human operator. Creating an effective tool for the warfighter requires that the designer has a good understanding of design processes and standards, fully comprehends the human warfighter, and understands when and how the warfighter can be most effectively supported. Thus, to become an effective support tool, a design must be created and brought to realization through the integration of the disciplines of systems engineering (SE) and cognitive engineering (CE). Unfortunately, the systems and cognitive engineering domains have not yet been well integrated. In fact, the gap between systems and cognitive engineering has continued to grow, due to shortcomings in both communities. For example, systems engineers may not be aware of the cognitive engineering domain or understand the primary goal of the community. Even if systems engineers are aware of CE, they may not have a full understanding of the methods and processes used by cognitive engineers or have a complete picture of the benefits that these methods and processes can provide. In a nearly mirrored fashion, cognitive engineers may not be aware of the 1

systems engineering domain or understand the primary goal of the community. Cognitive engineers who are aware of SE still may not have a complete understanding of the methods, processes, and standards used by systems engineers or have a complete picture of the benefits that the methods and processes provide. On both sides of this interdisciplinary gap, the communities are ineffective at sharing information in a way that is useful to their counterparts. Whereas cognitive engineers may describe performance benefits in terms of milliseconds saved in response time or benefits to situation awareness, system engineers may ask for benefits in terms of dollars saved. Because critical information cannot be communicated effectively, the gap remains between these two communities. To begin to bridge the gap between systems engineers and cognitive engineers, the authors are developing The Resource for Applied Cognitive Engineering and Systems Engineering (TRACE-SE). TRACE-SE is a prototype web-based tool designed for both cognitive and systems engineers to provide the information needed to create and support awareness and understanding of the processes and methods used by both the cognitive and systems engineering communities and to facilitate communication between these two groups. Using the TRACE-SE tool, both communities will better understand how to use their respective processes in order to develop systems that support superior decision making, improved safety, and greater operator productivity. In this paper, we describe the development process of TRACE-SE and the informal evaluations that have been conducted on the tool up to this point. We will begin by identifying and describing worry points within the systems engineering process where cognitive engineering techniques could be used to support human-centered development. We also explain how these worry points were organized into relevant themes. We will then describe findings from interviews with subject matter experts and feedback about how they envision using the TRACE-SE tool in engineering unmanned aerial vehicles. Finally, we will show how cognitive and systems engineers can use TRACE-SE to reach HSI and SE milestones and deliver effective products. The TRACE-SE Framework Defining the TRACE-SE Framework. The TRACE-SE framework was designed to serve as the navigational and organizational backbone of the web-based tool. One goal was to design a framework that would inform the cognitive engineer about the processes and milestones of the systems engineering process through a simplified summary of the procedures and products associated with common stages in acquisition. A second goal was to design a framework that would support both systems and cognitive engineers by identifying key opportunities (what we term worry points) when design might be enhanced in the systems engineering process by utilizing cognitive engineering methods. This framework is depicted in Figure 1. Figure 1. TRACE-SE analysis of DoD acquisition and the identification of worry points. 2

In support of the first goal, Aptima conducted a task analysis of the Department of Defense acquisition processes, as outlined in the Defense Acquisition Guidebook and the DoD 5000.1 and 5000.2 Directives (http://akss.dau.mil/dag/). This guidebook describes the five primary phases of design in the acquisition process (Concept Refinement, Technology Development, System Development and Demonstration, Production and Deployment, and Operations and Support), three key milestones (Milestones A, B, and C), and specific tasks and documents that must be completed within each of the five phases. Through this task analysis, Aptima created a simplified summary of the intricate DoD acquisition directives, highlighting the processes that exist at each phase of acquisition, as well as the milestones that must be achieved before design can proceed. In satisfying the second stated goal of the TRACE-SE framework, this analysis was used to indicate to both engineering communities opportune points to integrate cognitive engineering and systems engineering methods, through the identification of TRACE-SE worry points. Worry points identified critical design issues that should be considered during the acquisition process to ensure effective human-centered engineering. Three guiding principles were instantiated to ensure that identified worry points were useful: 1. Worry points were aligned with milestone, review, and decision points defined by the Defense Acquisition System program structure. 2. Worry points were distributed throughout the acquisition process, but were heavily centered on the early phases of acquisition, to ensure sufficient impact in the planning stages of the design process. 3. Because activities associated with each worry point require time and consumer resources, a careful analysis of costbenefit tradeoffs were considered (Bias and Mayhew, 1994; Hendrick, 1996; Nielsen, 1993). Applying these three principles and our knowledge of the acquisition process gleaned from the task analysis, Aptima identified a total of 27 worry points. These worry points were then organized into five larger themes. A subset of these themes and worry points is shown in Table 1. Table 1. Subset of TRACE-SE themes and worry points. THEME Identify and Satisfy User Needs Consider Constraints WORRY POINT Translate user needs into requirements Consider user s future needs Consider environmental conditions Consider personnel constraints Measure skills and train knowledge Each worry point was then specifically examined for effectiveness within the context of each of the five systems acquisition phases. In some cases, worry points were highly relevant in the early phases of the acquisition process and became less effective in guiding design as design proceeded through the later phases. For example, effective and efficient humancentered systems design would consider the worry point Translate user needs into requirements during the Concept Refinement and Technology Development phases of the acquisition process. The impact of this specific worry point would diminish as the acquisition process proceeds into later design phases. 3

In other cases, worry points are important to consider as design proceeds to more advanced stages of the acquisition process. Measuring skills and training knowledge is necessary to consider during the early phases of design and becomes increasingly important during the phases of System Design and Development and Production and Deployment. By mapping each worry point to specific phases in the systems acquisition process, we ensured that the most critical design decisions would be considered at the most opportune times. Applying Relevant Cognitive Engineering Methods. While the identified worry points and larger organizational themes inform the user what factors to consider during systems design and when it is most effective and efficient to consider those factors, these components of the TRACE-SE framework do not expressly define how to consider the factors. To address this need, Aptima reviewed cognitive engineering methods and applied key methods to each worry point. In our review, we identified and summarized over one hundred cognitive engineering methods. Examples include the use of a focus group, conducting a cognitive task analysis, or doing a field observation. To ensure that TRACE-SE remained agnostic to a specific approach or theoretical framework, all reputable methods were reviewed and summarized. Methods were then evaluated and down-selected for their utility, ease of use, and resource requirements. The most effective methods were then applied to the identified worry points. For example, to address the worry point Consider the user s future needs, TRACE-SE suggests conducting a cognitive task analysis or a focus group. The tool provides a brief description of each of the methods and notes key references that the user can seek for more detailed information. While the information provided on the TRACE-SE web resource does not provide sufficient information for the systems engineering user to conduct the methods, it does suggest what methods the user should consider and can assist the user in inferring the required resources to conduct the methods. Summary. Thus, the TRACE-SE framework provides three critical functions. First, the framework has been used to summarize the important, yet complex and intricate processes and products detailed in the DoD Systems Acquisition Guidebook. By summarizing this detailed information, TRACE-SE supports a cognitive engineer in understanding the systems engineering process, the deliverables and key milestones associated with this process, and the language used to discuss this process. By understanding these basic procedures, the cognitive engineer can effectively convey to the systems engineer how specific cognitive engineering methods might be used to enhance design. Second, the TRACE-SE framework defines worry points, which identify the most effective and efficient points within the acquisition process to integrate systems engineering and cognitive engineering approaches. Thirdly, TRACE- SE points to and briefly describes specific cognitive engineering methods that can be used to consider and address the identified worry points. By providing specific methodology information, TRACE-SE can serve as the starting point for ensuring that design includes effective methodology from both engineering communities. TRACE-SE Prototype Capabilities To this point, we have outlined the framework of the TRACE-SE acquisition process support tool. The defined themes and worry points serve as the organizational backbone of the TRACE-SE prototype tool (shown in Figure 2), providing a method for both systems engineers and cognitive engineers to navigate and drill down 4

through design concerns that are relevant to a given acquisition phase. The goal of these envisioned capabilities is to create and house a web community of practice, where members of the cognitive and systems engineering communities can interact, communicate, and share ideas. By creating a place for members of these communities to exchange ideas, significant strides can be made in bridging the gap between the two groups. Upon defining the specific functions of the TRACE-SE tool including the process support components and the community of practice comments of the web resource the functions were implemented in the prototype TRACE-SE tool, which was to be demonstrated and evaluated by experts in the cognitive and systems engineering domains. Figure 2. Prototype TRACE-SE interface, showing navigational panel at the top and site content in the lower two panels. Choose a Theme or Systems Acquisition Phase The TRACE-SE framework, in its most fundamental form, provides a method for supporting the acquisition process by identifying critical design concerns and applicable cognitive engineering methods for a specific acquisition phase. This drilldown method allows the user to access relevant static text-based information to gain a basic understanding of critical design concerns and approaches, as shown in Figure 3. Choose a Worry Point Investigate a CE Method To supplement this fundamental capability of TRACE-SE, Aptima compiled a set of functions that would allow the TRACE-SE tool to become dynamic and support the predicted increase in the exchange of information and knowledge across the two engineering communities. These envisioned functions included multimedia lectures by experts in the cognitive engineering or systems engineering communities to be presented in the style of a webinar, weblogs in which practitioners or students in one of the two communities could describe useful approaches or lessons learned, and message boards on which users could post questions and provide tips and answers to one another. Figure 3. Using the TRACE-SE framework to drill down to design concerns and applicable CE methods from a specific acquisition phase. Evaluation of TRACE-SE To provide a context for the evaluation of the tool, a use case was developed to describe a system engineer s use of the TRACE-SE tool in applying cognitive engineering methods during the development of a new unmanned vehicle platform. This use case was developed 5

using data from interviews that were conducted with the Reconnaissance Systems Wing at Wright-Patterson Air Force Base. The use case defined critical tasks in the systems acquisition process, outlined procedures that would be necessary to complete each of these tasks, and identified how the user would interact with TRACE- SE in conducting these tasks. One step of the use case is outlined in Table 2. Table 2. One example step of the use case. STEP 3: CONCEPT REFINEMENT STAGE System Acquisition Process Activity TRACE-SE User Activity Within this phase, an acquisition strategy is developed leading into Milestone A. The acquisition strategy identifies HSI responsibilities, describes the approach for meeting HSI requirements, and summarizes major elements of the training system required. Using the web resource, the systems engineer is faced with supporting the program strategy development by identifying specific HSI requirements. Worry Points should be considered. After constructing the use case, Aptima conducted an informal demonstration and evaluation of the TRACE-SE prototype tool. The Reconnaissance Systems Wing served as independent evaluators and were encouraged to provide feedback while specifically focusing on identifying the disconnects between the envisioned use of the prototype TRACE-SE tool (as defined in the use case they assisted in developing) and how systems engineers were expected to actually use the tool. The evaluators were also asked to provide feedback about the validity and utility of the processes and methods of both systems engineering (as defined in the summary of the DoD acquisition process) and cognitive engineering. Feedback from these experts indicated that the continued development of TRACE-SE could provide a valuable tool for both cognitive and systems engineering practitioners and would likely enhance the system development process. As additional evaluations are conducted, refinements to the organization, functionality, and the framework will be made to continually develop and improve the tool. Conclusions and Future Directions The TRACE-SE tool serves as an initial, but noteworthy, step in beginning to bridge the gap that exists between the cognitive engineering and systems engineering communities. To address the diverse reasons for the existence of this gap TRACE-SE takes a multi-faceted approach to supporting interdisciplinary design by not only serving as a resource for information about systems engineering processes, critical design issues, and cognitive engineering methods, but also by creating a home location for the creation of a community of practice. This online home provides a forum in which users can share, discuss, and debate ideas and approaches and ultimately create human-centered system designs that are grounded in effective interdisciplinary engineering methods. As TRACE-SE continues to develop, the framework will be broadened to account for system engineering processes beyond those dictated by the DoD Acquisition Guidebook. This broader scope will ensure the applicability and utility of the TRACE-SE tool both within the defense industry and beyond to commercial markets. The framework will also be applied to training and educating student systems engineers and cognitive engineers about the benefits of considering and using interdisciplinary approaches to system design. By educating new members of these communities about the importance of human-centered design 6

within systems engineering, we can increase the frequency with which these methods are integrated and continue to bridge the existing gap. References Bias, R. G. and D. J. E. Mayhew (1994). Cost Justifying Usability. Boston, MA, Academic Press. Department of Defense Directive 5000.1: The Defense Acquisition System (2003). http://akss.dau.mil/dag/help_welcome. asp Department of Defense Instruction 5000.2: Operation of the Defense Acquisition System (2003). http://akss.dau.mil/dag/help_welcome. asp Hendrick, H. W. (1996). Good Ergonomics is Good Economics: Presidential Address to the Human Factors and Ergonomics Society. In Proceedings of the Human Factors and Ergonomics Society 40th Annual Meeting. Human Factors and Ergonomics Society: Santa Monica, CA. Nielsen, J. (1993). Usability Engineering. Boston, MA, Academic Press. Acknowledgments This work was funded under contract FA8650-05-C6892 through AFRL/Wright- Patterson. Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the authors and do not necessarily reflect the views of AFRL. Special thanks to Dr. Emilie Roth s contributions to this work. 7