Agile Engineering of Scalable Enterprise-Level Capabilities Dr. R. Cherinka and Dr. R. Miller The MITRE Corporation 4830 W. Kennedy Blvd., Tampa, FL 33609 Phone: 813-287-9457, Fax: 813-287-9540 rdc@mitre.org, drbob@mitre.org ABSTRACT Net-Centric Solutions, as enabled through distributed service-oriented architectures, will have a significant effect on the way organizations address the development and integration of scalable enterprise-level capabilities. In these cases, we have found that traditional systems engineering approaches take too long and are too costly in order to meet the mission critical demands of users. Current research suggests that agile development offers a model for embracing and promoting evolutionary change throughout the lifecycle of a project. In this paper, we discuss our efforts to scale agile development best practices to engineer, assess and acquire enterprise-level capabilities for very large Department of Defense systems. We discuss a multi-year case study to evaluate and mature capabilities in support of the Global War on Terror. It is based on using a distributed innovation lab environment in conjunction with a series of warfighter workshops focused on themes and challenge problems identified by the user. The workshops are designed to provide hands-on warfighter immersion into emerging processes, concepts and capabilities combined with facilitated discussions to develop and/or refine CONOPS and Tactics, Techniques and Procedures. We highlight some of the capabilities provided, techniques used, challenges faced, lessons learned and how this approach impacted the user. Keywords: Agile Engineering, Agile Development, Information Management, Enterprise Systems Engineering 1. INTRODUCTION In previous papers, we introduced a methodology where MITRE is helping the DoD and U.S. Special Operations Command (USSOCOM) to adopt the use of agile development techniques for enterprise assessment and transition of capabilities in support of the Global War on Terror (GWOT) [3,4]. To provide context, the GWOT represents a new form of warfare that includes many complex dimensions and challenges [2,10]. Fundamentally, capabilities needed include global and tactical situational awareness, multiagency/nation collaboration & coordination; enhanced joint operations among diverse forces; information sharing and management; and enhanced sensor technologies. To quickly provide such capabilities to users supporting GWOT missions, we are using agile engineering techniques that rely on using a distributed innovation lab (ilab) environment in conjunction with a series of warfighter workshops focused on themes and challenge problems identified by users. These workshops leverage FFRDC, Government, Industry and Academia resources and net-centric distributed capabilities to facilitate agile capability assessment and transition opportunities in endto-end fashion. They are designed to provide hands-on warfighter immersion into emerging processes, concepts and capabilities combined with facilitated discussions to develop and/or refine Concept of Operations (CONOPS) and Tactics, Techniques and Procedures (TTPs). In this paper we present preliminary results and lessons learned from this multi-year case study. We highlight some of the capabilities assessed; the techniques used to quickly integrate them in a loosely-coupled fashion; the challenges we faced; and how the user was impacted. 2. AGILE ENGINEERING THROUGH WARFIGHTER WORKSHOPS There has been much work published in the literature on agile development and engineering [5,6,7,8,9]. We view agile engineering as a framework for understanding software engineering projects that embraces and promotes evolutionary change throughout the entire life-cycle of the project. In agile engineering, the up-front effort in analysis, design and documentation is de-emphasized. The focus is on frequent user engagement, small spirals, hands-on exposure to technologies, and capabilities & processes. An agile engineering approach provides an
overall environment designed to foster innovation, observe and capture emergent behavior & requirements, and identify potential failures and successes early in low risk/low cost ways. requirements and capabilities. Warfighter Workshops leverage the agile engineering process to immerse warfighters in end-to-end problem sets in a light weight fashion. Immersion in problem sets allows workshop participants to understand issues and major challenges that need to be addressed to solve mission-critical problems. Facilitated discussions during a Hot Wash are used to develop and/or refine CONOPS and Tactics, Techniques and Procedures (TTPs). The assessment of technologies used in the workshop provides faster feedback to technology developers on areas for additional work in an effort to accelerate transition of innovative solutions to operational use. 3. WARFIGHTER WORKSHOP CASE STUDY RESULTS In this section, we highlight preliminary results compiled from three warfighter workshops conducted over the past two years to investigate challenge problems related to GWOT. Figure 1. Traditional vs Agile Engineering Traditional Systems Engineering as depicted in Figure 1 is often very sequential in nature, and does not inherently facilitate strong user-engineer collaboration or risk taking. This formal process requires up front documentation and agreement on requirements, specifications, and contractual issues, typically resulting in longer and more costly acquisition cycles. As an alternative, agile engineering is very parallel with emphasis on userengineer interaction and feedback throughout. Formal documentation is replaced with user stories, test cases, working systems and light documentation. Benefits of the agile approach include: More accurate project estimates earlier in a project; Progress measured by tested software Increased toolset proficiency by agile teams through immersion Rapid software production by seasoned teams Reduces time-to-operational implementation Reduces risk of failure Proves out architecture sooner Validates business benefits Makes execution more successful, reducing risk Drives high IT value The first workshop focused on command center activities such as situational awareness and multi-intelligence (multi-int) correlation and exploitation. The results of this workshop were used to influence the research and development community. The second workshop investigated netted sensor capabilities and limitations. This workshop highlighted capabilities of current and emerging sensors, processing/visualization systems, and web-enabled services [1]. The results of the workshop were used to develop a preliminary enterprise sensor architecture. The third workshop built upon the previous workshops and was conducted as a distributed exercise involving users from the Department of Defense and Department of Homeland Security. It was conducted simultaneously at two different facilities: a Government Test Lab located in St Petersburg, FL and one of MITRE s Innovation Laboratories located in Hampton, VA. Over 120 attendees participated across the two locations. The main objective for this workshop was to examine capabilities and technologies for Multi-INT Exploitation, Collaboration, and Information Management across complex organizational boundaries. In our approach, we introduced the warfighter workshop as a tool for performing agile assessment of concepts,
IM JOC Chief COP Mgr Intel JOC Government Lab St. Pete, FL Internet Services Chief COP Mgr Operator AOC IM Operator Operator NSWTU Government Lab St. Pete, FL Various solutions associated with the technical challenges noted above were assessed by the users in the form of game playing. We used several open source gaming engines to stimulate real systems being used by the participants and to monitor how they used the technology to solve those challenges. Further assessment was done by the observers attending the workshop, from the perspective of operational utility, technical readiness/maturity and whether a technology shows promise as a candidate for transition. Predator UAV MITRE Lab Hampton, VA Figure 2. Workshop Scenario Figure 2 depicts the high-level scenario and operational connectivity for the third workshop. The scenario depicts the challenges associated with the implementation of a GWOT mission across complex organizational boundaries. The specific challenges investigated during the workshop included the following: Collaboration- GWOT missions will require coordination across complex organizational boundaries, including unplanned users. Technologies for achieving a dynamic, cross-domain collaboration, command and control, and dynamic sensor tasking across multiple mission partners were Tailored Common Operational Picture- Warfighters require a means to filter out extraneous/poor quality information to achieve a user defined COP. Technologies for discovering and displaying only relevant, context-sensitive sensor information were Unplanned Users- Diverse information types/formats published by the complex set of mission partners and unplanned/non-standard information sources need to be accommodated. Technologies for managing information to achieve data interoperability for both planned and unplanned mission partners were Multi-INT Fusion- Intelligence Centers need to fuse multi-int information from a diverse set of mission partners to support analysis/exploitation. Technologies for analyzing and exploiting multi-int information were Sensor Planning- Automated tools are needed for sensor emplacements planning and for analyzing the performance of multi-tier netted sensor networks. Tools for maximizing the performance of tactical sensor networks in the maritime environment were For all three workshops, user objectives were established based on a need to gain a better understanding of emerging GWOT capabilities in the above areas; these helped to construct the realistic scenarios to stress test key concepts. The results of these workshops were used to directly influence the acquisition products necessary to guide future investments. In addition, several of the capabilities were determined ready for field testing and were promoted to the next level of assessment by the users. 4. LESSONS LEARNED In this section, we highlight some lessons learned associated with the workshops conducted. The following general technical observations are made: Connectivity across labs and among multiple workshop participants is the weakest and most critical link. Interoperability and integration is made easier by using standardized loose coupler interfaces, mitigating system many interface issues [11]. Virtualization is an effective technique to quickly emulate complex network and system architectures Gaming engines are effective light-weight ways to stimulate users in realistic-enough settings Constant hands-on user engagement is a positive influence on the spiral capability assessment process In addition, we have learned a lot about the agile engineering process used and continue to mature it. Early in the process the team needs to keep asking themselves the following questions -- especially when evaluating what technology to use, how to use it, and how to assess its effectiveness: What are the warfighter goals in the workshop? How do these goals match the capabilities of a candidate systems/technologies being assessed for inclusion in future workshop?
How does this technology solve/address warfighter goals? What are the metrics to assess how well it works? Other lessons learned included: Early assessment of the candidate system maturity In order to better evaluate the maturity of technology under consideration early in the process, some questions to ask include: Are the interfaces to external systems welldocumented with examples and sample data? What is level of commitment of principal investigator and development team for including a system in the workshop? Is the developer for system available for on-site installation and training? Immature systems (e.g. early prototypes, alpha software, etc.) require more support, consequently the workshop should include only a few of these since they require much care and handling. Early familiarization with candidate systems The workshop engineering/integration and scenario teams need a demonstration of software early on. There were a few systems that were never demonstrated to the scenario team until late in the workshop process. The result was that they were left out of the scenario and/or misrepresented in their actual capabilities. In other cases the technology was force-fitted into the scenario to make it work. Early testing with representative data sets It is important to perform testing of the target systems with the types of data that is realistic and appropriate for the scenario context. Roles and responsibilities Having a "Master of Scenario" (i.e. orchestrator) was very helpful in keeping people informed of the schedule and who was doing what. During both dry-runs and live runs of the scenario, the MS was very helpful in keeping the scenario on track and making sure the "players" knew what was going on in the scenario -- prompting, if necessary, for what to do next. Scenario and script The scenario just needs a minimal degree of realism, but must also be developed in spiral fashion to allow the team to evaluate how well the scenario presented opportunities to showcase the value of the various technologies being demonstrated. The scenario can then be modified to add fidelity where it was deficient in showcasing technologies. It can also be trimmed down in other sections that introduced added complexity without any obvious benefits in demonstrating how the technologies being investigated show value. The realism of the scenario is not what is most important; it s how the workshop forces participants to experience real-world challenge problems and discover what attributes the technologies being demonstrated need to have in order to meet operational requirements. The concept of a teaser script to reinforce the value of the technologies being presented for VIP attendees can dramatically affect the perceived impact of the workshop. 5. CONCLUSIONS Our work continues to show that agile development best practices can scale to the enterprise-level, and could be used as an alternative to traditional systems engineering and acquisition approaches when applied to large systemof-system environments. The Warfighter Workshops conducted over the past few years have made significant impacts on technology development and transition. One of the major benefits of Warfighter Workshops is that they bring the operational and research communities closer together. Facilitated discussions between warfighters and technology developers educate operators on the art of the possible and inform technology developers on requirements and enhancements needed to accelerate transition. Prior workshops have facilitated the transition of technologies such as video data extraction and tagging, Multi-INT correlation and video dissemination service capabilities. Warfighter Workshops have exposed the Government to emerging technologies that are candidates for addressing operational challenges. Leading edge capabilities for enhancing situational awareness in the context of a common operational picture are now better understood. Workshop participants have also been exposed to other emerging technological advances in functional areas such as information management, cross-domain collaboration, multi-int sensor fusion and sensor web enablement for sensor network management and exploitation. The agile engineering process used to conduct Warfighter Workshops provides a number of benefits for capability assessment and transition. It maintains customer focus throughout entire capability lifecycle. It also focuses on holistic/enterprise view which enables program managers and requirement developers to keep their eye on bigger
picture. Utilizing the agile engineering process minimizes the gap between mission need and fielded capability. In summary, Warfighter Workshops provide a method to quickly evaluate capability alternatives and risks, and they help to identify, refine and verify CONOPS, requirements and priorities. 6. REFERENCES [1] M. Botts, G. Percivall, C. Reed and J. Davidson, OGC Sensor Web Enablement Framework: Overview and High Level Architecture, Open Geospatial Consortium Inc. White Paper Version 2.0, 2006. [2] R. Cherinka and R. Miller, Lessons Learned in Using Web Services for Enterprise Integration of Complex Systems in the Department of Defense, International Conference on Systematics, Cybernetics and Informatics (SCI), July 2004. [3] R. Cherinka, R. Miller, D. Edwards, J. Mathews, D. Pitcher, W. Sears, and T. Semanchik, Agile Capability Development, Assessment and Transition in Support of the Global War on Terror (GWOT), MILCOM 2007, October 2007. Department of Defense Hourglass,2007 International Conference on Enterprise Information Systems and Web Technologies (EISWT-07), July 2007. [5] J. Doughty, Introduction to Agile Development Presentation, MITRE Agile Development and Usability Engineering TEM, The MITRE Corporation, Sep 2007. [6] P. Hagan, Agile Methods Overview and Implications for Architecture Presentation, MITRE Agile Development and Usability Engineering TEM, The MITRE Corporation, Sep 2007. [7] J. Niski, Agile Software Development: Not for Lightweights, Technical Report, The Burton Group, Version 1.0, May 2007. [8] The Agile Manifesto, http://agilemanifesto.org. [9]Agile Engineering, Wikipedia, http://wikipedia.com. [10] D. Edwards et. Al., United States Special Operations Command C4ISR Enterprise IT Framework Report, The MITRE Corporation, March 2006. [11] D. Kaye, Loosely Coupled, The Missing Pieces of Web Services, RDS Press, 2003. [4] R. Cherinka and R. Miller, Engineering a Complex Information Enterprise: A Case Study Architecting the