System Maturity Assessment Roundtable

Transcription

1 System Maturity Assessment Roundtable March 12, 2009 Ronald Reagan Building - Washington, DC SUMMARY On March 12, 2009 a Roundtable was held at the, Washington, DC Campus with the purpose of providing system designers and developers, program and project managers, and researchers a platform to discuss and disseminate emerging knowledge in systems maturity indices (beyond Technology Readiness Level). The objective was to create a community of practitioners and researchers focused on new knowledge in system maturity indices and assessment. For the first half of the day, presentations were made from invited speakers on the emerging challenges and potential solutions in systems maturity indices and assessment. For the second half of the day, breakout groups were asked to address these four questions with respect to the future of system maturity assessment: 1. What are the real questions? 2. What do we know? 3. What do we need to know? 4. What could we do to learn that? This report is a summary of the results of this Roundtable and the presentations made by the invited speakers.

2 ATTENDEES Tom Alameda U.S. Army ARDEC Jim Bilbro JB Consulting Lisbeth Concho Robert Cuellar Department of Defense Ken Evans Institute for Defense Analyses Russell Fink JHU Applied Physics Laboratory Jerry Fisher National Reconnaissance Office Eric Forbes Northrop Grumman Company Peter Gentile Northrop Grumman Company Robert Henderickson U.S. Army ARDEC Shaun Hutton JHU Applied Physics Lab David Jakubek Office of the Secretary of Defense Chris James JHU - Applied Physics Lab DH Kim Office of the Secretary of Defense Roger Liu U.S. Army ARDEC Romulo Magnaye Mike McGrath ANSER Leonard Ogborn Science Applications International Corp. Thomas Owen U.S. Navy PMS 420 Has Patel Infologic, Inc. Jose Ramirez-Marquez Brian Sauser Gene Scampone U.S. Navy NAVSEA Chadd Sibert U.S. Army ODASA-CE Weiping Tan Brenda Wetzel JHU - Applied Physics Lab Joseph Uzdizinski Lockheed Martin Corporation 1

3 INTRODUCTION The growing need to assess the readiness or maturity of a complex system under development using a metric beyond the Technology Readiness Level (TRL) was formally presented during the 2006 Conference on Systems Engineering Research in Los Angeles, CA 1. It was then that a new metric Integration Readiness Level (IRL) was recognized as a necessary complement to the already established TRL in order to have a complete view of a system. Using both metrics, a System Readiness Level (SRL) was then conceptualized for calculating and mapping against a development lifecycle as an aid for decision-making. Almost three years later, the growing number of researchers and practitioners interested in this topic has grown exponentially. To begin to address these emerging developments, the Systems Development & Maturity Laboratory (SD&ML) under the sponsorship of a grant from the Naval Postgraduate School s Acquisition Research Program and support from the Applied Systems Thinking Institute, Northrop Grumman, and the U.S. Army Armament Research, Development, and Engineering Center held a Roundtable on March 12, 2009 at the Stevens Institute of Technology Washington, DC Campus with the objective of creating a community of practitioners and researchers focused on discovering new knowledge in system maturity indices and assessment, exchanging ideas, and attempting to chart a future direction for this field of research. For the first half of the day, presentations were made from invited speakers on the emerging challenges and potential solutions in systems maturity indices and assessment. For the second half of the day, breakout groups were asked to address these four questions with respect to the future of system maturity assessment: 1. What are the real questions? 2. What do we know? 3. What do we need to know? 4. What could we do to learn that? This report is a summary of the presentations made by the invited speakers and the results of the Roundtable breakout groups. 1 Sauser, B.J., D. Verma, J. Ramirez-Marquez, and R. Gove. (2006). From TRL to SRL: The Concept of Systems Readiness Levels. Conference on Systems Engineering Research. April 7-8, Los Angeles, CA 2

4 PRESENTATIONS OF CURRENT ACTIVITIES 2 Presentations and Presenters Systems Earned Readiness Management Brian Sauser, Assistant Professor, SD&ML Director, Beyond TRL - Using AD2 to Identify and Quantify Risk in Infusing Technology James W. Bilbro, JB Consulting International Dynamic Multipoint Optimization Application to Corporate Portfolio Management Robert Cuellar, Senior System Engineer, DoD - Information Assurance Research Crossing the Valley of Death: The Concept of Critical Research Elements Has Patel, President, Infologic, Inc. Implementation of a Systems Maturity Monitoring Methodology on the LCS Mission Modules Program Eric Forbes, Systems Engineer, Northrop Grumman Corporation Readiness Assessment with the Analytic Hierarchy Process (pdf) Russell Fink, Johns Hopkins University Applied Physics Laboratory Summary The need to develop a systematic method for evaluating systems under development becomes obvious when it is acknowledged that systems do not just consist of critical technology components but also of integration elements. Modern project management tools have been very successful and widely accepted for monitoring and evaluating progress in the development of individual technologies and simple systems. TRL, a descriptive metric for technology development, has also been increasingly used since its early introduction in NASA in the late 1980s. The logical step in the development of a systems monitoring methodology would have been the combination of both approaches and, indeed, some organizations may have actually done so albeit without publicizing their cases. Nonetheless, such a combination would still probably be inadequate since the integration elements continue to be neglected. With these in mind, Sauser presented the concept of a System Readiness Level (SRL) that was first introduced in 2006 and has been continually refined through a series of publications. Its basic idea is to combine TRL with a newly developed Integration Readiness Level (IRL) such that SRL becomes a metric with a system-wide perspective. SRL itself is not the result of observing directly the performance characteristics of the complete system. Rather, it is the result of determining the performance characteristics of the individual critical technology elements and their integration links, assigning the corresponding TRL and IRL values 2 For copies of the presentations made, see and follow the link for SMA Roundtable. 3

5 respectively and combining those values mathematically to calculate SRL. This is then plotted against a pre-specified development lifecycle in order to gage performance at any given point in time. A perceived shortcoming of SRL is that it can carry over the problems associated with TRL and IRL, especially its subjectivity. To address this, the measurement of TRL and IRL themselves must be the focus. That is, methodologies to make their measurements more robust should be pursued instead of trying to dismiss SRL itself as being too subjective. In addition to providing a system-wide view, SRL is best used as a bottom-up tool in order to identify maturity deficiencies that can cause top-level symptoms. More significantly, SRL can be applied in complex optimization algorithms in order to identify optimal development plans against which actual progress can be tracked and evaluated. It can also be used to inform the evaluation of alternative designs/architectures, substitute technologies, trade-offs among capabilities, cost and schedule and so on. With regard to monitoring and evaluating progress, SRL can be combined with modified project management principles in order to formulate a system-wide technique called System Earned Readiness Management (SERM). The application of SRL to a real system under development was reported by Forbes, Michaud and Gentile. In addition to calculating SRL for a system as a whole (the Littoral Combat Ship Mission Modules), they also showed how SRL can be used to determine the readiness of the program to perform the various operations that it is capable of doing. These operations or capabilities can be represented by strings of components (technologies and integrations) which can be evaluated independent of the entire ship itself. Understanding the true status of the system on an operational string level allows for the opportunity to field initial capability earlier and then add to it as other strings mature. Based on their actual experience, they conclude that the use of SRL, in addition to estimating system maturity, also provides valuable information by forcing stakeholders to view integration elements and the system as a whole. It can generate a better understanding of the program itself. Bilbro introduced the concept of Advancement Degree of Difficulty (AD 2 ) which can be used to define the levels of risk associated with the experience base of technology developers when there is a very high degree of uncertainty. It is a predictive description of what is required to move a system, subsystem or component from one TRL to another in terms of risk, impact on schedule and on costs. The issues associated with the development of an element are assessed using 5 specific areas: design/analysis, manufacturing, software development, testing and operations. Eventually, these will be expanded to include integration and other ilities (e.g., capability, scalability, etc.). Answers to these assessments are then rolled-up and filtered as a function of risk level. It was suggested that the AD 2 assessment is intended to provide focused and improved accuracy in the development of program/project cost, schedule and risk. It can provide the basis for the development of a Technology Development Plan. Cuellar advocated the need to focus on the application of SRL methodologies to an enterprise portfolio management process throughout a systems engineering-based lifecycle. In essence, 4

6 he identified 3 elements that are necessary to formulate a Lifecycle Portfolio Management methodology financial and maturity metrics, optimization algorithms and decision-making guidelines. The disconnect between Science and Technology (S&T) and Acquisition programs was highlighted by Patel. The development of basic technologies is not mapped on to a development path that defines which applications are going to be served by the research programs. To address this concern, he suggested that S&T projects should be classified as Critical Research Elements (CREs) which can then be mapped into the CTEs that are under development. There can be a one-to-one or one-to-many relationship between CREs and CTEs. CREs still need to be defined and eventually, guidelines and procedures must be developed to link CREs and CTEs in the DoD environments. Fink, Hutton, James and Wetzel found that TRL is inadequate in assessing the risk of each component of a large government program to develop the next generation of information assurance capabilities. The main drawback is that insufficient details about the risks are discernable because as designed, TRL fuses different dimensions of readiness. Thus, it can not be used to rank the technologies in terms of multiple readiness criteria that they have determined to be required for decision making. These are: requirements, maturity, plugability, availability, certification/accreditation, feature scope and technical support. To address this, they suggested the application of Analytical Hierarchy Program (AHP). However, the government program that is under development has too many technologies which are too complex for one person to understand. These make the straightforward application of AHP difficult and cumbersome so a modified AHP methodology was presented which utilizes an Intensity Factor for each criterion ranging in value from 1 to 9, with the latter being the most strongly important level of intensity assigned to factors during the pair-wise comparisons. Initial results indicate that the approach works well and feels right to systems engineers and the program sponsors. It is also repeatable and flexible and amenable to the development of tools to support analysis. 5

7 BREAK OUT SESSIONS: FUTURE DIRECTIONS The presentations summarized above generated emergent thought, which was carried over to the breakout sessions in the afternoon. Four discussion groups with diverse participants were organized with the aim of assessing the current body of knowledge on measuring the maturity of complex systems under development, future challenges/needs, and how to meet these. The ideas generated can be classified into the following categories: Relevance of maturity measures How do they impact the functioning of a milestone authority? They must be applied to historical program data ( program cadavers ) to see how they could have impacted the implementation and results. Are they equally relevant to small and large, short- and long-term development timetables, system of systems? Are they equally relevant to the different hierarchy levels of decision making? Do they apply universally to acquisition management, systems engineering or program portfolio management? Multiple attributes of maturity measures Which attributes should be incorporated in TRL, IRL or SRL and which ones should be evaluated separately? When should Manufacturing Readiness Level (MRL) assessment be conducted separately from TRL? How can the service dimension (as distinct from the product itself) of technology and integration be represented in readiness levels? Accuracy of metrics Should there be independent organizations similar to Underwriters Laboratories (UL) or Moody s performing the readiness assessments? How can we account for the impact of the experience or expertise of the evaluators (look to other fields such as radiology for insights)? 6

8 Do more assessors necessarily lead to better results? Success/failure stories program cadavers Does TRL and IRL evaluation impact each other s measurements? Applications and uses of maturity measures Recognize and understand the variations in the way readiness measures are used in different organizations How is human judgment incorporated/allowed in the assessments? Others Reduce redundancies in the technology development programs of various government agencies through an integrated technology management roadmap. Who should own the roadmap? 7