2 August 2017 Prof Jeff Craver So you are Conducting a Technology Readiness Assessment? What to Know

Transcription

1 2 August 2017 Prof Jeff Craver So you are Conducting a Technology Readiness Assessment? What to Know

2 Agenda items Challenges Statutory Requirement MDAPs TMRR Phase DRFPRDP Independent Review Panel (IRP) Technology Readiness Assessment and TRLS Critical Technologies (CT) Selection and Assessment Demonstrated in a Relevant Environment Approval by the Component Acquisition Executive (CAE). Reviewed by ASD(R&E) and DASD (DT&E) 2

3 GAO: Reasons Programs Fail (or experience significant cost and schedule impacts 1. Poor requirements management 2. Immature Technologies 3

4 Higher total MDAP RDT&E funding growth since original baselines WHY? Performance of the Defense Acquisition System: 2016 Annual Report U.S. Department of Defense 4

5 Program Development with Immature Technologies Of the remaining 36 current programs only 4 reported that all their critical technologies were matured to best practice standards when they began development.11 Another 11 programs reported having all critical technologies nearing maturity prior to system development. The remaining 21 programs reported either having one or more immature technologies at the start of development at this critical point in the acquisition process. GAO Technology Readiness Assessment Guide 5

6 Example: Gerald R. Ford Class Nuclear Aircraft Carrier (CVN 78) maintaining design stability depends on currently immature technologies fitting within the space, weight, cooling, and power reservations allotted them within the ship. Construction to date has been impeded by critical technology system delays, material shortages, and engineering challenges. Costs have grown by over 22 percent, GAO Technology Readiness Assessment Guide March 2014 DEFENSE ACQUISITIONS Assessments of Selected Weapon Programs 6

7 Technology Maturation as a Cost Driver Product development and associated technologies TRL at program initiation Cost growth Schedule delay Comanche helicopter Engine, rotor, FLIR, helmet mounted display, avionics Ford Jaguar automobile Adaptive cruise control, voice activated controls 5, 5, 3, 3, percent 120 percent 8, 8 None None GAO Technology Readiness Assessment Guide 7

8 DODI Jan 7, 2015 as amended Feb 2, 2017 TRA shall be conducted: On Critical Technologies (CT) demonstrated in a relevant environment In the TMRR Phase to reduce technical risks associated with CTs* As a statutory requirement on all MDAPs prior to the DRFPRDP By an independent panel appointed by the PM and reviewed by ASD(R&E) and DASD (DT&E) * Prior to MS C if program is entering lifecycle in EMD 8

9 Everywhere TRA appears in DODI Jan 7, 2015 as amended Feb 2, 2017 DODI January 7, 2015 Incorporating Change 2, Effective February 2, PROCEDURES d. Acquisition Process Decision Points and Phase Content. (4) TMRR Phase (b) Phase Description 3. There are a number of ways to structure this phase which should be tailored to reduce the specific risks associated with the product being acquired. Technology Readiness Levels, described in the Technology Readiness Assessment (TRA) Guidance (Reference (e) (f) ), should be used to benchmark technology risk during this phase; however, these indices are rough benchmarks, and not conclusive about the degree of risk mitigation needed prior to development. Deeper analysis of the actual risks associated with the preferred design and any recommended risk mitigation must be conducted and provided to the MDA. REFERENCES (e) (f) Assistant Secretary of Defense for Research and Engineering Guide, Technology Readiness Assessment (TRA) Guidance, April 2011, as amended ( ENCLOSURE 1 ACQUISITION PROGRAM CATEGORIES AND COMPLIANCE REQUIREMENTS Table 2. Milestone and Phase Information Requirements DODI Jan Change 7, , 02/02/2017 as amended Feb 2, 2017

10 MDAPs and non-mdaps Decision Authority MDAP ACAT I program or as determined by the MDA ACAT 1D (DAE) DAE or as delegated ACAT 1C (Component) Head of the DoD Component or CAE ACAT IAM DAE ACAT IAC - Head of the DoD Component or CAE MDAs for non-acat I programs should consider requiring TRAs for those programs when technological risk is present. ACAT II CAE ACAT III Designated by the CAE ACAT IV - Navy and Marine Corps only DODI enclosure 1 TRAs for the ASD(R&E) are not required for Major Automated Information System (MAIS) programs, non-mdaps, or MDAP Milestone C (MS C) decisions, except for MDAPs entering the acquisition system at MS C. 10

11 TMRR Phase PURPOSE: To reduce technology, engineering, integration, and life cycle cost risk to the point that a decision to contract for EMD can be made with confidence in successful program execution for development, production, and sustainment. DODI Jan 7, 2015 as amended Feb 2,

12 Development RFP Release Decision Development RFP Release Decision is extremely important Program will either successfully lead to a fielded capability or fail, based on the soundness of the capability requirements, the affordability of the program, and the executability of the acquisition strategy. Authorizes the release of RFPs for EMD and often for Low-Rate Initial Production (LRIP) options (For MDAPs and major systems, the MDA will determine the preliminary LRIP quantity at this decision point). Last point at which significant changes (i.e. requirements) can be made without a major disruption. Timing of the PDR relative to Development RFP Release Decision is at Component discretion. Source: Figure 7, DoDI Change 2 of Feb 2, 2017 DODI Jan 7, 2015 as amended Feb 2, 2017

13 Phased Acquisition Cycle with Decision Points Technology Maturity Assessment (Knowledge Building TRAs) For Programs that do not have a MS B GAO Technology Readiness Assessment Guide 13

14 Independent Review panel (IRP) Panel Membership Dependent on Technologies Involved Independent and therefore, not a member of the program IPT or any specialized interest group Recognized expert with proven experience (PM concurrence) Senior SME that has authority to speak for Dept/Command concerning respective technology No personal gain for program success or failure Has current and appropriate clearance level Will proactively work to de-conflict schedule as program plans change Maintain consistent IRP throughout duration of TRA Copeland NAVAIR TRA_TMA Process Training Brief Apr

15 DODI , Table 2 Technology Readiness Assessment Technology Readiness Assessment (TRA): A systematic metrics-based process that assesses the maturity of, and the risk associated with, critical technologies. Preliminary assessment required for Major Defense Acquisition Programs (MDAPs) in support of Development RFP Release Decision Point. Technology Readiness Levels (TRLs): Use: Adopted by DoD as a method of estimating technology maturity during the acquisition process. The TRL scale is measured from 1 to 9 with 1 being the least mature technology and 9 being the most mature. Caution: The TRL scale is a management tool. It does not mean that the product technology readiness level is accurate or even that it has gone through any official measurement process. For more information see DAU Continuous Learning Module CLE 021, Technology Readiness Assessments

16 What is a TRA? A TRA is an evaluation of the maturity of critical elements of a product s technologies, often called critical technologies. It is a normal outgrowth of the system engineering process and relies on data generated during the course of technology or system development. The TRA frequently uses a maturity scale technology readiness levels (TRLs) that are ordered according to the characteristics of the demonstration or testing environment under which a given technology was tested at defined points in time. Technology Readiness Levels (TRLs) can serve as a helpful knowledge-based standard and shorthand for evaluating technology maturity, but they must be supplemented with expert professional judgment. (Technology Readiness Assessment (TRA) Guidance April 2011) GAO Technology Readiness Assessment Guide 16

17 Why Are TRA s Important? High quality evidence-based TRAs provide managers and governance bodies with important information for making technical and resource allocation decisions on whether a technology or system is sufficiently mature to move past a decision point to the next acquisition phase, needs additional work, or should be discontinued or reconsidered in favor of more promising technology. The TRA results in the form of a TRA report also serve as input to other program management decisions to estimate cost, schedule, and risk. Importantly, TRAs provide a common language and framework or reference point to facilitate dialogue supported by well-defined metrics and methods across organizational disciplines, departments, and business functions. GAO Technology Readiness Assessment Guide Page 17

18 Technology Readiness Assessments (TRA) vs Technology Maturity Assessment (TMA) TRA is a formal requirement for MS B conducted by an independent review team outside the influence of the PM specific to Critical Technologies TMA is a periodic assessment of the ongoing maturity of the technology in preparation for an upcoming TRA and as knowledge points across the program management and risk management spectrum conducted at the discretion of and by the PM New and Novel? A technology is considered critical if: It proposes a significant risk to the success of the program especially relating to KPPs and KSAs Copeland NAVAIR TRA_TMA Process Training Brief Apr

19 GAO Technology Readiness Assessment Guide 2016 The Guide has two purposes: (1) describe generally accepted best practices for conducting effective evaluations of technology developed for systems or acquisition programs, and (2) provide program managers, technology developers, and governance bodies with the tools they need to more effectively mature technology, determine its readiness, and manage and mitigate risk. 19

20 20

21 TRA Completed 21

22 Four Characteristics of a High Quality TRA Credible - Assessment design, execution, and reporting activity reflects understanding of requirements, critical technologies, relevant or operational environments; assessment team has right knowledge and expertise Objective - Assessment is based on objective, relevant and trustworthy data, analysis, and information; free from internal and external organizational bias or influence Reliable - Uses disciplined processes that facilitate repeatability, consistency, and regularity Useful Stakeholders understand information; it has sufficient detail and is timely and can be acted upon GAO Technology Readiness Assessment Guide 22

23 Six Steps to Develop a High Quality TRA 1. Design TRA Strategy 2. Define Purpose, Develop Plan, and Assemble Team 3. Select Critical Technologies 4. Evaluate Critical Technologies 5. Prepare and Submit the TRA Report 6. Use TRA Results and Develop a Technology Maturation Plan MDAPs - The plan for conducting a TRA is provided to the ASD(R&E) by the PM upon approval by the Component Acquisition Executive (CAE). DODI GAO Technology Readiness Assessment Guide 23

24 Best Practice - Selecting Critical Technologies Step 3: Select Critical Technologies Critical technologies should be rigorously and objectively identified and documented to ensure the evaluation is objective and reliable, and the information is useful. There are 4 steps that should help organizations ensure that the process for selecting critical technologies is reliable GAO Technology Readiness Assessment Guide 24

25 BEST PRACTICE CHECKLIST: SELECTING CRITICAL TECHNOLOGIES A rigorous, objective, reliable, and documented approach, based on the WBS or other key program documents was used to initially identify critical technology candidates. The intended operational environment was considered, including potential adverse interactions with systems which the technology being developed must interface. A relevant environment was derived for each critical technology from those aspects of the operational environment that is determined to be a risk for the successful operation of that technology. Critical technologies were initially selected following a reliable process that is disciplined and repeatable with defined criteria using increasingly platform- or program-specific questions and requirements. GAO Technology Readiness Assessment Guide Critical technologies were defined at a level that is testable, which could include the software needed to demonstrate their functionality. The assessment team documented the reasons why technologies were selected as critical, including reasons why other technologies were not selected. The number of critical technologies chosen for assessment was not arbitrary but was based on solid analysis using the WBS, process flows, or other technical documentation. When significant program changes occurred, critical technologies were reassessed possibly causing some to be added or removed from the list of critical technologies. Subject matter experts with appropriate and diverse knowledge selected and reviewed the critical technologies. 25

26 Critical technology (CT) The adjective critical has several applications and context when referencing technology Mission Critical Technology List Critical Program Information Critical for Mission Success Critical Safety Items CT The term Critical Technology is used to uniquely identify immature technology that introduce significant risks to program A technology is considered critical if: (1) the system being acquired depends on this technology element to meet operational requirements (within cost and schedule limits), and (2) if the technology element or its application is either new or novel or in an area that poses major technological risk during detailed design or demonstration no longer in definition Copeland NAVAIR TRA_TMA Process Training Brief Apr

27 Criteria for Determining CT Impact to program success Has the technology been modified? Has the technology been repackaged such that a new and more stressful relevant environment is realized? Is the technology expected to operate in an environment and/or achieve a performance expectation beyond it s original design intention or demonstrated capability? Example set of criteria list to be developed and agreed to by MDA, PM, Independent Review Panel Copeland NAVAIR TRA_TMA Process Training Brief Apr

28 Best Practice - Selecting CTs Challenges in Selecting Critical Technologies Program officials sometimes disregard critical technologies when they have longstanding history, knowledge, or familiarity with them. This is problematic when these technology elements are reapplied to a different program or operational environment, particularly when being used in a novel way. The process to collect evidence for identifying critical technologies can be straightforward, the determination for what constitutes a critical technology is highly subjective, requiring knowledge, experience, and due professional care. For example, judgements need to be made about what a technology is, what makes a technology critical, and at what level. Correctly identifying and selecting critical technologies can prevent wasting valuable resources funds, capital acquisitions, and schedule later in the acquisition program. GAO Technology Readiness Assessment Guide 28

29 Best Practice - Evaluating Critical Technologies There are 4 steps that can help organizations ensure that an evaluation is objective and reliable by applying a disciplined and repeatable process. These steps can be tailored to accommodate organizational structures, processes, and policies. GAO Technology Readiness Assessment Guide 29

30 CT Assessment Criteria (Readiness) Technologies have been demonstrated in a relevant environment and whether risk has been reduced or can be reduced to an acceptable level for inclusion in an EMD program Is that it? DAU S&T CoP Decision Point Quicklook Assessment Instruments Tailored for a TRA by IRP/PM If someone told you this technology was TRL 6, would that be enough to convince you that the risk was adequately mitigated? I hope not. Defense AT&L: September October The Trouble with TRLs Frank Kendall tices%20lessons%20learned%20and%20tools.aspx 30

31 Limitations of TRA and TRLs GAO Technology Readiness Assessment Guide 31

32 Demonstrated in a Relevant Environment Adequate demonstration in a relevant environment (TRL 6) is one benchmark that is evaluated, but it is not the only consideration, nor necessarily dispositive Demonstration: An element of verification that involves the actual operation of an item to provide evidence that the required functions were accomplished under specific scenarios. The items may be instrumented and performance monitored. (Mil-STD 961E) Relevant Environment Defined by who? Customer, User, Requirements community, D,OT&E? Before the assessment process begins, the SME team must ensure a sufficient understanding of the requirements, identified capabilities, system and software architectures, CONOPS, and/or the concept of employment to define the relevant environments. The SME team must also ensure that its understanding of design details is sufficient to evaluate how the technologies will function and interface. (DOD Technology Readiness Assessment (TRA) Guidance April 2011) Required by WSARA (2009) 32

33 Reviewed by ASD(R&E) and DASD (DT&E) The plan for conducting a TRA is provided to the ASD(R&E) (FY 18 TBD) by the PM upon approval by the Component Acquisition Executive (CAE). A TRA is required by Department of Defense Instruction (DoDI) for MDAPs at MS B (or at a subsequent Milestone if there is no MS B). It is also conducted whenever otherwise required by the MDA. TRAs may have to be performed on all the competitors proposals in a competitive source selection. DOD Technology Readiness Assessment (TRA) Guidance April

34 Conclusions Required for MDAPs (ACAT I or as designated) recommended for ACAT II III TRA should be a tailored process using best practices TMAs should be conducted regularly in prep for TRA Relevant environments defined by customer for TRL 6 to be valid Independent review team members could come from almost anywhere not under the influence of the PM 34

35 References DODI Jan 7, 2015 as amended Feb 2, 2017 DOD Technology Readiness Assessment (TRA) Guidance April 2011 GAO Best Practices Technology Readiness Assessment Guide Copeland NAVAIR TRA_TMA Process Training Brief Apr 2016 Mil-std 961E Performance of the Defense Acquisition System: 2016 Annual Report U.S. Department of Defense Defense Acquisitions Assessments of Selected Weapon Programs - March 2014 GAO Points of Contact Mr. John Ortiz Project Manager, IT ortizj@gao.gov GAO Products 35