Defense Modeling & Simulation Verification, Validation & Accreditation Campaign Plan

Size: px

Start display at page:

Download "Defense Modeling & Simulation Verification, Validation & Accreditation Campaign Plan"

Homer Logan
5 years ago
Views:

Defense Modeling & Simulation Verification, Validation & Accreditation Campaign Plan John Diem, Associate Director (Services) OSD/AT&L Modeling & Simulation

1 Defense Modeling & Simulation Verification, Validation & Accreditation Campaign Plan John Diem, Associate Director (Services) OSD/AT&L Modeling & Simulation Coordination Office : January 24 27, Jan 2011 Page-1 DISTRIBUTION STATEMENT A -- Cleared for public release by OSR (case numbers 10-S-3029 & 10-S-3232 apply).

2 24-27 Jan 2011 Page-2 VV&A Campaign Plan Summary Established May 2010 to be broadly focused policy, standards, guides, tools, methodologies Driven largely by work led by Acquisition M&S Working Group On-going effort work in 2011 includes several efforts to gain feedback from users and sponsors Feedback will influence re-look of key policy and standards components in 2012, including: DOD ( DoD M&S VV&A ) MIL-STD-3022 ( Documentation of VV&A for M&S ) DOD VV&A Guidance ( The VV&A Recommended Practices Guide (RPG) )

3 Key VV&A Documents DoD VV&A Policy: DoDI DoD M&S VV&A DoD VV&A Guidance: The VV&A Recommended Practices Guide (RPG) MIL-STD-3022 Documentation of VV&A for M&S IEEE IEEE Recommended Practice for VV&A of a HLA Federation IEEE (R2002) IEEE Recommended Practice for Distributed Interactive Simulation (DIS) VV&A) Jan 2011 Page-3

4 24-27 Jan 2011 Page-4 DoD Instruction DoD M&S VV&A Implements policy, assigns responsibilities, prescribes procedures, and establishes common terminology Current version dated December 9, 2009 Modified to reflect updated format requirements as defined in DoDI (Oct 2007) Streamlined document that establishes responsibilities, procedures, information requirements, and key definitions Incorporated reference to MIL-STD 3022 Scheduled for review in 2014 work will begin in 2012 DoDI dated 9 December 2009 available at

ITEA 5 LVC Conference 24-27 Jan 2011 Page-5 DoD VV&A Recommended Practices Guide RPG promotes the effective application of VV&A methodology Defines philosophy, principles and methodologies Developed

5 ITEA 5 LVC Conference Jan 2011 Page-5 DoD VV&A Recommended Practices Guide RPG promotes the effective application of VV&A methodology Defines philosophy, principles and methodologies Developed by subject matter experts Built to address diverse audiences, situation and applications Web-based format intended to: Provide the foundation for multi-tiered, multidimensional information at various levels of depth and detail Be extensible to allow for the addition of information over time Allow the reader to build guidance tailored to a situation or application Available at vva.msco.mil

6 ITEA 6 LVC Conference Jan 2011 Page-6 MIL-STD : DoD-sponsored Tri-Service VV&A Templates Tiger Team developed templates for four core VV&A documents Accreditation Plan, V&V Plan V&V Report, Accreditation Report 2007: Processed through DoD M&S management system and Defense Standardization Program 2008: Approved as a DoD Standard Practice May be cited as a solicitation requirement and Data Item Descriptions (DIDs) included on Contract Data Requirements List (CDRL) DI-MSSM DoD M&S Accreditation Plan DI-MSSM DoD M&S V&V Plan DI-MSSM DoD M&S V&V Report DI-MSSM DoD M&S Accreditation Report 2009: DoD Instruction , DoD M&S VV&A, references MIL-STD-3022 Free: Available from the Acquisition Streamlining and Standardization Information System (ASSIST)

7 Information Shared across the Templates Accreditation Plan V&V Plan V&V Report Accreditation Report Executive Summary Executive Summary Executive Summary Executive Summary 1 Problem Statement 1 Problem Statement 1 Problem Statement 1 Problem Statement 2 M&S Requirements and Acceptability Criteria 2 M&S Requirements and Acceptability Criteria 2 M&S Requirements and Acceptability Criteria 2 M&S Requirements and Acceptability Criteria 3 M&S Assumptions, Capabilities, Limitations & Risks/Impacts 3 M&S Assumptions, Capabilities, Limitations & Risks/Impacts 3 M&S Assumptions, Capabilities, Limitations & Risks/Impacts 3 M&S Assumptions, Capabilities, Limitations & Risks/Impacts 4 Accreditation Methodology 4 V&V Methodology 4 V&V Task Analysis 4 Accreditation Assessment 5 Accreditation Issues 5 V&V Issues 5 V&V Recommendations 5 Accreditation Recommendations 6 Key Participants 6 Key Participants 6 Key Participants 6 Key Participants 7 Planned Accreditation Resources 7 Planned V&V Resources 7 Actual V&V Resources Expended 7 Actual Accreditation Resources Expended 8 V&V Lessons Learned 8 Accreditation Lessons Learned Suggested Appendices A M&S Description B M&S Requirements Traceability Matrix C Basis of Comparison D References E Acronyms F Glossary G Accreditation Programmatics H Distribution List Suggested Appendices A M&S Description B M&S Requirements Traceability Matrix C Basis of Comparison D References E Acronyms F Glossary G V&V Programmatics H Distribution List I Accreditation Plan Suggested Appendices A M&S Description B M&S Requirements Traceability Matrix C Basis of Comparison D References E Acronyms F Glossary G V&V Programmatics H Distribution List I V&V Plan J Test Information Suggested Appendices A M&S Description B M&S Requirements Traceability Matrix C Basis of Comparison D References E Acronyms F Glossary G Accreditation Programmatics H Distribution List I Accreditation Plan J V&V Report Jan 2011 Page-7 Information that is common and shared across the documents appears in italicized font in the table above

mil Implements MIL-STD 3022 and Data Item

8 24-27 Jan 2011 Page-8 DoD VV&A Documentation Tool Implements MIL-STD 3022 and Data Item Descriptions DI-MSSM DI-MSSM DI-MSSM DI-MSSM-81753

24-27 Jan 2011 Page-9 DoD VV&A Documentation tool Producers register here or log in if already registered. There is online help for using the tool, as well as help for VV&A in general.

9 24-27 Jan 2011 Page-9 DoD VV&A Documentation tool Producers register here or log in if already registered. There is online help for using the tool, as well as help for VV&A in general. The DVDT automates the MIL-STD-3022 documentation templates. Register, log in, and create a project to download a set of XMLenabled document templates for local use. Compliant with DoD M&S Community of Interest Discovery Metadata Specification (MSC-DMS). A benefit of using the tool is easier management of VV&A information. The tool is designed to walk the Producer through VV&A planning, implementation, and reporting.

10 24-27 Jan 2011 Page-10 IEEE Std Defines processes and procedures to implement VV&A for federations being developed using the HLA Federation Development and Execution Process (FEDEP) (IEEE Std ) Focuses on unique aspects of VV&A of federations as a higherlevel framework into which VV&A practices can be integrated and tailored for specific uses. Provides implementation-level guidance to VV&A practitioners.

11 24-27 Jan 2011 Page-11 IEEE Std Guidelines established for the VV&A of distributed interactive simulation (DIS) exercises. Provides how-to procedures for planning and conducting DIS exercise VV&A Intended for use in conjunction with IEEE Recommended Practice for DIS- Exercise Management and Feedback (IEEE Std ) IEEE-SA approved reaffirmation of IEEE Std and IEEE Std in June 2010 Presents data flow and connectivity for all proposed verification and validation activities and provides rationale and justification for each step. Provides VV&A guidance to exercise users/sponsors and developers.

12 24-27 Jan 2011 Page-12 Where to Obtain VV&A Standards MIL-STD-3022, Documentation of Verification, Validation, and Accreditation for Models and Simulations, 28 Jan IEEE Std , IEEE Recommended Practice for Verification, Validation, and Accreditation of a Federation An Overlay to the High Level Architecture Federation Development and Execution Process, 20 December IEEE Std , IEEE Trial-Use Recommended Practice for Distributed Interactive Simulation - Verification, Validation, and Accreditation, 20 July 1998 (reaffirmed June 2010) IEEE Standards are free to SISO Members

13 24-27 Jan 2011 Page-13 Acquisition Community Led VV&A Efforts - key inputs to Campaign Plan VV&A Strategy

14 24-27 Jan 2011 Page-14 Acquisition Community-led VV&A High Level Task (HLT) Summaries V-AQ-2: Risk Based Methodology for Verification, Validation and Accreditation (VV&A) The degree of VV&A required is explicitly tied to both M&S use and the user risk incurred if the M&S does not provide accurate results. A methodology that tailors VV&A planning and implementation based on known risk factors will provide a framework in which VV&A implementation trade-offs can be made, information/fidelity requirements can be assessed, and a VV&A cost model can be developed. V-C-2: Improving VV&A Implementation Increase VV&A implementation and enhance M&S credibility by transforming VV&A practices from current subjective methods into objective examples or use cases. Explore emerging technologies, standards, and applicable methods that could be applied to reduce costs, schedule, and improve reuse.

24-27 Jan 2011 Page-15 Risk-Based VV&A (M&S PE Funded V-AQ-2) The purpose of this effort is to: Improve Department of Defense (DoD) Modeling and Simulation (M&S) Verification, Validation,

15 24-27 Jan 2011 Page-15 Risk-Based VV&A (M&S PE Funded V-AQ-2) The purpose of this effort is to: Improve Department of Defense (DoD) Modeling and Simulation (M&S) Verification, Validation, Accreditation (VV&A) by establishing an effective risk-based methodology for VV&A. Risk VV&A Resources Balancing the cost of knowing against the risk of assuming. Since the cost of verifying and validating simulations is often high, V&V investments should be weighed against the risk of making a bad decision because of unreliable M&S results.

24-27 Jan 2011 Page-16 Systems Engineering Research Center VV&A Topic (M&S PE Funded V-C-2) Proposed research topic provided SERC: Explore & promote a deeper understanding of emerging & current

16 24-27 Jan 2011 Page-16 Systems Engineering Research Center VV&A Topic (M&S PE Funded V-C-2) Proposed research topic provided SERC: Explore & promote a deeper understanding of emerging & current approaches, technologies, or practices underway within the academic community & gain a broader perspective on their potential applicability to the M&S community V&V needs The 19 SERC university partners generated six responses, with selection of two proposals for implementation lasting 6 to 12 months Two performers: Georgia Institute of Technology University of Alabama Huntsville

24-27 Jan 2011 Page-17 Georgia Institute of Technology VV&A Study (M&S PE Funded V-C-2) The purpose of this SERC effort : Demonstrate how to address many of the gaps identified in this VV&A

17 24-27 Jan 2011 Page-17 Georgia Institute of Technology VV&A Study (M&S PE Funded V-C-2) The purpose of this SERC effort : Demonstrate how to address many of the gaps identified in this VV&A Research Topic by applying SysML and MBE/MBSE technology Engage practitioners in the DoD ecosystem to prioritize needs and to test resulting concepts Apply concepts in one or more key DoD test beds

24-27 Jan 2011 Page-18 University of Alabama at Huntsville VV&A Study (M&S PE Funded V-C-2) The purpose of this SERC effort : Focus on verification & validation (V&V) of complex systems A survey &

18 24-27 Jan 2011 Page-18 University of Alabama at Huntsville VV&A Study (M&S PE Funded V-C-2) The purpose of this SERC effort : Focus on verification & validation (V&V) of complex systems A survey & evaluations of useful tools Leverages experience working the government customers in the Army Aviation Directorate, Software Engineering Directorate, Redstone Test Center, and PEO- Aviation. Will benefit from insights resulting from Systems Architecture Virtual Integration (SAVI), an international industry consortium involving Boeing, Airbus, Lockheed Martin, BAE, FAA, SEI, etc

19 24-27 Jan 2011 Page-19

Domain-relevant Best of Breed M&S Tool Suite VV&A NAVAIR Use Case (M&S PE Funded V-C-2) 24-27 Jan 2011 Page-20 Multiple communities within the Department of Defense are interested in

was to: Bring knowledge of existing VV&A best practices to funded implementation of credibility assessments of high-priority DoD M&S applications; Examine the practicality of leveraging

20 Domain-relevant Best of Breed M&S Tool Suite VV&A NAVAIR Use Case (M&S PE Funded V-C-2) Jan 2011 Page-20 Multiple communities within the Department of Defense are interested in accelerating schedules, improving credibility, & reducing costs of technical execution of M&S support to decisionmakers through tailored accreditation support The scope of this effort was to: Bring knowledge of existing VV&A best practices to funded implementation of credibility assessments of high-priority DoD M&S applications; Examine the practicality of leveraging Service-specific VV&A for domain-relevant Department-wide tool suite applications; and Identify Service-specific needs & estimate both cost-avoidance & incremental costs for plan to address.

24-27 Jan 2011 Page-21 VV&A Program Management & Integration (M&S PE Funded V-C-2) The VV&A Subcommittee of the Acquisition M&S Working Group (AMSWG) is a coalition of the willing.

21 24-27 Jan 2011 Page-21 VV&A Program Management & Integration (M&S PE Funded V-C-2) The VV&A Subcommittee of the Acquisition M&S Working Group (AMSWG) is a coalition of the willing. Help evolve the practice of VV&A by: Reviewing and commenting on VV&A studies Providing input to the review and update of VV&A policy and standards Supported by: User feedback Gray Hair Support to Key Activities

22 24-27 Jan 2011 Page-22 DoD M&S VV&A Campaign Plan Way Forward Expand leverage of DoD, government agency and Industry standards for M&S VV&A Mature existing products with documented and archived lessons learned Emerging concepts (e.g., Risk-based VV&A) will result in a standard/guideline downstream

23 24-27 Jan 2011 Page-23 Key M&S VV&A Contact Information DoD Modeling & Simulation Coordination Office John Diem, Associate Director (Services) (703) , Acquisition Modeling & Simulation Working Group Chair Phil Zimmerman, DDRE/SE/SA (703) ; Support Team Ralph Gibson, BAH Support to M&S CO (Services VV&A) (703) ; Crash Konwin, BAH Support to DDRE/SE/SA (703) ;

24 24-27 Jan 2011 Page-24 Back-up slides

25 24-27 Jan 2011 Page-25 Acquisition Community Led VV&A Efforts - key inputs to Campaign Plan FY: FY: V-AQ-2 V-C-2 Risk Based Accreditation Improving VV&A Implementation Johns Hopkins APL Improving VV&A Implementation Systems Engineering Research Center VV&A Shortfalls & Needs Complex Systems Modeling Pilot Implementations (SIMAF & NAVAIR) VV&A/Distributed Joint VV&A/Vehicle Vulnerability Practitioners Forum AMSWG VV&A Sub-Committee ITEA LVC Sub-committee Gray Hairs Study Support

26 24-27 Jan 2011 Page-26 Risk-Based VV&A (FY09/10 M&S PE Funded V-AQ-2) Since the cost of verifying and validating simulations is often high, V&V investments should be weighed against the risk of making a bad decision because of unreliable M&S results. The purpose of this work effort is to improve Department of Defense (DoD) Modeling and Simulation (M&S) Verification, Validation, Accreditation (VV&A) by establishing an effective risk-based methodology for VV&A.

24-27 Jan 2011 Page-27 JHU/APL VV&A Implementation: Defining & Addressing Gaps (M&S PE Funded V-C-2) The purpose of the effort is to: Support the identification of VV&A gap areas that inhibit

27 24-27 Jan 2011 Page-27 JHU/APL VV&A Implementation: Defining & Addressing Gaps (M&S PE Funded V-C-2) The purpose of the effort is to: Support the identification of VV&A gap areas that inhibit effective implementation of VV&A processes; Develop supplemental capability required to implement the Risk-based Accreditation (RBA) methodology; and Evolve technology, in the form of automated tools and metadata specifications, that increase VV&A implementation efficiencies.

28 24-27 Jan 2011 Page-28 Standard Intended Use & Tailoring Intended to support VV&A of DoD models, simulations, federations, and other types of distributed simulations by providing a common framework for documenting information produced during the VV&A processes. Specifies procedures on documenting information obtained through implementing the VV&A processes when the outputs will be used to supplement decisionmaking in the DoD. These types of information, at least some of the time, are obtained via contract from commercial firms. Tailoring guidance for contractual applications. Invitation for bids, request for proposals, and contractual statements should tailor Appendices A through D in accordance with the scope of the contractual effort.

29 ITEA 29 LVC Conference Jan 2011 Page-29 RPG Assessment Overview Purpose of the Recommended Practices Guide (RPG) Assessment Assess the viability of the web-based RPG Identify immediate problems that inhibit the RPG s effective use Identify potential improvements that could enhance the RPG s use Identify advanced technology features to increase the RPG s reach Assess Primary RPG characteristics Content: The information contained in the website Presentation and Structure: Organization and navigation Technology: Web-based features RPG Update Status Phase I updates completed (e.g., fixing hyperlinks, updating special topics documents) Elements of Phase II updates in process (e.g., developing new special topics, updating examples)

30 24-27 Jan 2011 Page-30 IEEE Top-Level Process View

31 24-27 Jan 2011 Page-31 IEEE Activities

32 24-27 Jan 2011 Page-32 DIS Exercise VV&A Process

33 24-27 Jan 2011 Page-33 DIS Exercise Life Cycle and VV&A

Systems Engineering Initiatives for Verification, Validation and Accreditation of DoD Models and Simulations

Systems Engineering Initiatives for Verification, Validation and Accreditation of DoD Models and Simulations Philomena M. Zimmerman ODDR&E/Systems Engineering 13 th Annual NDIA Systems Engineering Conference