Institute for Defense Analyses 4850 Mark Center Drive Alexandria, Virginia 22311-1882 Modeling & Simulation Roadmap for JSTO-CBD IS CAPO Dr. Don A. Lloyd Dr. Jeffrey H. Grotte Mr. Douglas P. Schultz CBIS // Decision Support January 10, 2007
Outline IT as a CBDP commodity The Roadmap problem The Roadmap solution Advantages & disadvantages FY08 program build Impacts Beyond the Roadmap CBIS 2007 2
IT as a CBDP commodity What are we talking about? Computer processors, servers and platforms Communications protocols and infrastructure Development tools & environments Interfaces (e.g. JCID component of JWARN) Methodologies Of these, only interfaces and methodologies are likely candidates for CBDP basic and applied S&T. We will focus on methodologies They account for more than 90% of the M&S/B S&T program They are the basis for Modeling & Simulation development They are algorithms and heuristics, alone or in combinations They pose unique research challenges for user requirements CBIS 2007 3
M&S differs from other CBDP commodities Not just for tools deployed to the warfighter, but also required to support internal CBDP functions Analysis Training Plans and concept development Programmatics Test & Evaluation CBDP M&S draws from a broad pool of basic research Numerical mathematics and information theory, but also physics, chemistry, materials science, atmospheric science Methods are not specific to CBRN Fundamental research product is documentation of: - Experiments, observations, theorems, phenomenologies - Data and their concise generalizations, i.e. small m models - Results are not specific to Modeling & Simulation Their research products are usually not software CBIS 2007 4
M&S differs from other CBDP commodities End-user context is more complicated M&S does not exist in a vacuum In CBDP, M&S is part of a decision support system, for some user-base, to address some set of problems Real world CBRN data used to drive M&S is dirty Utility of M&S is based on decision outcomes and risks, not technical performance measures Additional requirements of software VV&A (I)V&V focuses on technical merits of software solution Accreditation must also consider use-case and risk Chain of evidence begins with the basic research documentation Closest analog for accreditation is military utility of M&S tool These differences suggest that M&S should be managed differently from CBDP materiel The research opportunities and objectives may not be obvious CBIS 2007 5
The Roadmap in a nutshell What are the CAPO responsibilities to CBDP? Satisfy known capability gaps in IS basic research Stimulate new capabilities developed from IS basic research CAPO perceptions BAA is inefficient, too many responses, most wide of the mark Difficult to forecast value of any particular project Unsure whether right things are delivered to Program Symptoms we observed No objective criteria for evaluating research candidates Mixing of 6.1, 6.2 and 6.3 activities under 6.2 Lack of transparency to proposal writers and reviewers Roadmap strategy based upon Review of CBDP, DMSO and other DoD guidance Informal interviews (JPM-IS, JPEO, JRO, JCD-X, T&E & others) Participation in BAA review process for FY06 & FY07 CBIS 2007 6
Roadmap objectives Make CBRN information systems research and methodologies available for transition when mature. Improve alignment of JSTO M&S investments with CBDP needs - Formalize process for obtaining best advice at right times - Describe and measure the value of CBRN information - Develop objective criteria for evaluating candidate solutions - Customize approaches to tech push and requirements pull - Accommodate M&S requirements for internal Program functions Assert new measures for the health of the research plan - What is the gold standard for basic research? - Revisit periodically to measure progress and realign efforts Acknowledge other stakeholder responsibilities Work within the Implementation Plan for CBDP Focus on research, not software development Be consistent with or improve upon existing JSTO business model CBIS 2007 7
RDT&E 6.1 & 6.2 activities Result of 6.1 and 6.2 research is not usually a software product. Real currency of research is the scientific documentation, report or article JSTO M&S 6.3 funding limited to accumulating data to support transition Budget Activity 1, Basic Research. systematic study directed toward greater knowledge or understanding of the fundamental aspects of phenomena and of observable facts without specific applications towards processes or products in mind. Examples: Heuristics, information theory, threat agent science Products: Peer reviewed paper or equivalent Budget Activity 2, Applied Research. systematic study to understand the means to meet a recognized and specific need translate promising basic research into solutions short of system development with a view toward developing and evaluating the feasibility and practicality of proposed solutions Examples: Error analysis, scalability and feasibility analyses of 6.1 research Products: Technical report or equivalent Some FY05/06 JSTO M&S efforts were categorized 6.2 but included 6.3 software development activities, which are a PM responsibility. CBIS 2007 8
Roadmap obstacles Problem definition Too little analysis to know what the technical objectives should be Decision problems are harder than they look No connection between tech performance and operational effectiveness Confusion between basic and developmental S&T M&S program management Too little analysis conducted to know whether M&S is required Need for M&S assumed, but often unsubstantiated Acquisition paradigm leaves Program requirements unsatisfied Competing authorities initiate M&S efforts Who pays, why and how? Confusion between data requirements and M&S These problems usually occur together, but the Roadmap can only address the first. CBIS 2007 9
The Roadmap solution Formalize the process for obtaining best advice prior to writing BAA Adopt IPT approach with mix of CBDP and outside participation Specialize strategies for Requirements pull and Technology push Specific objective measures up-front - For comparison of competing solutions - For greater transparency to proposal writers, and reviewers Leverage existing solutions - Not all required methodologies are unique to CBRN applications Emphasize peer-reviewed, journal quality report as the basic research product This is the gold standard of research quality Make this an obligation of new and continuing research projects Adds to collective CBDP and DoD knowledge base Provides some assurance that whether a success or failure, the lessons learned are not lost CBIS 2007 10
(Pre-BAA) IPT functions Requirements pull IPT functions Recognize whether requirements are adequately defined for tech base Specify the decision context that defines and supports the required capability Define metrics for value of M&S information in decision context Translate operational and analytic requirements into a quantitative specification Determine whether data supporting research are available or must be acquired Determine whether quantified requirements possible without further study Distinguish basic and applied research from customer-developer responsibilities Review published research for acceptable candidates Evaluate research products for satisfaction of requirements and metrics Technology push IPT functions Review research proposals from a broad range of disciplines Ask for subject matter reviews on concepts you are unfamiliar with Articulate a concept for using CBRN information Ask for and recognize applicability to CBRN info problems Identify practical research objectives Identify potential customers or recipients for new IS functionality in CBDP CBIS 2007 11
Roadmap advantages More efficient use of 6.1 and 6.2 research dollars Manages risk in the basic research plan JSTO cultivates the state-of-the-art in practices and knowledge. BAA review process tailored to benefit decision makers Customers derive benefits of scientific and operational expertise Customers obtain best possible solution for specific needs Expect possibly fewer replies to BAA, but of generally higher quality Tech base able to effectively respond to quantitative requirements Improve concepts for information tools and establish their utility Clearer research performance criteria Roadmap is flexible Make CBDP IS research process available for analytic, training and other unwritten requirements Open process further to new ideas or concepts that enhance or extend CBDP IS capabilities Push and pull procedures can run concurrent or not Roadmap performance can be measured with a gold standard CBIS 2007 12
Managing the IPTs will require Roadmap disadvantages More time More people Wider variety of expertise Commitments to meet regularly Coordination of S&T plan with DHS, DARPA More expensive than current approach CBIS 2007 13
FY08: A rebuilding year FY08 begins the transition to technology push Articulate a CBDP concept for using CBRN information Key questions to ask of any basic research opportunity What is the motivation for the subject as a research topic? What are the prevailing theories or phenomenological approaches? What experiments have been conducted, and how do they reconcile with theoretical work? What kinds of problems do experts think the subject matter could be applied to? What feasibility studies have been conducted? What successful applications of the research? What attempts have failed and why? Use what is learned in FY08 to select best CBDP opportunities in FY09 and out Asking for written subject reviews, not software solutions CBIS 2007 14
Roadmap impact on multiple communities Contractors/developers Easier to write proposals that go to your strengths Implementation contracts revert to Program or Tech. Demonstration Manager Universities Most viable basic research candidates should come from universities But, many programs not used to proposing for DTRA funding Service Labs/FFRDC Labs Source of military smarts for technology Likely recipient of an intermediate technology transition Manage application and early development as technology demonstration very important role CBD Program officials Best approach to managing risk in basic research plan you will ever get, easier to measure health of a diverse research plan Avoids over-commitment to novelty, balances well with incremental research plans CBIS 2007 15
M&S management observation M&S is a poor candidate for acquisition Requirements documents capture the wrong thing they describe the tool but not the process and consequence of using the tool Acquisition Program Manager inherits all of the overhead and management apparatus used to make boots and gloves, but has no flexibility to respond to internal Program requirements. Need a Configuration Control Board represented by all CBDP components and users to direct the PM. Example: JICM is a Program of Record, with evolving requirements, managed by a CCB. CBIS 2007 16