Making DARPA META Goals Come True: How do we Revolutionize Verification and Validation for Complex Systems?

Transcription

1 Making DARPA META Goals Come True: How do we Revolutionize Verification and Validation for Complex Systems? Dr. Kirstie L. Bellman Computers and Software Division; Aerospace Integration Science Center (AISC) The Aerospace Corporation June 17, 2010 S5 2010, WP AFB The Aerospace Corporation 2010

2 Overview The purpose of talk is to stimulate a critical discussion on what we need to revolutionize V&V and to design provably correct systems The state of testing/ensuring correct behavior in large complex systems The DARPA META program goals Asking the tough questions For more information, Bellman@aero.org June 17, 2010

3 Guaranteeing Correct Behavior in Large Complex Systems: Observations (1) Planned testing often cut off as funds dry up in schedule-driven developments Little negative or stress testing Often only well-known domain or operator anomalies addressed happy path testing testing lead, contractor Fault Management Systems are developed late and inadequately Not handling multiple errors, emergent phenomena, and until recently, any software-driven errors Unit tests not adequately updated as code base changes during development Late integration testing Elaborate (wonderful) models and formal methods not integrated into analysis, test, and evaluation process For more information, June 17, 2010

4 Guaranteeing Correct Behavior in Large Complex Systems: Observations (2) Regression testing inappropriately applied Inadequate guidelines for how much of the system can be regression tested Done in lieu of adequate re-testing of code base/siqt testing FMECA still largely excludes SW-driven SPFs; SW SPFs largely unrecognized Many leading corporations appear to have insufficient skill base for good testers/modelers Required architectural artifacts used as documentation not to analyze, engineer, formulate tests Inappropriate parallelization of tasks creates architectures and design artifacts after code/system developed For more information, June 17, 2010

5 And in the future These and other problems lead to enormous costs and schedule delays And things are only going to get more difficult as systems continue to grow in size and complexity For example, the new adaptive capabilities we are trying to introduce into all DoD platforms The following slides are provided courtesy of Paul Eremenko, PM, from his Industry Day Brief for the DARPA META program (Dec. 22, 2009) The views expressed are the author s/presenter s own and do not represent those of DARPA, the DoD, or the US Government. For more information, Bellman@aero.org June 17, 2010

6 Aerospace and defense systems have experienced significant growth in development time and cost with increasing complexity Next-Gen Platform Historical Cost Growth (not adjusted for inflation) Aerospace Systems (1960 present) Automobiles (1960 present) Integrated Circuits (1970 present) 8-12%/yr 4%/yr ~0%/yr New IC design flow MIL-STD-499A Aerospace Vehicle 1990s ~5X Reduction in Development Effort New automotive design flow Automobile 1960s Aerospace Vehicle 1960s Automobile 1990s Pentium META Goal Integrated Circuit Next Gen Integrated Circuit 1960s Intel 8088 Intel 286 Intel 386 Automobile Next Gen Xeon Note (*): Not a great metric. But that s what we have today. META will come up with better metrics.

7 Indicia and consequences of complexity growth Trends in Complexity-Related Metrics for Various Systems 1960s 1990s Next Gen Aircraft 1.5 W/kg* 25 W/kg* 60 W/kg* Spacecraft 120 kg 3,000 kg 10,000 kg Surface Ships 1 kw/ton* 1.5 kw/ton* 5 kw/ton* Power Transfer States Comm. Paths Aerospace Software 10 ksloc 2,000 ksloc 10,000 ksloc Development Time 4 years 12 years? Cost Growth PPBS (1962) MIL-STD-499 (1969) 8-12%/yr Automobiles 10 3 parts 10 4 parts 10 5 parts Automotive Software 0 1,000 ksloc 100,000 ksloc Development Time 60 months 36 months 18 months Cost Escalation ~0%/yr System complexity has grown substantially Exponential growth in many metrics Acquisition process is vintage 1962 Systems engineering process dates to 1969 No advances to manage complexity growth No hardware-software trades or co-design Complexity-related phenomena worsen High cost of change: changes propagate quickly & widely in tightly integrated systems Emergent behaviors: unanticipated interactions, multi-mode failure cascades Fragility: unanticipated vulnerabilities Consequences for DoD systems are dire Lack of adaptability to change & new needs Longer development timelines Significant cost escalation (especially I&T) Cost and schedule variance Integrated Circuits 10 3 transist transist transist. Development Time 36 months 36 months 36 months Cost Escalation ~0%/yr I&T = integration and test; SLOC = source lines of code Note (*): Wattages refer to electrical power generation capability, which also drives thermal dissipation and EMI.

8 A major cause of these phenomena is the industry s failure to update a 1960s-vintage systems engineering, integration, and test process SWaP used as a proxy metric for cost, and dis-incentivizes abstraction in design MIL-STD-499A (1969) Systems Engineering Process As Employed Today System decomposed based on arbitrary cleavage lines... Re-Design Conventional V&V techniques do not scale to highly complex or adaptable systems with large or infinite numbers of possible states/configurations Cost Optimization System Functional Specification System Layout Verification & Validation SWaP Optimization SWaP Optimization Power Data & Control Thermal Mgmt Subsystem Design Component Design Subsystem Testing Component Testing Resulting architectures are fragile point designs... and detailed design occurs within these functional stovepipes Unmodeled and undesired interactions lead to emergent behaviors during integration SWaP = Size, Weight, and Power V&V = Verification & Validation Desirable interactions (data, power, forces & torques) Undesirable interactions (thermal, vibrations, EMI)

9 The goal of the META program is to dramatically shorten the design, integration, and verification timeline for aerospace systems Demonstrate 5X compression in development time for a complex aerospace system Develop practical, observable metric of complexity for cyber-physical systems to enable cyber-vs-physical implementation trades; improve parametrization of cost & schedule Develop a quantitative metric of adaptability associated with a given system architecture to support trade-offs between adaptability, complexity, performance, cost, and other attributes Develop a structured design flow employing hierarchical abstraction and model-based composition of electromechanical and software components Develop a component and manufacturing model library for a given systems domain through extensive characterization of desirable and spurious interactions, dynamics, and properties Develop a verification flow that generates probabilistic certificates of correctness for the entire cyber-physical system based on stochastic formal methods, scaling linearly with size Apply the above framework and toolset to design, manufacture, integrate, and verify a complex aerospace system 5X faster than with a conventional design/build/test approach We think that 5X is doable, but certainly DARPA-hard Software VLSI Automotive

10 Complexity Some general observations on management of complexity Today s Systems Managing complexity really means reducing design complexity for a given capability By analogy to Kolmogorov complexity, any cyber-physical system has a minimum-complexity design that delivers an equivalent capability A complexity-based design process will try to find the theoretical minimum A good complexity-based design process can also lead to a reduction in organizational complexity of the design organization There are several ways of proving success for a new complexity management approach Measure some agreed-upon complexity metric for a system designed by old and new way Build a system that is more complex than we know we can build today Demonstrate cost reduction during design, integration, & test of a complex system Demonstrate schedule compression during design, integration, & test of a complex system... the last is simplest and most controlled, although META will use a complexity metric also Complexity reduction, schedule compression, etc. do not come for free Additional constraints and overhead are imposed on the design The META-designed aircraft, for instance, is likely to be less weight-optimized Capability Similarly, integrated circuits, automobiles, model-generated software, etc. sacrifice optimality in exchange for faster time to market, higher reliability, and lower NRE costs Theoretical Minimum

11 Comparison of some common complexity metrics between electromechanical and software systems Software Systems SLOC (Source Lines of Code) Function Points Cyclomatic Complexity Kolmogorov Complexity Length of Correctness Certificate Electromechanical Systems SWaP (Size, Weight, and Power) Information Content Length of Correctness # components (desired + undesired interactions) (Length of System Blueprint) Certificate Pro s: Easily observable Linearly additive Con s: No info on microstructure Poor cost correlation Pro s: Relatively easy to observe Provides incentives for abstraction Limited empirical data shows good cost correlation Clear traceability to design heuristics Con s: Dynamical coupling? Heterogeneiety of components? Definition of component? Pro s: Theoretical appeal and wide range of applicability Limited empirical data shows excellent cost correlation Con s: Challenging to observe Not easily additive Unclear what heuristics would evolve to minimize this metric Pro s: Rigorously correct metric Potentially possible to estimate Con s: Difficult to compute precisely for all but the simplest systems Source: Flowe et al. (2009); UTRC (2009). Source: Suh (1990); Hoult & Muter (1993); Collopy & Eames (2009).

12 Hierarchical abstraction is pervasive in many of the most complex systems, but not in aerospace/defense platforms Information Networks Integrated Circuits Biological Systems Exhibit consistent behavior in spite of tremendous heterogeneiety of components Internet Application Layer (Data) Designed by manual lay-out, analogous to that employed in aerospace & defense design flows today Intel 4004 Intel Pentium 4 Functional Level (Architecture) Designed using model-based hierarchical layer composition methods and supporting tools Encode the design of a highly complex system in ~25,000 genes by re-use of conserved processes & functions at lower levels of abstraction Anteroposterior & Dorsoventral Axes, Compartments (Body Plans) Transport Layer (Sessions) Register Transfer Level (Blocks) Signaling, Matrix, Junctions, Epithelia (Multicellularity) Network Layer (Packets) Logic Level (Logic Gates) Nucleus, Organelles, Sexual Reproduction (Eukaryotic Functions) Data Link Layer (Frames) Circuit Level (Transistors) Metabolic Pathways (Prokaryotic Functions) Physical Layer (Bits) Physical Level Proteins Source: Weste et al. (2000) Source: Kirschner & Gerhart (2005)

13 A model-based, hierarchical composition approach can lead to lowercomplexity designs and better, more adaptable architectures

14 Testing to Exhaustion With a model-based representation of the entire cyber-physical system, resulting designs can be proven correct-by-construction Stochastic Formal Methods

15 The META program builds the enabling tools and models, and culminates in the rapid development of a complex DoD platform Phase 1a: Design Flow Development Complexity metric for cyber-physical systems Observable and objective Enables hardware-software trades Develop cost and schedule parametrics Determine scope of applicability Power and thermal management Avionics, controls, payload, and data bus Aerostructure and structural components Engines & weapons as discrete components Development of model-based design flow Optimal deployment of hierarchical abstraction Optimization with respect to complexity metric Selection/development of modeling language Development of composition rule-set Design of supporting tool Stochastic model-based verification techniques Develop stochastic formal methods Sensitivity analysis to model accuracy Non-proprietary design toolset Phase 1b: Toolset Implementation Implementation of supporting software toolset Requirements for component characterization Hypothetical end-to-end system development Employing notional component library Validate result with respect to program metrics Phase 2: Component & Manufacturing Model Library Development Specification of demo system domain Tactical aircraft, rotorcraft, etc. Component selection for library inclusion Component characterization Component testing as needed to develop model to specified level of accuracy Static component attributes (size, weight, etc.) Component interfaces (physical, data, power) Spurious interactions (vibration, thermal, EMI) Component dynamics (dynamic response) Quantitfication of model uncertainty Hypothetical end-to-end system development Employing real component library Validate result with respect to program metrics Development of demo system specifications Timely for Phase 3 performer bid Standard modeling language Model data set for a given class of systems Supplier consortium (à la AUTOSAR) Phase 3: Rapid Development Demo Design, manufacturing, integration, verification Confirm through standard developmental tests Short flight test program (~3 months) Tactical UAV Attack helicopter/uav Next gen long range strike Next gen MRAP Next gen UGV/tank Document resultant program metrics Detailed monitoring of level of effort by task Resultant schedule and cost metrics Demonstrated system reliability vs. model Quantify performance differences Capture and quantify maintainability, adaptability, and other architectural attributes Platform transition to service partner Systems engineering (SE) curriculum infusion DDR&E/SE promulgation of SE standard

16 Some Tough Questions for Our Discussion Are formal methods/enabling modeling technologies mature enough to support META goals? Why is META likely now? What do we need to support a revolution? Proof by construction what are its limits? Will any of these methods scale up for adaptive systems? What are the hard questions I m not asking? For more information, Bellman@aero.org June 17, 2010