Assurance Cases: New Directions & New Opportunities* John C. Knight University of Virginia February, 2008 *Funded in part by: the National Science Foundation & NASA A summary of several research topics from our group at UVA Topics covered: Assurance argument fallacies Accident investigation Assurance based development Assurance based communication New directions in certification More details available from papers 2 University of Virginia Things I Like Safety-Critical Systems 3 University of Virginia 4 University of Virginia The Safety Case comprehensive and defensible argument that a system is acceptably safe to operate in a particular context. [T. Kelly] The safety case communicates: High-level safety objectives Evidence that objectives have been met Argument linking evidence to objectives Assumptions, justifications, and other context Does it always communicate: Accurately? Completely? 5 University of Virginia Assurance Case Has To Be Right Can we construct arguments that are free of fallacies? Can we check arguments? What is the effect of a fallacy? What should certifiers do with assurance cases? Let s look at some published assurance cases (actually safety cases) 6 University of Virginia
Safety Case Survey Eurocontrol RVSM Reduced Vertical Separation Minimum Examined three industry safety cases: Eurocontrol RVSM Pre-Implementation SC Eurocontrol Whole Airspace ATM SC Opalinus Clay Waste Repository SC Fallacy Using the Wrong Reasons Drawing the Wrong Conclusion Red Herring A 5 3 B 5 Total 6 3 Two reviewers noted frequency and nature of fallacies observed in each safety case. Fallacious Use of Language Hasty Inductive Generalization Omission of Key Evidence 2 4 2 4 4 Total 5 8 29 7 University of Virginia 8 University of Virginia Fallacious Argument Example G2.3 Red What Herring Is Wrong Here? G2.3.. FC RVSM & Transition Training specified. G2.3..2 FC Aircraft Contingency training specified. S2.3.. St2.3. Argue that there is sufficient direct evidence of flight crew training design validity. G2.3..3 Flight planning training specified. Flight crew training design complies with safety requirements. G2.3..4 Hazards and risks controlled and mitigated. S2.3..4 Arguing From Ignorance? 8.2.8.4 Absence of outstanding issues with the potential to compromise safety The current safety analysis, despite a wide range of of assessment cases that were derived in in a careful and methodical way, has not identified any outstanding issues with the potential to to compromise safety. Opalinus Clay Clay Safety Case PISC 5.4.3 & 5.4.4 PISC 5.4.6 9 University of Virginia 0 University of Virginia Assurance Case Fallacy Taxonomy So go ask the philosophers Circular Reasoning Circular Argument Circular Definition Diversionary Arguments Irrelevant Premise Verbose Argument Fallacious Appeals Appeal to Common Practice Appeal to Improper/Anonymous Authority Appeal to Money Appeal to Novelty Association Fallacy Genetic Fallacy Mathematical Fallacies Faith in Probability Gambler s Fallacy Insufficient Sample Size Pseudo-Precision Unrepresentative Sample Unsupported Assertions Arguing from Ignorance Unjustified Comparison Unjustified Distinction Anecdotal Arguments Correlation Implies Causation Damning the Alternatives Destroying the Exception Destroying the Rule False Dichotomy Omission of Key Evidence Omission of Key Evidence Fallacious Composition Fallacious Division Ignoring Available Counter-Evidence Oversimplification Linguistic Fallacies Ambiguity Equivocation Suppressed Quantification Vacuous Explanation Vagueness University of Virginia 2 University of Virginia
C0 System hazard analysis C04 A320 flight envelope G03 Control logic enforces flight envelope constraints on pilot. C03 A320 FCS operating procedures ST0 Argument by addressing all credible hazards G02 Hazard of aircraft exiting flight envelope sufficiently mitigated. G05 Direct control law provides pilot override mechanism. S0 Control logic design G0 Airbus A320 FCS is safe to operate. ST02 Argument for compliance with applicable safety regulations G04 Control logic will not command hazardous maneuver. S02 Model checking analysis C02 DO-78B standard Verification Approach Developer Fallacy Taxonomy Legend G: Goal (property to be shown) C: Context (inclusion indicated by ) ST: Strategy (type of argument being made to support goal) S: Solution (factual basis for the argument) : remains to be supported Certifier Management 3 University of Virginia 4 University of Virginia Suppose Argument Is Wrong Despite verification of assurance case, it might still contain fallacies Effect might be to lead to failure: Accident during operation System not safe despite developers thinking it was If fallacy or fallacies remain, assurance case is map for finding it Base accident investigation on assurance case 5 University of Virginia Enhanced Assurance Case Lifecycle Failure Known Fallacies Assurance Case Pandora Fallacy Taxonomy New Fallacies Pandora Accident Investigation Process Lessons 6 University of Virginia Assurance Based Development Primary goal: Focus on the assurance case, not the software Approach: Define top-level goal as to solve the problem Develop the assurance case completely This implies creation of the evidence Part of the evidence is the software development artifacts Not taking this approach leaves assurance in doubt Traditional development is going after the wrong goal 7 University of Virginia 8 University of Virginia
Assurance Based Development Requirements Software Development Required Evidence Supplied Evidence Goals Assurance Case Development Software Assurance Case 9 University of Virginia 20 University of Virginia Communications Graph Bringing Things Together Domain Experts Regulators Systems Engineers Software Engineers How crucial is this communication? General Public Content? Notations? Validation? Verification? 2 University of Virginia Make communication during development explicit in the safety/assurance case Establishes necessary communications quality as a goal Develop assurance/safety argument that communications goal will be met Incorporate appropriate techniques: Formal languages, CLEAR, etc. 22 University of Virginia Requirements Argument G. The system is fit for use Better Requirements Argument G. The system is fit for use This is a very different way to look at things G.. Any system that is fit for use G.2. Any system that meets the specification G.3. The system meets the specification G.. Any system that is fit for use G.2. Any system that meets the specification G.3. The system meets the specification G... The technique used to capture requirements produces complete and correct requirements G..2. The technique used to review requirements catches incorrect or missing requirements Legend Assurance claim Is supported by G... The domain experts' understanding of the problem to be solved is complete and correct G..2. The domain experts' understanding of the problem to be solved is adequately conveyed by the requirements document to its readers 23 University of Virginia 24 University of Virginia
Determining Airworthiness Of Unmanned Air Systems Project being conducted for Navy Air Command Unmanned Air Systems present new challenges for Navy Air Approach based on safety cases Significant overlap with challenge faced by FDA Challenge: Aircraft come from variety of manufacturers Manufacturers do not develop comprehensive evidence Need to certify because of aircraft s immediate value 25 University of Virginia 26 University of Virginia UAS Airworthiness Challenge Is the FDA similar? Strength of a Safety Case Compelling Argument Navy Airworthiness Certification UAS Manufacturing Construct based on available rather than desired evidence Safety Case Or Or Reveal Necessary Evidence or or Argument Defines Valid Operational Contexts 27 University of Virginia 28 University of Virginia Multiple Safety Cases For Single Aircraft Conclusion Safety Cases Patrol Monitoring Iraq Iraq Marine Surveillance East East Coast Crowd Monitoring Manhattan Assurance of dependability is crucial We need to know that the system will operate properly Presently we hope it is achieved by: Ad hoc methods and experience Prescribed, rigid processes In Assurance Based Development: Assurance case is the focus, not the software Development decisions influenced by impact on assurance Allows a precise selection of development techniques 29 University of Virginia 30 University of Virginia
Contact E-mail address: knight@cs.virginia.edu For more information see: http://www.cs.virginia.edu/knight/ http://dependability.cs.virginia.edu/ 3 University of Virginia