SAFETY CASES: ARGUING THE SAFETY OF AUTONOMOUS SYSTEMS SIMON BURTON DAGSTUHL, 17.02.2017
The need for safety cases Interaction and Security is becoming more than what happens when things break functional insufficiencies Security is becoming more than what happens when things are broken into exploit of functional insufficiencies Traditional Functional View of Causes of Hazards Random HW Hazard Systematic Systems View Hazard Current safety standards do not address these issues well, we need to argue safety from first principles Random HW Systematic Functional Insufficiencies Physical Collaboration Deliberate Manipulation Weakness exploits 2 CR/AEX CE-SRA Burton 17/02/2017
The need for safety cases Example: Questions to be answered by a safety case for machine learning. Are the functional and performance requirements on the ML function well defined? Is the operational profile well defined and understood? Is the ML technique the most effective algorithm for implementing the function? Is the residual risk associated with the ML function technique appropriate for the context? Have the training data led to a sufficiently accurate approximation of the target function? Has the probability of adversarial examples been minimised? Does the performance of the function during operational match its overall requirements? Does the distribution profile during operation match the assumptions made during training? What statistical performance statements can be extrapolated from system validation? Does the ML function operate within the integrated system context as expected? How effective are the tests at reasoning about the performance of the ML function? Is the execution platform (HW/SW) robust against systematic and random HW failures? Is sufficient rigor applied within the development process to ensure that processes and guidelines are followed when specifying, training and validating the ML functions? 3 CR/AEX CE-SRA Burton 17/02/2017
What is a safety case? (ISO 26262) Definition in ISO 26262, Part 1 (Vocabulary) Argument that the safety requirements for an item are complete and satisfied by evidence compiled from work products of the safety activities during development Definition in ISO 26262, Part 10 (Guidelines) The purpose of the safety case is to provide a clear, comprehensive and defensible argument, supported by evidence, that an item is free from unreasonable risk when operated in the intended context 4 CR/AEX CE-SRA Burton 17/02/2017
What is a safety case? (ISO 15026) Assurance Case (ISO 15026 Part 1 Vocabulary): reasoned, auditable artefact created that supports the contention that its top level claim (or set of claims), is satisfied, including systematic argumentation and its underlying evidence and explicit assumptions that support the claim(s) An assurance case consist of Claims Evidence Argumentation Justification Assumptions Assurance: Grounds for justified confidence that a claim has been or will be achieved. 5
Process vs. concept oriented cases Argument based on development approach (Process) Argument based on safety concept (Product) ISO xxxx plan case work products (evidence) case goals concept Follows structure of safety plan, Often documented as a structured list of work products Follows structure of safety concept, Includes rationale for technical decisions made in the project A combination of both approaches is effective 6
Goal Structuring Notation (GSN) Graphical notation that represents the elements of an assurance case and the relationships between them Shows how goals (claims) can be broken down into sub-goals until they can be supported by direct reference to available evidence Documents strategies adopted as well as context information, including assumptions and justifications Can be structured hierarchically and modularly Principle aim is to improve the comprehension of the assurance case thus enabling rigorous review and analysis Context <Context Identifier> <Reference to contextual information or statement> Strategy <If all sub goals are true then is sufficient to establish the claim that higher level goal is true> <Solution Identifier> <Reference to an evidence item or items> Goal <Presents a claim forming part of the argument> <Strategy identifier> <Describes the nature of inference between a goal and ist supporting goals> Evidence <Undeveloped sub goal> Assumption <Assumption Identifier> <Intentionally unsubstantiated statement> <Justification Identifier> <Statement of rationale> Justification A Sub-goal J 7
What makes a good safety case? The safety case should argue how the available evidence can be interpreted to argue: The sufficiency of the safety concept to maintain the safety goals The robustness, completeness and integrity of the development approach <Context Identifier> <Reference to contextual information or statement> <Presents a claim forming part of the argument> <Strategy identifier> <Describes the nature of inference between a goal and ist supporting goals> <Assumption Identifier> <Intentionally unsubstantiated statement> <Justification Identifier> <Statement of rationale> A J All assumptions related to the safety concept, have been identified, analysed and validated Diverse measures and sources of supporting evidence increase the strength of the argument Ideally a combination of structure (e.g. using GSN), explanatory text and referenced artefacts <If all sub goals are true then is sufficient to establish the claim that higher level goal is true> <Solution Identifier> <Reference to an evidence item or items> <Undeveloped sub goal> 8
Example (V.Incomplete) System Context Description of the classes of objects the robot can detect Requirements E.g. Derived from binding standards defining safeguards System Envelope E.g. physical limits of the system (e.g. camera resolution, braking distance, ) Passive friendly safety The robot will come to a stop (with a sufficiently high probability) if it detects a human in ist trajectory path with sufficient time to allow the human to also come to a stop Functional Requirements...Move from A to B Kinematic constraints Assumptions on environment Movement profiles of each class of moving object A Predict trajectory and come to a safe stop Argument over the technical approach taken E.g. Surfaces, Lighting, number of moving objects, other obstacles,... A Detection Accuracy Objects are detected and classified according to object movement profiles Includes argument for perception accuracy, training of machine learning algorithms, etc. Mathemati cal Proof Algorithms are principally sound Analysis of algorithms have demonstrated their fundamental ability to converge Algorithms are implemented correctly Verfication of the implemented algorithms Algorithms are valid in the systems context System level validation o the algorithms...rationale for properties being demonstrated... Published studies Peer Review Model checking Static analysis of code Software tests System tests Safe hardware All dangerous hardware failures occur with an acceptably low probability and detected at run time Includes technical hardware safety concept (e.g. redundancy), references to FMEA, FTAs, fault injection tests etc. 9 CR/AEX CE-SRA Burton 17/02/2017
Towards run-time dynamic safety cases Statically configured, closed loop systems Model-based system specification Upgradeability and connectivity Component-based safety contracts Open Context, full automation patterns for fail operational systems Cloud computing of safety functions Ad-hoc collaboration patterns for adaptive safety concepts Model-Based Systems and Security Engineering Challenges Formal verification of SW and system properties Formal validation of requirements models Formal verification of machine learning Formal verification of run-time contracts Model-based safety analysis Component-fault trees Jenkins-time safety analyses Automated safety concept optimisation for fail operational systems Run-time evaluation of safety properties (safety contracts) Assurance cases Modular assurance cases Assurance cases for SOTIF * Collaborative assurance cases * SOTIF: of the intended function 10
Thank you! Any Questions? 11