System Safety. M12 Safety Cases and Arguments V1.0. Matthew Squair. 12 October 2015

Transcription

1 System Safety M12 Safety Cases and Arguments V1.0 Matthew Squair 12 October Matthew Squair M12 Safety Cases and Arguments V1.0

2 1 Introduction 2 Overview 3 Methodology 4 But do safety cases work? 5 Limitations, advantages and disadvantages 6 Conclusions 7 Further reading 2 Matthew Squair M12 Safety Cases and Arguments V1.0

3 Introduction 1 Introduction 2 Overview 3 Methodology 4 But do safety cases work? 5 Limitations, advantages and disadvantages 6 Conclusions 7 Further reading 3 Matthew Squair M12 Safety Cases and Arguments V1.0

4 Introduction Learning outcomes Understand what a safety case is Be able to critically review the content and argument of a safety case Be able to structure and prepare the content of a safety case Understand the strengths and weaknesses of the technique 4 Matthew Squair M12 Safety Cases and Arguments V1.0

5 Overview 1 Introduction 2 Overview 3 Methodology 4 But do safety cases work? 5 Limitations, advantages and disadvantages 6 Conclusions 7 Further reading 5 Matthew Squair M12 Safety Cases and Arguments V1.0

6 Overview Overview The Nimrod safety case process was fatally undermined by a general malaise: a widespread assumption... that the Nimrod was safe anyway (because it had successfully flow for 30 years) and the task of drawing up the safety case became essentially a paperwork and tickbox exercise. C. Haddon Cave, The Nimrod Review 6 Matthew Squair M12 Safety Cases and Arguments V1.0

7 Overview Overview Safety cases Originated in the British chemical industry CIMAH regulations Applied to oil industry after the Piper Alpha oil rig fire Applied to UK Rail after Clapham junction accident Have become part of the EU safety culture Embedded in various safety standards DEF-STAN DEF (AUST) 5679 Australian DMO SAMS Framework CMMI SAFE+ IEC Matthew Squair M12 Safety Cases and Arguments V1.0

8 Overview Overview Despite it s prevalence there are serious concerns about it s practical application [Haddon-Cave 2009] and theoretical underpinnings We ll look at the theory and application of safety cases with a focus on arguments in the context of acquisition We ll also discuss the problems and limitations of safety cases 8 Matthew Squair M12 Safety Cases and Arguments V1.0

9 Overview How is a safety case different to MIL-STD-882? A MIL-STD-882 system safety program Is acquisition focused (customer-supplier) Addresses proximal (system) causes of accidents Safety Assessment Report is analogue ish to a safety case A Safety Case Can be operation (operator-regulator) Convince a regulator the plant is safe to operate (WHS) Can be acquisition developed (DEF STAN ) Can be goal (more usual) or rule/standard based* *Safety cases have traditionally formed part of goal (performance) based safety regimes 9 Matthew Squair M12 Safety Cases and Arguments V1.0

10 Overview Why do it? Various reasons You may need a tool to manage operational safety You may wish to reduce liability risk The regulator may require as a permit to operate You may want to structure and organise safety documentation You may want to communicate system risk to stakeholders Be clear about the purpose Different stakeholders may mean very different things when it comes to safety cases, be clear about your purpose and who it serves when you prepare one 10 Matthew Squair M12 Safety Cases and Arguments V1.0

11 Overview Key definitions Safety argument. A safety argument is a clear, comprehensive and defensible argument that explains how the available evidence supports the overall claim of acceptable safety within a particular context [Kelly 1998] Safety case. A safety case is a structured argument, supported by a body of evidence, that provides a compelling, comprehensible and valid case that a system is acceptably safe for a given application in a given environment (i.e a context) [MOD (UK) 2007] Safety case report. The physical artifact(s) that presents the safety argument and case. Normally the safety case report is not a standalone document and will refer out to supporting evidence. 11 Matthew Squair M12 Safety Cases and Arguments V1.0

12 1 Introduction 2 Overview 3 Methodology 4 But do safety cases work? 5 Limitations, advantages and disadvantages 6 Conclusions 7 Further reading 12 Matthew Squair M12 Safety Cases and Arguments V1.0

13 Methodology [Bishop, Bloomfield 1998] 1 Identify safety requirements 2 Identify system architecture and outline the safety case 3 Assessment (preliminary) of concept design safety trades 4 Progressive elaboration of the design & safety case in parallel 5 Integrate into final safety case 6 Plan for long-term support infrastructure 7 Review and approval 8 Long-term monitoring and audits of areas of concern of support processes to gather field evidence to support assumptions 9 Revise to reflect system and context changes 13 Matthew Squair M12 Safety Cases and Arguments V1.0

14 Contents of a safety case Contents Contains at a minimum[kelly 1998]: Supporting evidence on which the case is based, because argument without evidence is unfounded A high level argument, because evidence without argument is unexplained May include a number of separate sub-arguments A convergent conclusion as to the acceptability of the system A meta-argument as to why the argument and evidence should be believed because both evidence and argument can be faulty [Hawkins et al., 2011] Is the totality of the safety evidence NOT just a safety case report Structure and organisation is essential to achieve clarity 14 Matthew Squair M12 Safety Cases and Arguments V1.0

15 Contents of a safety case Toulmin s model of practical arguments Current practices in formal safety argument are based on the practical argument model [Toulmin 1958] Focuses on the justification aspects of arguments rather than inferential. Argument parts consist of facts (evidence), conclusions, warrants, backing and qualifiers The warrant is why it s considered to move from the fact to the conclusion The rebuttal is a legitimate constraint that may be placed on the conclusion drawn Backing is evidence introduced if the warrant on the face of it is not credible 15 Matthew Squair M12 Safety Cases and Arguments V1.0

16 Contents of a safety case Toulmin s model (cont d) 16 Matthew Squair M12 Safety Cases and Arguments V1.0

17 Contents of a safety case A small philosophical quibble The problem is that Toulmin developed his model so that one could analyse an argument, that is argument is used in the verb sense Safety arguments tend to inherently skew to an advocacy position, and the rebuttal part of Toulmin s model gets overlooked, that is in safety arguments the word argument is used as a noun From there it is a small step to the narrative fallacy e.g. presenting all that good data that the system is safe 17 Matthew Squair M12 Safety Cases and Arguments V1.0

18 Contents of a safety case A small philosophical quibble The problem is that Toulmin developed his model so that one could analyse an argument, that is argument is used in the verb sense Safety arguments tend to inherently skew to an advocacy position, and the rebuttal part of Toulmin s model gets overlooked, that is in safety arguments the word argument is used as a noun From there it is a small step to the narrative fallacy e.g. presenting all that good data that the system is safe Of course there s very little evidence of rare catastrophic events because they re, well, rare Matthew Squair M12 Safety Cases and Arguments V1.0

19 Formal notations Formal notation for safety arguments Two formal notations are available Goal Structuring Notation (GSN). Developed by Kelly & others, there is a GSN community standard Claims, Arguments, Evidence (CAE). Developed by Bishop & others, supported by Adelard s Safety Case Editor tool Both are graphical in nature to assist in clarity of argument Both are based on Toulmin s practical argument structure Clarity does not denote soundness The use of one particular notation or another does not infer any greater or lesser soundness upon the actual worth of the argument 18 Matthew Squair M12 Safety Cases and Arguments V1.0

20 Formal notations Graphical notations for safety arguments GNS versus CAE notation 19 Matthew Squair M12 Safety Cases and Arguments V1.0

21 Developing the safety case Developing the safety argument (GSN notation) 1 Establish top level goals (customer/statutory) 2 Record the stakeholders for the goals 3 Define derived requirements (standards, codes etc) 4 Establish (3) as goals (or constraints) and link to top goals 5 Break down the top level goals into sub-goals 6 Show how design & analysis decisions meet goals via strategies 7 Record the decisions as they are made 8 Justify strategies Evidence versus argument Evidence without argument is unexplained, argument without evidence is unfounded 20 Matthew Squair M12 Safety Cases and Arguments V1.0

22 Developing the safety case Example fragment of a safety argument in GSN notation 21 Matthew Squair M12 Safety Cases and Arguments V1.0

23 Developing the safety case Dealing with scale and complexity GSN has been extended in reason years to include Safety case modules. Allow the partitioning of cases into more easily managed modules and module interfaces (systems of systems approach) Safety case patterns. Standardised templates to encourage re-use of successful arguments [Kelly, McDermid 1997] 22 Matthew Squair M12 Safety Cases and Arguments V1.0

24 Developing the safety case Example modular safety case Figure: Eurocontrol RVSM pre-implementation safety case 23 Matthew Squair M12 Safety Cases and Arguments V1.0

25 Developing the safety case Example modular safety case (cont d) Figure: Eurocontrol RVSM Implementation module 24 Matthew Squair M12 Safety Cases and Arguments V1.0

26 Developing the safety case Safety case patterns Figure: Safety pattern: functional safety argument 25 Matthew Squair M12 Safety Cases and Arguments V1.0

27 Maintaining the safety case Safety case maintenance In theory, a safety case should be maintained till system retirement Example The Long Term Safety Review of the U.Ks Magnox reactors, quoted in [Kelly 1998] found that lack of maintenance to the original safety case had caused it to become inconsistent with current plant design and operations. The review further found that adding to and re-evaluating a safety case that has become out of date with respect to current safety standards was problematic In practice, unless effort is expended to maintain the case it rapidly falls out of date A commitment to maintain requires regulatory & corporate buy in For some facilities (such as nuclear) the system life may be up to a century, longevity of evidence becomes a problem 26 Matthew Squair M12 Safety Cases and Arguments V1.0

28 Maintaining the safety case Safety case maintenance One of the biggest challenges is maintaining the safety case in the face of system changes We would like to use the safety case to assess changes for safety impact We also have to repair the case after a change has been made, hopefully in a cost effective fashion A graphical safety argument with traceability structures is invaluable for these purposes [Kelly, McDermid 2001] 27 Matthew Squair M12 Safety Cases and Arguments V1.0

29 Challenging the safety case Safety arguments as scientific hypothesis The best tool that we have for differentiating between a good theory and a bad one is the scientific method: our hypothesis is that our system is safe the argument is why we think this is justified in science a justifiable hypothesis is not considered proven in science the hypothesis is then challenged by others but with safety argument is this (ever) the case? The safety case as proof fallacy An unchallenged safety case is essentially an appeal to authority argument, authority in this case being how impressive the report is 28 Matthew Squair M12 Safety Cases and Arguments V1.0

30 Challenging the safety case So how do we challenge a safety case? Four broad avenues of attack: Deconstruction Refutation Disconfirming evidence And Matthew Squair M12 Safety Cases and Arguments V1.0

31 Challenging the safety case So how do we challenge a safety case? Four broad avenues of attack: Deconstruction Refutation Disconfirming evidence And... proof by construction that is have an accident or near miss (not recommended) 29 Matthew Squair M12 Safety Cases and Arguments V1.0

32 Challenging the safety case So how do we challenge a safety case? Four broad avenues of attack: Deconstruction Refutation Disconfirming evidence And... proof by construction that is have an accident or near miss (not recommended) The above might seem a lot but (for example) a claim that the likelihood of a LOCA accident is 10 9 per reactor year is a very strong statement, and strong statements demand strong proof surely? 29 Matthew Squair M12 Safety Cases and Arguments V1.0

33 Challenging the safety case Deconstruction Based on the work of french philosopher Jacque Derrida on the theory of meaning (and it s inherent indeterminacy) and his use of it in critiquing philosophical arguments [Armstrong, Paynter 2002] Derrida s view on arguments An argument is defined by what it ignores and the perspectives it opposes (explicitly or implicitly) 30 Matthew Squair M12 Safety Cases and Arguments V1.0

34 Challenging the safety case Deconstructionist technique Develop a counter argument that seems warrantable and use this to expose the internal flaws and contradictions in the original case 1 Reversal. Reverse the argument, ignore how warranted the original is & look for warrantable counter-arguments 2 Displacement. Compare the relative warrantedness of both 3 Evaluate the three possible end states The original argument is found to need revision The counter argument is found to need revision They both turn out to be equally compelling 1 1 Due to the limits of deductive closure 31 Matthew Squair M12 Safety Cases and Arguments V1.0

35 Challenging the safety case Deconstruction (Class exercise) Modelling software reliability Argument. Software failures occur randomly because of the random nature of inputs from the environment that trigger latent faults and that we can apply classical reliability techniques. What might be a warrantable counter argument, or arguments? 32 Matthew Squair M12 Safety Cases and Arguments V1.0

36 Challenging the safety case Refutation of argument [Greenwell et al. 2006] Challenge the specific arguments on the basic of fallacious argument structures and refute them 33 Matthew Squair M12 Safety Cases and Arguments V1.0

37 Challenging the safety case Disconfirming evidence Challenge the evidence with disconfirming evidence Based on Karl Popper s concept of the science project as one of trying to disconfirm theories not confirm them Consider Quality of the evidence provided (pool size, outlier handling, magic bullet approaches) Hazard control coverage metrics (is the argument vulnerable) Independence and dissimilarity of evidence sources Then go out and gather strongly disconfirming evidence that targets the gaps 34 Matthew Squair M12 Safety Cases and Arguments V1.0

38 But do safety cases work? 1 Introduction 2 Overview 3 Methodology 4 But do safety cases work? 5 Limitations, advantages and disadvantages 6 Conclusions 7 Further reading 35 Matthew Squair M12 Safety Cases and Arguments V1.0

39 But do safety cases work? Practical and theoretical problems with the approach A number of of significant safety cases have been reviewed, and problems found with them Magnox reactor safety review Haddon enquiry into the Nimrod disaster Ladkin analysis of the EUROCONTROL RVSM safety case Knight analysis of Opalinus Clay Nuclear repository safety case None of these were minor projects, so it appears that even when great care should be taken, flawed arguments still appear The theoretical problem is that for high consequence systems the likelihood must be very, very low and we must have a very high faith in the argument that this is so. Do we? 36 Matthew Squair M12 Safety Cases and Arguments V1.0

40 Limitations, advantages and disadvantages 1 Introduction 2 Overview 3 Methodology 4 But do safety cases work? 5 Limitations, advantages and disadvantages 6 Conclusions 7 Further reading 37 Matthew Squair M12 Safety Cases and Arguments V1.0

41 Limitations, advantages and disadvantages Limitations of the method Limitations Relies upon correspondence between safety argument and safety case Relies upon peoples ability to reason and argue effectively, there s not a lot of evidence that people are actually good at this 38 Matthew Squair M12 Safety Cases and Arguments V1.0

42 Limitations, advantages and disadvantages Advantages Advantages are that Is almost mandatory if working in a goal based regulatory environment Is invaluable in organising the safety program documentation tail Can promote thought and discussion, if used appropriately Can provide a change safety impact assessment capability in service 39 Matthew Squair M12 Safety Cases and Arguments V1.0

43 Limitations, advantages and disadvantages Disadvantages Disadvantages are that it Can become over time, another tick the box exercise Is vulnerable to the narrative fallacy Has a tendency to become an advocacy piece Is very hard to review effectively without formal training Can become an administrative burden that is perpetually chasing the system 40 Matthew Squair M12 Safety Cases and Arguments V1.0

44 Conclusions 1 Introduction 2 Overview 3 Methodology 4 But do safety cases work? 5 Limitations, advantages and disadvantages 6 Conclusions 7 Further reading 41 Matthew Squair M12 Safety Cases and Arguments V1.0

45 Conclusions Conclusions Safety cases emerged out of the political and industrial landscape of England in the late 1970 s, they reflect a particular societal viewpoint on both who should be responsible for managing major hazards should be managed and therefore how they should manage them. They are in the end another tool, neither an end in themselves nor demonstrably the only way to assure the safety of complex systems. Their current demonstrated deficiencies perhaps more demonstrate the difficulty humans have in arguing rigorously and logically, than any specific limitations of the method 42 Matthew Squair M12 Safety Cases and Arguments V1.0

46 Further reading Bibliography [Armstrong, Paynter 2002] Armstrong, J. M. and Paynter, S. P. (2002). Safe Systems: Construction, Destruction and Deconstruction. In: Redmill, F. and Anderson, T. (eds.), Current Issues In Safety Critical Systems, pp , Springer-Verlag, Berlin. [Bishop, Bloomfield 1998] Bishop, P. G. & Bloomfield, R. E. (1998). A Methodology for Safety Case Development. In: F. Redmill & T. Anderson (Eds.), Industrial Perspectives of Safety-critical Systems: Proceedings of the Sixth Safety-critical Systems Symposium, Birmingham [DoD (US) 1993] DoD (US) (1993) Standard Practice for System Safety (1993) US Dept of Defense Standard MIL-STD-882C, 19 January [Greenwell et al. 2006] Greenwell, W. S, Holloway, M., C. Knight, J.C., (2006) A Taxonomy of Fallacies in System Safety Arguments, Proceedings of the 2006 International System Safety Conference. [Haddon-Cave 2009] Cave, C.H. (2006) An Independent Review Into the Broader Issues Surrounding the Loss Of The RAF Nimrod MR2 Aircraft XV230 In Afghanistan in 2006, The Stationary Office, Tech. Rep., Matthew Squair M12 Safety Cases and Arguments V1.0

47 Further reading [Hawkins et al., 2011] Hawkins, R., Kelly, T., Knight, J. and Graydon, P. (2011) A new approach to creating clear safety arguments, in Proc. Safety Critical Systems Symp., Feb [Kelly, McDermid 1997] Kelly T, McDermid J. (1997) Safety case construction and reuse using patterns. In: Proc. 16th Intl. Conf. Computer Safety, Reliability, and Security (SAFECOMP97). New York, [Kelly 1998] Kelly, T.P., (1998) Arguing Safety, A Systematic Approach to Managing Safety Cases, Doctoral Thesis, Dept of Computer Science, University of York [Kelly, McDermid 2001] Kelly T, McDermid J. (2001) A systematic approach to safety case maintenance. Reliability Engineering and System Safety 2001;71(3): [MOD (UK) 2007] UK MoD (2007) Defence Standard Issue 4: Safety management requirements for defence systems, HMSO. [Toulmin 1958] S. E. Toulmin, S.E., (1958) The Uses of Argument, Cambridge University Press, Matthew Squair M12 Safety Cases and Arguments V1.0