A New Systems-Theoretic Approach to Safety. Dr. John Thomas

Transcription

1 A New Systems-Theoretic Approach to Safety Dr. John Thomas

2 Outline Goals for a systemic approach Foundations New systems approaches to safety Systems-Theoretic Accident Model and Processes STPA (hazard analysis) CAST (accident analysis)

3 Goals for a systemic approach Need to expand our view of safety Safety is dependent on many aspects Technical, human, organizational, etc. Need to understand the whole system of interactions Need to build in safety from the start Versus waiting to assure a finished design is safe Handle challenges in modern systems Traditional approaches developed for relatively simple electro-mechanical systems Software and digital complexity make exhaustive testing impossible Role of humans is changing Unanticipated and unexpected emergent system behavior

4 Goals for a systemic approach Need to expand our view of safety Safety depends on many aspects Technical, human, organizational, etc. Technical factors Easy to focus on independent random failures But other technical problems pose growing challenge Design errors Incomplete/incorrect requirements Esp. accidents from software operating as required Incorrect assumptions Technology

5 The problem doesn t exist in any single component It exists in the interactions among many components

6 Goals for a systemic approach Need to expand our view of safety Safety depends on many aspects Technical, human, organizational, etc. Software Doesn t fail like hardware Curse of software Most software-related accidents result from flawed requirements Technology

7 Goals for a systemic approach Need to expand our view of safety Safety depends on many aspects Technical, human, organizational, etc. Human behavior / social factors Human error more than random component failure Need to look deeper than human-machine interface Must consider: Clumsy automation, mode confusion, etc. How technology might induce human error Human error often a symptom of deeper trouble (Dekker) To fix, need to understand why it would make sense at the time Human Technology

8 China Airlines 006 Autopilot compensates for single engine malfunction Autopilot reaches max limits, aircraft turns slightly Pilots not notified Autopilot at its limits Pilots disengage autopilot for manual control Controls return to default Aircraft immediately nosedives

9 Goals for a systemic approach Need to expand our view of safety Safety depends on many aspects Technical, human, organizational, etc. Engineering and development Engineers are human too! Design/requirements errors are another form of human error Fixing design/requirements problems is not enough What about the processes that created them and analysis methods that overlooked them? Human (operations, engineering, etc.) Technology

10 Goals for a systemic approach Need to expand our view of safety Safety depends on many aspects Technical, human, organizational, etc. Stakeholders: vendors, regulators, contractors, public, etc. Organizational, managerial, leadership, culture, etc. Clearly impact safety, but too easily ignored How can we anticipate these influences? How do we include them in a systemic approach? Organizational Human / Social Technology

11 Goals for a systemic approach Need a hollistic view of safety Cannot consider these factors in isolation Highly dependent on interactions These are complex socio-technical systems Social must be integrated with the technical

12 STAMP: a systems approach (Nancy Leveson) A new view of safety based on systems theory Treat safety as a dynamic control problem Safety requires enforcing constraints on system behavior Accidents occur when interactions among components violate those constraints Safety a control problem, not just failure problem Captures dysfunctional interactions and unsafe system behavior Whether due to failures, design errors, flawed requirements, human behavior, etc. Includes unanticipated and unexpected behaviors Includes systemic factors for accidents Nancy Leveson, 2012, Engineering a Safer World

13 Safety as a control problem Examples O-ring did not control propellant gas release in field joint of Challenger Space Shuttle In HPCI example, did not adequately control the flow of water into the plant At Fukushima, did not control the release of radioactivity from the plant Software did not adequately control descent speed of Mars Polar Lander

14 Control Actions Controller Process Model Feedback Controlled Process STAMP Controllers use a process model to determine control actions Accidents often occur when the process model is incorrect Four types of hazardous control actions: 1) Control commands required for safety are not given 2) Unsafe ones are given 3) Potentially safe commands but given too early, too late 4) Control action stops too soon or applied too long Explains software errors, human errors, component interaction accidents, components failures 14 Copyright John Thomas 2013

15 Example Safety Control Structure

16 Control structure examples (from completed analyses)

17 HPCI/RCIC

18 Safety Control Structure

19 More Detailed Control Structure

20 Copyright John Thomas 2013 Proton Therapy Machine High-level Control Structure Gantry Cyclotron Beam path and control elements

21 Proton Therapy Machine High-level Control Structure Antoine PhD Thesis, 2012 Copyright John Thomas 2013

22 Proton Therapy Machine Control Structure Antoine PhD Thesis, 2012 Copyright John Thomas 2013

23 Image from: Chemical Plant

24 ESW p354 Copyright John Thomas 2013 Chemical Plant Captures interactions between Management, Operations, Technology, Engineering, etc. Image from:

25 Ballistic Missile Defense System Extremely complex system But the complexity is managed Image from: 21_Missile%201_Bulkhead%20Center14_BN4H0939.jpg Safeware Corporation

26 Copyright John Thomas 2013 U.S. pharmaceutical safety control structure Image from:

27 CAST and STPA CAST Accident Analysis STPA Hazard Analysis How do we find inadequate control in a design or accident? STAMP Model Accidents are caused by inadequate control Nancy Leveson, 2012, Engineering a Safer World 27 Copyright John Thomas 2013

28 Systems Theoretic Process Analysis (STPA) Method of applying STAMP for a design Integrates safety into system engineering Can drive design from the beginning of project (more efficient) Can also analyze hazards in existing design Starts at very high-level of abstraction Scalable to extremely complex systems Can help identify unexpected accident scenarios

29 STPA (System-Theoretic Process Analysis) STPA Hazard Analysis STAMP Model Identify accidents and hazards Construct the control structure Step 1: Identify unsafe control actions Step 2: Identify causal factors and control flaws Control Actions Controller Controlled process Feedback 29

30 Identifying Unsafe Control Actions Control Action Not providing causes hazard Providing causes hazard Incorrect Timing/ Order Stopped Too Soon / Applied too long

31 STPA Step 2 Inappropriate, ineffective, or missing control action Controller Inadequate Control Algorithm (Flaws in creation, process changes, incorrect modification or adaptation) Actuator Inadequate operation Control input or external information wrong or missing Process Model (inconsistent, incomplete, or incorrect) Sensor Inadequate operation Missing or wrong communication with another Controller controller Inadequate or missing feedback Feedback Delays Controller Delayed operation Conflicting control actions Process input missing or wrong Controlled Process Component failures Changes over time Unidentified or out-of-range disturbance Incorrect or no information provided Measurement inaccuracies Feedback delays Process output contributes to system hazard 31

32 Is it Practical? STPA has been or is being used in a large variety of industries Nuclear and Electrical Power Spacecraft Aircraft Air Traffic Control UAVs (RPAs) Defense Automobiles (GM, Ford, Nissan?) Medical Devices and Hospital Safety Chemical plants Oil and Gas C0 2 Capture, Transport, and Storage Etc.

33 Is it Practical? (2) Social and Managerial Analysis of the management structure of the space shuttle program (post-columbia) Risk management in the development of NASA s new manned space program (Constellation) NASA Mission control re-planning and changing mission control procedures safely Food safety Safety in pharmaceutical drug development Risk analysis of outpatient GI surgery at Beth Israel Deaconess Hospital Analysis and prevention of corporate fraud

34 Does it Work? Most of these systems are very complex (e.g., the U.S. Missile Defense System) In all cases where a comparison was made: STPA found the same hazard causes as the old methods Plus it found more causes than traditional methods All components were operating exactly as intended but complexity of component interactions led to unanticipated system behavior Examples: missing case in software requirements, timing problems in sending and receiving messages, etc. Sometimes found accidents that had occurred that other methods missed Cost was orders of magnitude less than the traditional hazard analysis methods

35 One Example: Blood Gas Analyzer (Vincent Balgos) 75 scenarios found by FMEA 175 identified by STPA Took much less time and resources (mostly human) FMEA took a team of people months to perform STPA took one person two weeks (and he was just learning STPA) Only STPA found scenario that had led to a Class 1 recall by FDA (actually found nine scenarios leading to it)

36 Automating STPA Hazards Hazardous Control Actions Formal (modelbased) requirements specification Can automate most of Step 1 (but requires human decision making) Formal underlying discrete mathematical models allow for automated consistency/completeness checks (can detect conflicts) Have not yet automated Step 2 (causes of unsafe control actions) 36

37 Thank you! Interested in systems approach to security? STAMP / STPA works for security too! Book: Engineering a Safer World MIT Press, 2012 (Nancy Leveson) STPA Primer More examples, exercises Search Google for STPA Primer