Safety in large technology systems. Technology Residential College October 13, 1999 Dan Little

Transcription

1 Safety in large technology systems Technology Residential College October 13, 1999 Dan Little

2 Technology failure Why do large, complex systems sometimes fail so spectacularly? Do the easy explanations of operator error, faulty technology, or complexity suffice? Are there managerial causes of technology failure? Are there design principles and engineering protocols that can enhance large system safety? What is the role of software in safety and failure?

3 Surprising failures Franco-Prussian war, Israeli intelligence failure in Yom Kippur war The Mercedes A vehicle sedan and the moose test Chernobyl nuclear power meltdown

4 Therac-25 high energies computer control rather than electromechanical control positioning the turntable: x-ray beam flattener 15,000 rad administered rather than 200 rad

5 Causes of failure Complexity and multiple causal pathways and relations defective procedures defective training systems human error faulty design

6 Technology failure sources of failure management failures design failures proliferating random failures storming the system design for soft landings crisis management

7 Information and decision-making Information flow and management of complex technology systems complex organizations pursue multiple objectives simultaneously complex organizations pursue the same objective along different and conflicting paths

8 Sources of potential failure hardware interlocks replaced with software checks on turntable position cryptic malfunction codes; frequent messages excessive operator confidence in safety systems lack of effective mechanism for reporting and investigating failures poor software engineering practices;

9 Causes of failure The causes of accidents are frequently, if not almost always, rooted in the organization--its culture, management, and structure. These factors are all critical to the eventual safety of the engineered system (Leveson, 47).

10 Organizational factors Large-scale engineered systems are more than just a collection of technological artifacts: They are a reflection of the structure, management, procedures, and culture of the engineering organization that created them, and they are also, usually, a reflection of the society in which they were created (Leveson, 47).

11 Advice for better software design design for the worst case avoid single point of failure designs design defensively investigate failures carefully and extensively look for root cause, not symptom or specific transient cause embed audit trails; design for simplicity

12 Design for safety hazard elimination hazard reduction hazard control damage reduction

13 System safety builds in safety, not simply adding it on to a completed design deals with systems as a whole rather than subsystems or components takes a larger view of hazards than just failures emphasizes analysis rather than past experience and standards

14 System safety (2) emphasizes qualitative rather than quantitative approaches recognizes the importance of tradeoffs and conflicts in system design more than just system engineering

15 Hazard analysis development: identify and assess potential hazards operations: examine an existing system to improve its safety licencing: examine a planned system to demonstrate acceptable safety to a regulatory authority

16 Hazard analysis (2) construct an exhaustive inventory of hazards early in design classify by severity and probability construct causal pathways that lead to hazards design so as to eliminate, reduce, control, or ameliorate

17 Safe software design control software should be designed with maximum simplicity (408) design should be testable; limited number of states avoid multitasking, use polling rather than interrupts design should be easily readable and understood

18 Safe software (2) interactions between components should be limited and straightforward worst-case timing should be determinable by review of code code should include only the minimum features and capabilities required by the system; no unnecessary or undocumented features

19 Safe software (3) critical decisions (launch a missile) should not be made on values often taken by failed components -- 0 or 1. Messages should be designed in ways to eliminate possibility of compute hardware failures having hazardous consequences (missile launch example)

20 Safe software (4) strive for maximal decoupling of parts of a software control system accidents in tightly coupled systems are a result of unplanned interactions the flexibility of software encourages coupling and multiple functions; important to resist this impulse.

21 Safe software (5) Adding computers to potentially dangerous systems is likely to increase accidents unless extra care is put into system design (411).

22 Human interface considerations unambiguous error messages (Therac 25) operator needs extensive knowledge about the theory of the system alarms need to be comprehensible (TMI); spurious alarms minimized operator needs knowledge about timing and sequencing of events design of control board is critical

23 Control panel anomalies

24 Risk assessment and prediction What is involved in assessing risk? probability of failure prediction of consequences of failure failure pathways

25 Reasoning about risk How should we reason about risk? Expected utility: probability of outcome x utility of outcome Probability and science How to anticipate failure scenarios?

26 Compare scenarios nuclear power vs coal power automated highway system vs routine traffic accidents

27 Ordinary reasoning and judgment well-known fallacies of ordinary reasoning: time preference framing risk aversion

28 large risks and small risks the decision-theory approach: minimize expected harms the decision-making reality: large harms are more difficult to absorb, even if smaller in overall consequence example: JR West railway

29 Scope and limits of simulations Computer simulations permit experiments on different scenarios presented to complex systems Simulations are not reality Simulations represent some factors and exclude others Simulations rely on a mathematicization of the process that may be approximate or even false.