Lessons Learned: 100 Questions That Should Be Asked during Technical Reviews

Transcription

1 SSED Application Example Lessons Learned: 100 Questions That Should Be Asked during Technical Reviews Seminar on Aerospace Mishaps and Lessons Learned 2004 MAPLD Conference 7 September 2004 Paul Cheng (310) Paul.g.cheng@aero.org

2 Why Do Satellites Fail? Unclassified U.S. Government Satellite Failures, 1990 Present Date Program Problem/Outcome Engineering Mistake Technology Surprise 04/90 Hubble A defect in the tool used both in manufacturing and in QA misshaped the mirror 07/92 TSS -1 Deployment mechanism jammed by a bolt added after I&T 09/92 Mars Observer Oxidizer reacted with braze, jamming regulator and bursting tank during pre ssurization 08/93 NOAA 13 The battery charger had low dimensional tolerance shorted out by a screw 10/93 Landsat F Pyrovalve ignited fuel nearby 01/94 Clementine CPU froze due to overload, allowing the thruster to deplete fuel 05/94 MSTI 2 Contact lost, probably due to micro meteor oid/debris impact or charging 12/95 Skipper Solar arrays miswired on drawing I&T did not ascertain current direction 02/96 TSS -1R Contamination within the tether caused arcing 08/97 Lewis Flawed GN&C design caused tumbling not saved due to inadequate monitoring 10/97 STEP -4 Damage by launch vibration. Ground test strategy improper 10/98 STE Solar array too hot, fatiguing solder joint s. Analysis used wrong configuration 12/98 MCO Unit mix-up in ground soft ware, coupled with vulnerable navigation scheme, caused trajectory error 01/99 Mars Polar Lander Requirement error prevented touchdown sensors from being protected against de - ployment shock. Engine shut down premature ly 03/99 WIRE A start-up transient in the pyro electronics controller prematurely ejected the telescope cover 08/01 Simplesat Transmitter arcing 07/02 Contour Plume analysis, based on similarity, misled by typo in an AIAA paper Count 14 6 Since

3 Remember Past Mistakes to Avoid Repetition Fools say that they learn by experience. I prefer to profit by others' experience. Otto Bismarck 100 Questions: Driver s Ed Movie for Engineers Based on lessons extracted from SSED data: Like Susan Lee did for NEAR! 79 catastrophic failures 32 major events (e.g., loss of an instrument) 21 ground problems (e.g., unit damaged during vibe) 3 recoveries of dead missions Examine: How did the mistake occur? What prevented its detection? Why did a flaw bring down the system?

4 The Thrust of Questions Questions are grouped in: Requirements Heritage and Qualification-by-Similarity Analysis Fault Management Embedded Software and Database Interface Parts, Materials, and Manufacturing Process Testing and Evaluation For example: Q 3-1 (Analysis): Have all critical analyses been placed under configuration control? See Lessons: 26 (STE Failure) and 83 (AC 70/71 Failures) Hyperlinks explain the context

5 Mars Polar Lander Failure Systems Requirement stated: The touchdown sensors shall be sampled at 100 Hz. The sample process shall initiate to keep processor demand constant. However, sensor data shall not begin until 12 m above the surface. Legs deployed; Unprotected sensors registered shock Software read stored sensor status; shut down engine This requirement did not flow down to software requirements One requirement, one statement Q 1-3 (Requirements): Are there lumped/nested requirements?

6 Launch Vehicle Failure A dual-payload launcher was used for a single payload. Hardware engineers redlined spec drafted by software engineers to facilitate wiring, and designed harness based on redlines Redlines fell through mission spec s cracks S/W and H/W incompatible Systems engineer failed to verify - viewed mission spec as software document and not subject to configuration control Generic test masked problem Q 8-15 (Testing): Does the system being tested represent the flight configuration? Forward Payload Aft P/L = Bridge Wire = Squib Firing Circuit = Interface Connection Mission Unique Generic Core Generic Configuration P/L Software Commanded Failed Mission P/L Hard Wired

7 Representative Questions for Electrical Engineers Are units and tolerances specified? See Mars Climate Orbiter failure* and Huygens launch pad damage Do testing independently confirm development results? See Hubble mirror aberration* Are handover procedures between two sources of control well defined? See START launch failure Does the harness design preclude mismating? See BP-TD launch failure *: Report available on klabs.org

8 Some Questions Specifically for Digital Engineers Can a momentary glitch cause a crash (will logic devices improperly reset following a brief undervoltage, for example)? See Delta 178 and Titan A-20 failures How are databases verified? See Centaur TC-14 failure Will unexpected inputs cause the computer to freeze, without a way to autonomously reboot? See Clementine failure and SPIRIT anomaly* Can the fault protection logic be set off too easily (e.g., can phantom sensor readings spoof the fault management system into taking precipitous actions)? See Ariane 501* and Atlas/Mariner 1 failure

9 More Items EEs Rarely Think of, but Should: Ambiguous drawing instructions Opposite engineering convention (right- or lefthand coordinates? Positive- or negative- ground?) Wiring crossover between two drawings Commandability after OBC faults disabled receivers Revivability of solar array regulator after battery drain Fratricide by pyro devices In-rush current welding relays shut FOD-caused shorting and arcing...

10 Using 100 Questions in Practice A satellite uses many low-shock deployment devices Consisting of spools of tightly wound wires Actuated by electrically severing restraining wires Logic Control Power Supply Arming Firing Relay Relay Releases Solar array, etc. Four problems found: Constant-voltage firing circuit may fail (SAFER lesson) Routing both arming and firing relays to one PLD (WIRE) If deployed wires touch firing circuits, battery can drain; power distribution board may overheat (Deep Space 1) Test circuits are constant-current (not flight-like)

11 In Closing Petroski s Law of Design: To engineer is human Akin's Laws of Spacecraft Design: Space is a completely unforgiving environment. If you screw up the engineering, somebody dies! For additional interesting quotes, see klabs.org