Some caveats Summary

Transcription

1 Better by Design Graham Freeburn Newell & Budge Testing Solutions Tel: +44(0) Fax: +44(0) The Menu Aperitif Starter Introduction A light bulb.. Bad Design, No Design Main Course Working with developers Learning from Engineering Desert Some caveats Summary

2 A Light Bulb.. Testers only find failure.. Testing is (still) often too little, too late How real are requirements? Tester s aren t the only ones with problems A tale of two alarm clocks. Is good design (with tester involvement) a way forward? Bad design.. No Design.. Why people don t design Some examples Software Tale of Two Alarm Clocks You can use an eraser on the drafting table or a sledge hammer on the construction site. Frank Lloyd Wright

3 Why people don t design Old Days Get the &*!%*? Coded! Fixed Architectures Try it and see New World Time Pressure Development Tools Requirements Code Re-use Packages / COTS Software So many so little time.

4 August 2004 Generic command and control system Originally for Fire Brigade Modified for use by Coastguard Crashed and reverted to orignal design Turn left into High Street.. In a helicopter! Newell & Budge, Newell & Budge, 2004 Software 4

5 Last month! UK Government Department Desktop upgrade Caused 80,000 machines (80% of total) to fail No so officials had to communicate by handwritten faxes Tale of Two Alarm Clocks the sublime Time set Alarm On/Off Alarm set Hour Drowse Minute Source: The Power of Modern Testing, Les Hatton, Scottish Testing Group (STG), Oct 2003

6 Tale of Two Alarm Clocks the ridiculous Random Numbers More Buttons Mode -ARGH! Desk Source: The Power of Modern Testing, Les Hatton, Scottish Testing Group (STG), Oct 2003 Alarm Clock 2 some examples Time setting Press MODE button THREE times. Then press SELECT and SET repeatedly Alarm Setting Press MODE button TWO times. Then follow time setting. Hourly chime on/off Press SELECT and MODE button simultaneously Alarm on Press DATE and SELECT button simultaneously Stop watch/lap counter Press MODE button ONCE and then beg for mercy as clock gibbers with entertaining series of random beeps Turning alarm off Hurl out of window after standing at clock screaming. Source: The Power of Modern Testing, Les Hatton, Scottish Testing Group (STG), Oct 2003

7 Working with Developers Make the case for design Get involved Designing for Testability Testing the Design UML Design Mapping Quality Planning Early Test Design Exploiting Bad Design Working with developers - designing for Testability As applications become more complex, spreading over multi-tiers, across varying networks, and including different client configurations, application architecture testing will become increasingly mandatory» GIGA GROUP Software Design for Testability standards (00s) Design for Testability standards (late 80s) UML (late 90s) Large scale ICs (early 70s) Integrated Circuits (late 60s) Transistors (1960s) Object Languages (early 90s) 3GLs, 4GLs ( 70s & 80s ) Assembly language (1960s) Source: Achieving ebusiness Quality by Design, Sam Guckenhiemer, Rational, Software Test Automation, 2001

8 Working with developers - testing the design Sequence Diagrams Class Diagrams Use Case Diagrams Collaboration Diagrams Statechart Diagrams UML Activity Diagrams Packages & Object Diagrams Component & Deployment Diagrams Working with developers - testing the design Design Mapping - 4C compliance Correctness Completeness Activity Diagram 1. States are identified by the same name 2. Events, guard conditions and activities are identified the same State Diagram Use Cases correspond with Activities Actors are identified by the same name Actors are identified by the same name Objects are identified by the same name State trigger events are Class methods Use Case Diagram Actors are identified by the same name Actor Hierarchy Diagram Actors are identified by the same name Class/Type Diagram States should correspond State events should correspond with Sequence messages Use Case Name is identical Diagram Associations are described in Description Extension Points correspond Includes correspond Actors are identified by the same name Actors are identified by the same name Objects should correspond Sequence messages are Class object methods Use Case Textual Description 1. Use Case steps should correspond with Sequence Diagram message 2. Sequence begins in a state that satisfies the Use Case pre conditions 3. Sequence ends in a state that satisfies the Use Case post conditions Sequence Diagram Consistency Source: The Road to UML is paved with Good Intentions, Dion Johnson, StarEast 2002 Clarity

9 Working with developers - testing the design Quality Planning Options / Solutions Attributes Attribute 1 Attribute 2 Opt A Opt B Opt C Opt D Opt E etc Attribute Impact Sum 60%+20% c=0.6 40%+ 10% c=0.2 Attribute 3, etc 75%+ 10% c=0.8 Option Impact Sum Costs Benefit to cost ratio Source: Tom Gilb, Principles of Software Engineering Management Addison Wesley, 1988 Working with developers - exploiting bad design Exploratory / Rapid Testing Cem Kaner, James Bach Simultaneous observation design and execution of tests Heuristics, Exploratory branches Brain on, eyes open, test! 17 User Interface Attacks 6 system Interface Attacks 2 Software/OS Interface Attack approaches

10 Learning from Engineering What is good design? Boeing 777 Paperless design allowed simulated assembly of 777 Worked so well that only a nose mock-up was built before assembly first flight vehicle Building Bridges Design of experiments (DOE) Information Feedback The Importance of Context Learning from Engineering - what is good design? Useful It works well and functions as promised. It does what it is expected to and satisfies a minimum or appropriate level of performance Usable It has appropriate ergonomics and user interface, considering how, where, how often and who will be using it. Desirable Producible Profitable Differentiated

11 A bridge to nowhere.. Spiral ramp of the larger 'bridge to nowhere', with a stub end where it should have been extended to the other (there were two!). This end of the bridge has no guardrails at all, and the base of the spiral is protected by a high spiked fence. The other end does have guardrails, but finishes abruptly with a 60 foot drop. - M8 Motorway, Charing Cross district of Glasgow Learning from Engineering - design of experiments (DOE) DOE have been applied in Quality control in many engineering fields for several decades Aim : to maximise the amount of information gained in an experiment by optimising the combinations of independent variables In Software Testing Terms Combination or Combinatorial Testing E.g. All Pairs Conditions Browser Server Modules (values on n) (194 bugs) (171 bugs) Table 1. Number and percent of faults triggered by n-way Conditions (percent) (cumulative (percent) (cumulative percent) percent) Source: An Investigation of the Applicability of the Design of Experiments to Software Testing, D. Richard Kuhn & Michael J. Reilly, Proceedings, 27 th NASA/IEEE Software Engineering Workshop, December 2002

12 Learning from Engineering - Information Feedback (design to operation) Source: Mapping Function to Failure Mode during Component Development, Irem Y Tmer and Robert B. Stone Research in Engineering Design 14 (2003) Learning from Engineering - context is everything Mission Find Important Problems Assess Quality Certify to Standard Fulfill Process Mandates Satisfy Stakeholders Assure Accountability Advise about QA Advise about Testing Advise about Quality Maximize Efficiency Minimize Cost Minimize Time Development Product Project Lifecycle Project Management Configuration Management Defect Prevention Development Team Test Process Strategy Logistics Work-products Requirements Product Mission Stakeholders Quality Criteria Reference Material Test Team Expertise Loading Cohesion Motivation Leadership Project Integration Test Lab Test Platforms Test Tools Test Library Problem Tracking System Office Facilities Source: James Bach, Satisfice Inc. 2003

13 Some caveats The need for good requirements analysis isn t diminished Economics Practicality of design proving Total Quality Costs Cost of Quality Costs Costs for prevention and detection Minimum Failure Costs Quality Level Source: Juran s Quality Control Handbook, 1988 ISBN Summary Get involved in design For everyone s benefit Consider using Design Mapping Use Cases, Scenario testing They are powerful weapons in the war against faults Learn to exploit bad design Rapid Testing / Exploratory Testing Get in early!

14 Resources The Rational Unified Process An Introduction, 2 nd Edition Phillippe Kruchten, ISBN Total Design, Stuart Pugh, ISBN No Design Secrets : Products Industrial Designers Society of America (IDSA) ISBN