6 Secrets of Exploratory Testing in Embedded Systems. Shmuel Gershon Tel Aviv, July 2013

Transcription

1 6 Secrets of Exploratory Testing in Embedded Systems Shmuel Gershon Tel Aviv, July 2013

2 Who are you people? Testers Managers Web Explorers Programmers All of the others in one Embbeded Aficionados Curious In the wrong conference? 2

3 What am I doing here 3

4 Is this the tester you hire? PERSONAL FREEDOM RESPONSIBILITY of the individual tester OPTIMIZE THE QUALITY OF THE WORK LEARNING + DESIGN + EXECUTION + RESULT INTERPRETATION... THROUGHOUT THE PROJECT 4

5 Remember our Super Tester? "A style of software testing that emphasizes the PERSONAL FREEDOM and RESPONSIBILITY of the individual tester to OPTIMIZE THE QUALITY OF THE WORK by treating test-related LEARNING, test DESIGN, test EXECUTION, and test RESULT INTERPRETATION as mutually supportive activities that run in parallel THROUGHOUT THE PROJECT" (Cem Kaner) 5

6 Exploratory Testing by Cem: "A style of software testing that emphasizes the PERSONAL FREEDOM and RESPONSIBILITY of the individual tester to OPTIMIZE THE QUALITY OF THE WORK by treating test-related LEARNING, test DESIGN, test EXECUTION, and test RESULT INTERPRETATION as mutually supportive activities that run in parallel THROUGHOUT THE PROJECT" (Cem Kaner) 6

7 Embedded Systems: An embedded system is a computer system designed for specific control functions within a larger system, often with real-time computing constraints. It is embedded as part of a complete device often including hardware and mechanical parts. == It is hidden, untouchable, inside other systems! 7

8 Test This: 8

9 Now Test This: 9

10 Now Test This One: 10

11 And Finally Test This One: 11

12 Common Direction SDK API Automated Tests Automation Suite Seems as if reality forces you into it Handless Operation 12

13 Why Breaking the Pattern? Stiffness Long Feedback Loop Borderless infrastructure Premature learning

14 radio signal speed is slow (a signal from Mars to Earth takes Exploring the Untouchable Rovers on celestial bodies (such as the Mars Rovers) cannot be easily remote-controlled since the between 3 and 21 mins). A rovers is capable of operating autonomously with little assistance from ground control for data acquisition, but requires human input for identifying promising targets, and determining how to position itself. 14

15 Exploring the Hidden 15

16 16

17 How to Break the Pattern? 17

18 6 Principles Traduttore, Tradittore Caveat Testor Stay in Control Simple Autonomy Probes Principles Judge Not Verbs Cut the Middleman 18

19 Probes 1) Probes Have a system that gives you information, all the information, unfiltered, from all the system. You can filter it elsewhere. - Signals - Logs / States - Results - Timers / Watchdogs 19

20 Verbs 2) Verbs Name your actions: Put your hands inside the system. - If it can do something, you can do it too. - Not only direct inputs controls the environment - Sometimes the way to control it is to know how the system deals with environments and inputs - Your simulation + the system rules = state 20

21 Cut the Middleman 3) Cut the Middleman The system should allow tests to flow, without stopping to re-automate. - Short action and reaction cycles increase engagement and cognition 21

22 Judge Not 4) Be cautious in judgment Computers help get results, testers interpret results. - You can t possibly teach a computer how an unknown bug looks like 22

23 Stay in Control Simple Autonomy 5) Stay in Control If you are exploring, you need the steering wheel. - Cruise Control may be for known areas - Automatic Pilot may be for getting started 23

24 Traduttore, Tradittore Caveat Testor 6) Traduttore, Tradittore We are building an abstraction, and all abstractions are leaky. All models are false but some models are useful. George E. P. Box 24

25 The only summary slide you need to remember: It s all about building an automated system for manual tests 25

26 (Free!) Bonus Benefits! Find Limits When Automated Tests are needed they can use the same Manual infrastructure Debug, Troubleshooting, Bug Isolation 26

27 More Bonus! Automation Layers Test Cases Automation System Structure Test Structure 27

28 Credits, Questions, Answers Michael Stahl Gary Draishpits Cem Kaner Michael Bolton Doron Blazer Doron Maliach Yonatan Tidhar Bat-Sheva Honigstein Intel Sensors Team o My Manager too! QA & Test 28