Game Production: testing

Transcription

1 Game Production: testing Fabiano Dalpiaz 1

2 Outline Lecture contents 1. Intro to game testing 2. Fundamentals of testing 3. Testing techniques Acknowledgement: these slides summarize elements from chapters 1 to 3 and 6 to 13 of the book Game testing: all in one by Charles P. Schultz, Robert Bryant and Tim Langdell 2005, Course Technology, ISBN:

3 1. Intro to game testing Why is testing important? Game software is complex People write game software and people make mistakes Software tools for building games are not perfect There is a lot of money at stake for games to succeed Multiple platforms, configurations, devices People expect more out of every game you make Paying customers don t like bugs Critics are standing by ready to rate your game 3

4 1. Intro to game testing Two rules of game testing Rule 1 Rule 2 4

5 1. Intro to game testing Rule 1: don t panic You encounter game project panic when you are Unfamiliar Unprepared Under pressure Unrested Nearsighted quite often, this corresponds to crunch time Luckily, there are systematic techniques to get prepared to crunch time 5

6 1. Intro to game testing Rule 2: trust no one You need to test something when that something cannot be trusted If you trust component X, you don t need to test it Do not trust developers Only a few lines of code have changed, so don't inspect any other lines The new audio subsystem works the same as the old one, so you only need to run your old tests It is your responsibility to provide evidence that it works Find the balance between test and trust: why? 6

7 1. Intro to game testing The PIANo TV sequence A sequence of activities to characterize game testing Play the game Identify bugs Amplify problems (generalize it) Notify the team Testify to others Verify the fix 7

8 1. Intro to game testing a. Playing the game Playing to crash it As opposed to playing to have fun Testing can be fun but testers don t choose what to test, when, and how to play All play is done with a purpose Explore an area of the game Check the enforcement of certain rules Look for a specific kind of problem Follow the instructions Repeat the same sequence 8

9 1. Intro to game testing b. Identifying bugs Testing has two purposes Find defects in the game code and design Demonstrate which parts of the game are working well Given a test The test finds no problem PASS The test finds a problem FAIL Cannot even get to relevant part of the test BLOCKED Important: passing all tests does not mean that the game has no bugs Why? 9

10 1. Intro to game testing b. Identifying bugs: tester personality Not all testers will find the same bugs The effect personality type (Myers-Briggs Type Indicator) Judger: you prefer a structured, ordered, and fairly predictable environment Perceiver: you prefer to experience as much of the world as possible 10

11 1. Intro to game testing c. Amplifying problems Objective: narrow down the scope for the developers Help them localize the problem Get the defect fixed right the first time Reduce overall time and cost spent on the problem Key principles to do so Early bird detection: identify bugs as soon as new features are introduced ( don t wait crunch time) Find all places in the game where the bug occurs Not only in the place you identified it first Increase the frequency Eliminate unnecessary steps to get it appear Find more frequent scenarios 11

12 1. Intro to game testing d. Notifying the team Record the information and notify the developers Typically, via software tools Bug tracking systems It is essential to deliver an accurate report Proper title Informative description Type of bug: crash, rendering issue, change request, Priority Replicability (How? Always?) Any other useful information (e.g., screenshots, logs, videos) 12

13 1. Intro to game testing d. Notifying the team defect typing Function error: one that affects a game capability or how the user experiences the game. The code providing this function is missing or incorrect in some or all instances E.g., missing code to activate turbo in a car racing game Assignment: it is the result of incorrectly setting or initializing a value used by the program or when a required value assignment is missing E.g., scoreboard in a soccer game initialized at 1-1 Checking: the code fails to properly validate data before it is used E.g., using = instead of == 13

14 1. Intro to game testing d. Notifying the team defect typing Timing: has to do with the management of shared and real-time resources E.g., the bg music of level 1 does not stop when level 2 starts Build/Package/Merge: the result of mistakes in using the game source code library system, managing changes to game files, or identifying and controlling which versions get built E.g., code from two incompatible versions are compiled Algorithms: faults in the developed algorithms E.g., NPC freezes, either due to unforeseen situation or due to excessive thinking 14

15 1. Intro to game testing d. Notifying the team defect typing Documentation: occur in the fixed data assets that go into the game E.g., grammar errors and typos, empty help screen Interface: occurs at any point where information is being transferred or exchanged E.g., passing the wrong value of one argument 15

16 1. Intro to game testing e. Testify to others Testers get emotionally attached to their found defects but the final decisions are made by others The project or development manager, typically The change control board, in large organizations these people are not as much attached to the defects How are these decisions made? Prioritization: are there more urgent bugs? Crash or not? Fixing too expensive A certain amount of bugs may be acceptable a business decision, in the end 16

17 1. Intro to game testing f. Verify the fix Eventually a fix will be created (not always though) What do testers do then? Repeat the same sequence of steps to check that the bug has been correctly fixed note that this may introduce new bugs! 17

18 2. Fundamentals of game testing Ensuring quality Basic quality means the game should be fault-free Not the only factor though What else determines the quality of a game? Quality of the story Quality of the game mechanics Quality of in-game audio and visual effects Beauty of the visual style Use of humor and exaggeration NPC Artificial Intelligence (AI) Clear interface Ease of use Compatibility with target platform (console, smartphone, ) 18

19 2. Fundamentals of game testing Appraising quality: not only testing The actual quality of a game is defined by its design and code implementation How to appraise the quality? Testing is a prominent technique but not the only one Testing is expensive! Soft techniques to catch defects as they are introduced Walkthroughs (up to 10 people) Reviews (5-6 people) Checklist-based reviews (author-reviewer) Inspections Adherence with standards (user interface, code naming) 19

20 2. Fundamentals of game testing Measuring quality: sigma level Sigma level: defectiveness of code relative to its size Defects per million lines of code Lines of code are normalized, according with the language 1 line of C code = 3 lines of assembly code 1 line of Perl code = 15 lines of assembly code Sigma 6 = outstanding Sigma 5.5 = very good All depends on how many defects are actually found! 20

21 2. Fundamentals of game testing Measuring quality: phase containment Phase containment: success of the team at finding defects in the code Measure: phase containment effectiveness (PCE) Faults identified in the phase they were introduced are known as errors Typically found by walkthrough, reviews, testing Faults identified in later phases are known as defects Typically found by customers PCE = #in-phase faults / (#faults in that phase and subsequent phases) 21

22 2. Fundamentals of game testing Black-box testing Most game testing is black-box: testing is done from outside the application No knowledge of the source code is required 22

23 2. Fundamentals of game testing White-box testing White-box testing gives the tester the opportunity to look into the source code Extremely difficult, due to the complexity of the code This is done for certain situations Tests performed by developers prior to integrating new code Modules that will become part of a reusable library Methods/functions that are essential part of a game engine Parts that can be used by modders Low-level routines to interact, e.g., with new hardware 23

24 2. Fundamentals of game testing Test cases and test suites Test case: a single test performed to answer one question Test suite: a collection of test cases Every tester is assigned a set of test suites He/she is responsible for them throughout the project In its simplest form, a test suite is a series of test cases that the tester can perform sequentially Typically, test suites are more elaborated IF/THEN/ELSE LOOPS 24

25 2. Fundamentals of game testing Managing testing The overall testing phase shall be managed continuously Just like other phases of (game) production What to check? Are testing suites effectives? Are testers effective? Are we successfully tackling all severe defects? Is testing on track? What to do when achieving bad results? Apply compensation plans Typically defined before that through risk assessment 25

26 3. Testing techniques Combinatorial testing Testing is a trade-off between cost and amount of testing Combinatorial testing is a good mix Pairwise combination: each value you use for testing needs is combined at least once with each other value of the remaining parameters Take the following example with three parameters Gender = Male, Female Light saber = One handed (1H), two handed (2H) Force = Light, Dark 26

27 3. Testing techniques Combinatorial testing A full combinatorial table would require 8 tests exponential in the number of parameters and possible values per parameter 27

28 3. Testing techniques Combinatorial testing A pairwise combinatorial value shall define one test case for every pair of input parameters Male Gender paired with each Light Saber choice (1H, 2H) Female Gender paired with each Light Saber choice (1H, 2H) Male Gender paired with each Force choice (Light, Dark) Female Gender paired with each Force choice (Light, Dark) 1H Light Saber paired with each Force choice (Light, Dark) 2H Light Saber paired with each Force choice (Light, Dark) 28

29 3. Testing techniques Test flow diagrams (TFD) Graphical models representing game behaviors from the player s perspective Testing = travel through the diagram Every TFD consists of elements Flows connecting states Triggering events Actions to be performed Special states: IN (start) and OUT (end) Semantics: when a certain event occurs, and currently in state S, execute a certain action to reach state S 29

30 3. Testing techniques Designing a TFD Basics: start from the IN state, create a set of states and flows that lead to the OUT state Example: consider the feature of picking up a weapon and its ammo while the game properly keeps track of ammo count and performs correct audible and visual effects 30

31 3. Testing techniques Designing a TFD When the player gets a gun, gun effects must be played, and the player has a gun 31

32 3. Testing techniques Designing a TFD When the player gets ammo, ammo effects are played, and the player has ammo 32

33 3. Testing techniques Designing a TFD supporting dropping ammo and gun 33

34 3. Testing techniques Designing a TFD and getting ammo/gun when the player has gun/ammo already 34

35 3. Testing techniques Designing a TFD Support shooting all ammos, get loaded gun, 35

36 3. Testing techniques Designing a TFD Define the acceptance criteria for each event/action Action AmmoEffects Check that the Bio-Rifle ammo sound is made Check that the game temporarily displays "You picked up some Bio-Rifle ammo" in white text above the gun icons at the bottom of the screen Check that the temporary text on the display fades out slowly Event DropGun Hit the "\" key to drop your selected weapon 36

37 3. Testing techniques TFD paths Test path: a series of flows, specified by the flow numbers in the sequence in which they are to be traversed an individual test case Paths begin at the IN state and end at the OUT state. A set of paths provides behavior scenarios appropriate for prototyping, simulation, or testing Many paths are possible for one TFD! So, which paths should testers execute? Minimum path generation: every flow is followed 1+ times Expert path generation: based on the expertise of testers 37

38 3. Testing techniques TFD paths: exercise Use minimum path generation: every flow is followed 1+ times 38

39 3. Testing techniques Combinatorial testing or TFDs? Depends on the type of functionality to test The following table provides some guidance 39

40 3. Testing techniques Cleanroom testing Cleanroom testing: exercise software to determine mean time to failure (MTTF) measurements over the course of a development project Applied to games to assess why customers find problems even after games were tested for thousands of hours Essence of cleanroom testing: produce the tests that play the game the way players will play it Objective: prevent the most common problems that players would encounter 40

41 3. Testing techniques Cleanroom testing: usages Different usages imply using different game elements Mode-based usage Single player, campaign, multiplayer, online Player-type usage Achiever, explorer, socializer, killer Real-life usage Log usage from real players E.g., from previous editions, similar games, or early builds 41

42 3. Testing techniques Cleanroom testing: generation Cleanroom tests can be created for any testing method Required input: usage probability To be assigned to each step in the test Example: Halo advanced controls Features Usage probabilities per feature 42

43 3. Testing techniques Cleanroom testing: generation Generated tests (example, no details) 43

44 3. Testing techniques Cleanroom testing: generation An example for cleanroom TFDs The figure shows probabilities One can generate test paths only for most likely flows (or start from those) In inverted usage, emphasis on less frequent functions 44

45 Mandatory literature Memon, Atif M., Martha E. Pollack, and Mary Lou Soffa. "Hierarchical GUI test case generation using automated planning." Software Engineering, IEEE Transactions on 27.2 (2001): Nie, Changhai, and Hareton Leung. "A survey of combinatorial testing." ACM Computing Surveys (CSUR) 43.2 (2011):