Lesson 07 : Playtesting

Transcription

1 Lesson 07 : Playtesting TO DO: - Read this document. - Read Players Making Decisions: Chapters 6 and 7. - Read the Threes Mails ( You don t have to read the entire document, but make sure you read at the very least up to the table of contents, skim the mails, and read the conclusion. This is an important postmortem because it focuses deeply on the ups and downs of developing an original game. - Complete the assignment at the bottom of this document. ( Playtests ) The Alien Mind 2 Playtests 2 Benefits of Playtesting 2 Difficulties of Playtesting 3 Playtesting Methods 4 Assignment: Playtests 8 Grading Criteria 9 This assignment is estimated to take around ten hours to successfully complete at a high level. This estimate is just a guess. Your actual times may vary. This information is provided to help you plan your work schedule. 1

2 The Alien Mind In the previous lesson, we discussed how the very fact that people differ in personality traits means that they will differ in how they approach, use, and enjoy a game. The designer needs to understand that the minds that interact with the game will be alien to him or her and must treat them as such. We, as designers, guess at how the alien minds will react to different stimuli, but we can only hypothesize. Our major tool for proving or refuting those hypotheses is the playtest. Playtests Playtests are the science part of game design and one of the most important techniques in a game designer s toolbox. You ve seen playtesting in Game Design 1 and will see it again in detail in Game Design 2 and other classes. It is so important that it must be reiterated time and again because it is incredibly difficult to do correctly! Benefits of Playtesting Playtesting is one of the most difficult things to do as a game designer. We have to sit back and watch people struggle with and not enjoy our games that we've spent blood, sweat, tears, and sleepless nights over. But think of this: if you didn't playtest the game, you would never have a chance to fix what is wrong with it and every one of your players will struggle like the one you are watching. Project postmortems like the ones you can read in GameDeveloper Magazine or on Gamasutra.com often highlight strange jumps in insight gleaned from playtesting. For instance, in Team Fortress 2, Valve was noticing that players were dying from the same snipers over and over again. Players were reporting in the playtest that they had no idea what was going on, that they would just walk out and die. It was very frustrating. It was something the designers hadn't thought of. They designed the maps, so they knew exactly where the snipers would hide. This lead to the creation of the Deathcam. Whenever a player died, the camera would zoom into the location of the player's killer. Not only did that inform the losing player where the sniper was camping (which cut down on camping), but it also allowed the winning player to perform a taunt when on the "Deathcam". The playtest helped identify a problem whose solution created additional gameplay hooks. Richard Garfield, when working on the original Magic: the Gathering, had one of his playtesters say that he had the best card in the game. That when he cast it, he would guarantee victory on the next turn. Garfield had him show the card. The card was "Time Walk, which read, "Opponent loses next turn." Garfield had written it to mean "Opponent skips his next turn" but the playtester understood it to mean "Opponent loses game next turn." Playtesting allowed that ambiguity to show up! What s obvious to you may not be obvious to the alien mind. 2

3 One of the most difficult things to understand is that it is impossible to know how beneficial a playtest will be before the fact. You may have five playtests that yield nothing, only to have the sixth be a bounty of usable feedback. You must never quit. You must playtest long after you think you've squeezed all the blood from the stone. Once you get a "good" playtest, it can be so tempting to quit and say that your game is done! Difficulties of Playtesting Playtesting is not easy. It requires you to find an appropriate set of players, set them up to play the game, and conduct a scientific examination of them interacting with what is an incredibly personal artifact. The most common pitfall that keeps novice designers from conducting successful playtests is the fear of critique. Even if you are very good at handling criticism, there is still a layer of your consciousness that dismisses criticism, saying either "Well, he didn't get it. Stupid playtester" or "That's fine that he didn't have fun. The game isn't done yet." At the other end of the spectrum is the designer who demands that everyone accept his/her vision at face value. For that designer, the playtest can never yield valuable results except egostroking. This kind of designer is lampooned in the Penny Arcade comic below: Some designers mask this fear of critique with an attitude of superiority to playtesters, that they will only tell you things you already know. This is partially true. I remember playtests at EA during Superman Returns for the DS (oh God) where kids would tell us that you should be able to go inside every building. That was obviously a technical impossibility, but it didn't stop hordes of playtest kids from asking for it. Disgruntled designers would then say that playtests were worthless based on this bad feedback, ignoring all other possible benefits it might have just to protect their egos. 3

4 One of the other difficulties with playtesting is that it often gives you advice that you don't want to hear. In writing, there is an aphorism: "Murder Your Darlings. What this means is that when you create something that you think is clever and beautiful, it can be difficult upon revision to go back and see that it does not serve the story at all. It is merely a clever turn of phrase for cleverness' sake. In games, it is sound advice as well. We are often blind to what our darlings are, but playtesting can identify them: segments or mechanics that aren't fun, are odd for variety's sake, or just don't make sense. Sometimes our playtests identify these problems. We then have to let go of ego and remove them or edit them into a usable state. This is very difficult! Remember: every good novel has edits and rewrites. Every good film has script rewrites and scenes that were shot and dumped. Why should a game just be the first thing you can think up? Perhaps the most difficult element of playtesting is finding good playtesters. We likely do not have dossiers of individuals who will come and give good feedback for you on a moment's notice for free. But the most available people for us often aren't the best individuals for a playtest. Your mother or your girlfriend/boyfriend aren't the best testers. They likely want to see you succeed and will be overly nice to you, even if only subconsciously. Your game designer/enthusiast friends also usually aren't good playtesters. They find it difficult to play as a normal player would. They often want to offer you solutions instead of identifying your problems. They see directions that the game could go that you long since dismissed. While they can provide useful commentary, they have a difficult time being unbiased playtesters. Finally, you have to find people who are the target of what game you want to make. If my friend's roommate is available, but she only plays shooters, she may not be the best playtester for my Match-3 game. If you cull your social circle down, finding available playtesters can be very difficult. The best playtesters are strangers who don't owe you anything and have no preconceived notions about you or your game, but the difficulty there is obviously convincing them to playtest for you in the first place. Another difficulty of playtesting can be in how to interpret the results. Not every playtester is correct. Because they have different backgrounds and experiences, they can give conflicting feedback. Joe thinks that the game is too hard and Jill thinks that the game is too easy. Who is right? Often, designers will "side" with the playtester who confirms his or her held beliefs. If the designer thinks the game is too easy as is, he will dismiss Joe and embrace Jill's opinion. This, on its own, is a mistake. Jill may be right, but more testing Is needed. Nonetheless, you will have a limit on how many playtests you can reasonably do due to many constraints. At some point, you need to decide what feedback to take and what to ignore. I've found no reasonable heuristic for this. Only by practice can you learn to extrapolate singular playtest data to classes of players. Playtesting Methods Now we come to the obvious question: how do we run a playtest? 4

5 In the previous section, we talked about finding "good" playtesters, folks who will give honest feedback without sugar-coating (friends/family) or being hyper-critical (hardcore gamers). Let's say you've got these playtesters ready, willing, and able. Sit them down in a comfortable place with your game. Get yourself a notebook. If you can record the session (and the playtesters consent), then do so. The playtest is about collecting good data. One of Microsoft's testing lab records video of player faces and hands along with the footage on screen so that designers can correlate facial emotions with what they are trying to accomplish. Most of us don't have that kind of luxury, so use what you have available. Sit the players down and tell them to play the game as they would at home. Do not tell them what the game is about or what they have to do, unless your game is in such a state that it would be impossible to play otherwise. Do not explain anything. Remember: the goal of the playtest is not for the playtester to have fun, but for you to gather data. You will not be shipping a copy of yourself with the game, so any information you give the player is beyond what your release players will get. There is a technique in usability testing developed in the 1980s at IBM called the "Think Aloud Protocol". In it you prompt the user to narrate everything they are doing, thinking, and feeling as they test. This is very hard for people to do while playing a game! They remember for a few seconds, but every time they are faced with a challenge, they focus their attention on the challenge and forget to talk. It's up to the tester then to continually remind them by asking questions like "How are you solving that puzzle?" "What is your goal here?" This is the closest we have to getting "inside" a playtester's thoughts. By keeping them talking about their ideas and motivations, we can track down poor assumptions, frustrations and desires. For instance, if you keep seeing players go to a certain area on the map, you may not think anything of it. But if you keep getting testers who when thinking aloud say "Oh, I thought there would be a treasure there", then you may want to consider what about that area of the map is drawing players there and perhaps actually implementing treasure there. If a player cannot find a way out of a room and keeps using the attack button on a door, by hearing his thoughts of "I want to open the 5

6 door", you know that he isn't trying to attack the door and that you may want to consider merging the use and attack buttons. The important point is to always keep your user talking with as little prompting from the facilitator as possible. It's really frustrating to watch a playtester fail again and again. You will just want to give them the answer so they can continue, but you must resist to see what the tester will do and how they are feeling. The only time you will want to direct the user is when your whole playtest will unravel if you do not. Note where the "give up" point is and direct the user to the next thing. Back to the ego point again because it bears repeating. It is very easy when a tester complains about something that isn't finished to object and say "Yeah, well that part isn't done yet" or "We know. We are working on it" but you must resist. In fact, if you say anything, you should ask "Why?" This will allow you to dig deeper into what the player desires. Remember that your job is not to make him or her happy, but to get good data. While the player is playing the game, you will be writing down everything you can. Write as fast as you can. What is happening? Where is the tester failing? What did he/she get right away? How long did it take him/her to achieve Objective X? Everything that you notice should be written down to analyze later. Don't make any judgements now! Just the facts! There will be time enough for judgment later. Always ask "Why?" The idea of Socratic questioning is very useful in playtesting to dig out the fundamental issues. One of the problems with it is that players often don't know what the underlying issues are. Ask users why they feel the way they do, what evidence they have that supports that feeling and what that feeling means for other elements in the game. For instance, if they try to jump on a ledge that the designer has an invisible wall in front of, ask "Why are you jumping there?" The user may say "Because it looks like I can jump there." The facilitator might then ask "Why do you think you can jump there?" or "Why would you want to jump there?" These questions can open up alternate strategies that players may have that would not have occurred to the designers. Maybe the art should be changed so it doesn't look like you can jump there. Or maybe the designers should make it that you can jump there. What strategies would that open up? At the end of the playtest, have a survey available for them with a mix of open-ended qualitative questions and quantitative questions. Your qualitative questions should be like "What did you find most fun about the game?" "What frustrated you about the game?" By seeing what the testers write first, you will know what they value or abhor the most. These kinds of questions also serve as a catch-all, allowing testers to answer questions you never even thought to ask! Your quantitative questions should be specific so that you can compare them over testers. "On a scale of 1-5, how likely would you be to buy this game?" "On a scale of 1-5 how fun did you find Level 1? Level 2?" This way, over time you can compare changes. You implement a new boss on Level 3. How did the Level 3 scores change? They used to average a 3.0, now they average an 4.0. With these numbers, I bet you could make a pretty accurate interest curve. 6

7 When should you playtest? The general consensus is that it is never too early to playtest! The most practical time to playtest is as soon as you have a playable build. Some designers go further, playtesting analog versions of their digital game ideas. This is applicable in many situations. We did a bit of analog playtesting early at EA to very successful results. Most game makers want to wait as long as possible so that they can get their planned content in. The problem with this approach is that while you are making the game, it is always incomplete. If you wait until the game is complete to do your testing, you are too late! It's already done! What changes could you possibly make at the end besides little nudges? Early on your playtests could have stopped you from making content or mechanics that did not work, but at the end, you may have to rewrite entire sections of a game if it isn't working. The right idea is to playtest as soon as you can reasonably get useful results to avoid large amounts of rework or releasing a game that doesn't mesh well with players. Hardcore gamers, folks "in-the-know" and game designers will often give commentary in the form of solutions. "The champion's attack speed needs to go up." "The level is too long." "The main character isn't tall enough." But these solutions are solving underlying problems. Respectively: "The game feels unresponsive" "There isn't enough to do in the level." "I don't feel powerful." By figuring out the underlying problems beneath the solutions, you are able to come up with more creative approaches to solving common problems. Imagine that a publisher is viewing a build of a racing game. "The game needs more red cars," he said. A laughable publisher comment if ever there was one. But the smart designer figured out that his real problem is that the game didn't feel fast enough. The publisher related red cars with speed. He lowered the camera angle of the game. A camera closer to the ground tends to feel faster. The publisher loved it and they never had to add additional cars. The designer turned a multi-week project of adding new cars into a 10-second variable adjustment. One of the most applicable quotes about playtesting I ve ever seen is actually from a writer, Neil Gaiman: "Remember: when people tell you something's wrong or doesn't work for them, they are almost always right. When they tell you exactly what they think is wrong and how to fix it, they are almost always wrong." 7

8 Assignment: Playtests In this assignment, you will run a simulated playtest. This playtest is simulated because you will be choosing a game that already exists and is not within your control to change based on the playtest feedback. This should be fun! 1. First, choose a game that you want to use for the playtest. It s best not to choose your favorite game, but instead one that you know has some issues. This way, your playtest will be likely to have more content. This isn t an issue with real playtests, but is necessary for this assignment. 2. Choose an area of focus for the playtest. What system or mechanic are you trying to generate data for? Be sure to choose a focus that is testable. Focus your playtest on one system or mechanic: combat, movement, inventory management, skill trees, level design, puzzle design, etc. Make sure that your area of focus is noted in your playtest notes that you submit. 3. Next, choose at least three playtesters that do not have experience with the game that you have chosen. You may not be one of these three playtesters. 4. Conduct the playtest using the methods from the book and the lesson above to answer the question of the goal you established in step two. Make sure you are taking copious notes during the session because you will need them to fill out the questions below. Use the Think Aloud protocol. Ask your playtesters to talk through their issues and resist any urges to help them through parts or provide information unless they are so stuck that they cannot continue. 5. Write up a report of the playtests. You should have one report for each playtester that includes the following: The playtester s name The playtester s demographics (age, gender, etc.,) The playtester s familiarity with games in general, the genre of the chosen game and the specific game chosen. What was the length of the playtest session? A narrative summary of the playtest session. Some things to consider when writing your narrative summary: What problems did the tester identify? What items did the playtester mischaracterize? (I.e., did he/she think that a door texture was a real working door?) What elements did he/she enjoy or hate? What leading questions did you ask and what were the answers? (A leading question is the Why are you jumping there? example from above.) What design lessons can you yield from the playtest? Make sure this connects back to your area of focus, unless your playtester did not generate any useful or meaningful data, in which case you should make note of it and talk about useful data they did generate about any other systems/mechanics in the game. 6. The end of your report should include a general conclusion that analyzes and synthesizes the design lessons you learned from each separate playtest. How would you use the data you collected to make a meaningful improvement in the game s next iteration? 7. The final component of your report should be a survey. Design a survey for each 8

9 playtester to fill out. Collect the results. Each should ask at least five questions including: What was the most fun/interesting part of the game? What was the least fun/ most frustrating part of the game. The remaining three questions should focus on the goal you decided on in step two. If this is not your first time through this class, you may not reuse a game for which you playtested for this assignment previously as this would be a violation of Full Sail s academic dishonesty standards as noted in Lesson 01 and in your policy manual. Then save this document with the form LastNameFirstName-Playtests.pdf and submit through FSO by the posted due date. Your name must appear in the document as well. Grading Criteria Where possible, we strive to make grading criteria objective to avoid confusion from students and to streamline the grading process. However, some elements of coursework are bound to be conceptual and subjective. In these cases, we defer to the expertise of the course instructor to apply subjective elements from the rubric in a way in which a qualified neutral observer would agree. Line items in the rubric below marked with a octothorp (#) are elements in which this method is used to determine the student s grade. Showstoppers Showstoppers are elements that will cause the assignment to immediately receive a zero. A showstopper is one or more of the following: the work is plagiarized, the work is submitted again with few or no changes from a previous attempt, the work contains significant spelling, grammar, or usage errors such that it is difficult for the grader to understand what is provided, the work is largely the class materials without adding significant original work, the work violates the spirit of the assignment instructions to a significant degree, any other criterion forbidden by the instructor or Full Sail University. Otherwise, the student s grade is based on the following: 9

10 Superior Satisfactory Unacceptable # Area of Focus A focus was chosen for the playtest that focused the play and feedback to a specific issue. That area of focus was clearly stated in the submitted documentation. Up to 10 points. # Playtesters Student selected appropriate playtesters for the exercise who have not played the chosen game before and who match the game s target market and complexity. Up to 10 points. # Required Elements All required elements listed above are accounted for in all three playtests, accompanied by appropriate narrative or explanation when necessary. Up to 30 points. # Survey Each playtester was surveyed with the required questions and questions that helped support the goal of the playtest. These results are presented clearly in the report. Up to 20 points. # Presentation Playtest results are presented clearly and formatted well with no reasonable errors in spelling, grammar, or usage. Up to 20 points. A focus was chosen for the playtest, but that goal did not focus the play and feedback. The focus was clearly stated in the submitted documentation. Up to 5 points. Student selected appropriate playtesters, but these playtesters failed to give accurate or complete feedback. Up to 5 points. Required elements above are all accounted for, but some may be cursory, unclear, or show a violation of playtest protocol. Student may miss one of the required elements and still receive this result if the element was not crucial to the understanding of the playtest. Up to 20 points. Each playtester was surveyed with the required questions but the additional questions may have done a better job towards helping understand the goal of the playtest. Up to 15 points. Playtest results could be presented or formatted more clearly. There may be a small amount of minor spelling, grammar, or usage errors. Up to 15 points. A focus was not chosen or was not clearly stated in the documentation. 0 points. Playtesters were inappropriate or the student did not include three or more playtests with appropriate subjects. 0 points. Sufficiently egregious violations in this area may result in a zero for the entire assignment and/or may require a resubmission. 0 points. Student is missing multiple required elements. Sufficiently egregious violations in this area may result in a zero for the entire assignment and/or may require a resubmission. 0 points. Survey is missing, incomplete, or irrelevant to the goals of the playtest. There are significant issues with spelling, grammar, usage, presentation or formatting. Sufficiently egregious violations in this area may result in a zero for the entire assignment and/or may require a resubmission. 0 points. Delivery n/a Uses the correct file format(s), uses correct naming convention, submits one document, does not violate any other submission guidelines listed above. - Up to 10 points Does not meet the qualifications listed to the left. May be asked to resubmit and will receive an additional penalty of 10 points per day until the elements to the left are corrected. 0 points. 10

11 11