The Quest for Quality Maps. Dr. Parker Lowrey Senior Game Analytics Specialist (Senior Data Scientist)

Transcription

1 The Quest for Quality Maps Dr. Parker Lowrey Senior Game Analytics Specialist (Senior Data Scientist)

2 Game Analytics Team Who? We are data scientists and video game enthusiasts (~ 20 in all) Backgrounds in machine learning, mathematics, computer science, and IT. What? We use data analysis to improve games by engaging directly with the studios. Project flavor Big data and small data Investigatory and predictive work Select titles we have contributed we get into the credits! Call of Duty: Ghosts and Advanced Warfare Skylanders: Swap Force and Trap Team Call of Duty: Online Destiny

3 Enough about them.. What about me? Biochemistry undergrad.. Worked for bioinformatics startups Doctorate in Mathematics from University of Texas-Austin Postdocs at Western Univ., Yale Univ., and Univ. of Wisconsin-Madison. How did I wind up in gaming? Used Battlefield and Call of Duty games to keep in touch with friends as I moved from place to place. Get to use high-powered math and statistics on a hobby (YEA!)

4 Okay, Maps.. What Maps? Q: What do I mean by map? A: The level in an online multiplayer match in the Call of Duty franchise. No idea still? How about a video:

5 Good Maps, Why Do They Matter? Good maps keep players engaged in a First Person Shooter (FPS) over the entire lifecycle of the product. Long term engagement is important for the success of the title and the long-term success of the franchise. Over a career, a player will play a match on the same map hundreds to thousands of times.

6 This is silly.. or is it?

7

8

9

10 Stonehaven One of Call of Duty s Least Favorite Maps

11 Stonehaven Player Perspective

12 Nuketown 2025 One of COD s most beloved maps

13 Art or Science? Currently, game level creation is firmly in the hands of designers and artists Our group doesn t want to change that We just want to ensure the artists know the effect of various design decisions What do I mean about science? Statistical analysis Principles of big data and ML Quantify map aesthetics Isolate interacting features Predict ratings

14 How did we quantify maps? Objective Player data data Play-style Score per minute Killstreak usage Kill/Death ratio Global data Terrain Theme Weather Time of Day Perimeter Time Time to Traverse Map Central Feature Present Gridded data % Indoor % Traversable % Light/Shadow % Clutter % Sloped Terrain

15 How did we quantify maps? Nodemaps We convert the level to a succinct mathematical representation. Lanes = Edges Decision junctions = Nodes

16 How did we quantify maps? Nodemaps Call of Duty: Ghosts Map: Warhawk

17 How did we quantify maps? Nodemaps What do various multi-edge nodes mean? 1: End of a path 3: Fork or T of a lane 2: Obstacle in lane e.g. ladder 4: Intersection of two lanes

18 How did we quantify maps? Nodemaps Great! We ve reduced an uncountable feature set and reduced it to a finite graph. How do we analysis this? Markup Language We code this up in a markup language Run through python graph library Invariant extraction Density Low clustering Percent 4-edge nodes Minimum spanning tree High clustering

19 How did we quantify maps? Subjective data Survey s to determine player map preference (approximately 8000) It Sucks! Bad Neutral Good BFF! Bottom 2 Box (B2B) Top 2 Box (T2B) Top Box (TB) Expert survey Measures hard to quantify issues like enemy visibility Used to find proxy objective variables Example: Compared to other title maps, was this map easy to learn?

20 Modeling Techniques Big Data Applied machine learning techniques to the player-map level This allowed us to incorporate and correct for player performance and skill on the map Although 10k data points with 150 variables isn t enormous, it is setup this way to incorporate millions of records that will fallout of AW map voting. Gradient Boosted Machines» Linear based methods Techniques used to investigate drivers of preference: Tree methods Conditional Trees CART Ensemble methods Random Forests Extra Trees Conditional Forests» SVM, Logistic regression» A TON if variable selection techniques (for better or worse)

21 Modeling Techniques Small Data We want to understand how playstyle interacts with map attributes to affect map preference. What is playstyle? Sniper: a player that is mostly sedentary and kill distance long. Run N Gun: a player that is mostly running and relies on surprise flanking to rack up kills Problem: players rarely are a pure playstyle. often changes with game mode and map

22 Modeling Techniques Small Data Approach: build logistic models on player-vote data with percent of matches as each playstyle. Extrapolate to pure playstyle Apply machine learning techniques Since we are dealing with aggregates, we are reduced to a small sample size ( < 100 points) with 150 variables. Difficult to isolate and give significance of variables. Settled on ExtraTrees.

23 Did it Work? Call of Duty: Advanced Warfare Different movement capabilities Survey data Map Voting data Performance still in flux Survey is biased (not based on tenure, but instead churners) Full model of translating map voting to ratings incomplete Still, even with bias, we exceeded expectations and profound pathing information was derived.

24 Future Direction: Getting Past Survey Data Surveys cost money, map voting is free and abundant. Goal: Input Map Voting Model Output Survey Rating Model survey data to high degree of accuracy using map voting.

25 Future Direction: Immediate Actions Automate nodemap creation Explore other areas of pathing Create automatic tool for designers (rather than insights) Evaluate effect of game mode on preference..

26 Game Analytics Team YOU WANT TO WORK FOR US! Attend infamous launch parties! Your friends will be envious! Get into credits for game titles! Lots of swag! Buck the trend: SoCal rules! We have PS4 s on our desk! parker.lowrey@activision.com LinkedIn: linkedin.com/in/parkerlowrey