Games and Big Data: A Scalable Multi-Dimensional Churn Prediction Model

Transcription

1 Games and Big Data: A Scalable Multi-Dimensional Churn Prediction Model Paul Bertens, Anna Guitart and África Periáñez (Silicon Studio) CIG 2017 New York 23rd August 2017

2 Who are we? Game studio and graphics middleware company based in Tokyo (spin-off of Silicon Graphics) YOKOZUNA data: Research unit of Game Data Science providing individual player predictions to game studios Goals: predict player behavior, scale to big data and provide an intuitive result visualization 2

3 Churn prediction in Free-To-Play games This is all about churn When a player is going to exit the game? In terms of days1,2, level, played hours We focus on the top spenders: whales less than 2% of the players, 50 % of the revenue General model that adapts to diverse games and datasets we define churn as 10 days of inactivity (coming back only 1% revenue) the definition of churn in F2P games is not straightforward Parallelizable algorithm applied in a production environment scalable to Big Data up to tens of millions MAU 1) 2) Rothenbuehler J. et al., Hidden markov models for churn prediction. Periáñez A. et al., Churn prediction in mobile social games: towards a complete assessment using survival ensembles 3

4 The model: Survival Ensembles 4

5 Challenge: modeling churn Survival analysis focuses on predicting the time-to-event, e.g. churn when a player will stop playing? Classical methods, like regressions, are appropriate when all players have left the game Censoring Problem: dataset with incomplete churning information Censoring is the nature of churn Survival analysis is used in biology and medicine to deal with this problem Ensemble learning techniques provide high-class prediction results 5

6 Challenge: modeling churn Two approaches: Churn as a binary classification Churn as a censored data problem One model: Conditional Inference Survival Ensembles2 deals with censoring high accuracy due to ensemble learning Survival Analysis Survival analysis methods (e.g. Cox regression3) do not follow any particular statistical distribution: fitted from data Fixed link between output and features: efforts to model selection and evaluation 2) 3) Hothorn T. et al., Unbiased recursive partitioning: A conditional inference framework. Cox. D.R., Regression Models and Life-Tables. 6

7 Conditional inference survival ensembles Survival Tree Conditional Survival Ensembles Split the feature space recursively Make use of hundreds of trees Based on survival statistical criterion the root node is divided in two daughter nodes Outstanding predictions Conditional inference survival ensemble use a Kaplan-Meier function as splitting criterion Robust information about variable importance Overfit is not present Not biased approach Maximize the survival difference between nodes A single tree produces instability predictions 7

8 Conditional inference survival ensembles Two steps algorithm: 1) the optimal split variable is selected: association between covariates and response 2) the optimal split point is determined by comparing two-sample linear statistics for all possible partitions of the split variable Random Survival Forest4 RSF is based on original random forest algorithm5 RSF favors variables with many possible split points over variables with fewer 4) 5) Ishwaran H. et. al, Random Survival Forests. Breiman L. et. al, Random Forests. 8

9 Kaplan-Meier estimates6 Cumulative survival probability Step function that changes every time that a player churns Output in terms of level and playtime (hours played) 6) Kaplan E. L. et. al., Non-parametric estimation from incomplete observations. 9

10 Results Conditional Inference Survival Ensembles 10

11 Features selection Daily logins, purchases, playtime and level-ups player attention: information per day (e.g.playtime per day) player loyalty: mean over several different time periods time elapsed until first and last day to information (e.g. time from last purchase) player intensity: total amount (e.g. total in-app purchases) player level (concept common to most games) 11

12 Features selection Daily logins, purchases, playtime and level-ups player attention: information per day (e.g.playtime per day) player loyalty: mean over several different time periods time elapsed until first and last day to information (e.g. time from last purchase) player intensity: total amount (e.g. total in-app purchases) RPG free-to-play game Action battle card game popular in Japan Long-term loyal players player level (concept common to most games) 12

13 Censored data problem results Predicted Kaplan-Meier survival curves as a function of playtime (hours) and level for new or existing players 13

14 Validation -- Churn prediction Survival Ensembles median survival level, i.e. level when the percentage of surviving in the game is 50% Cox Regression 14

15 Validation -- Churn prediction Survival Ensembles Cox Regression median survival playtime, i.e. number of played hours when the percentage of surviving in the game is 50% 15

16 Validation -- Churn prediction Model IBS7 Survival Ensemble Cox Regression Kaplan Meier bootstrap cross-validation error curves for the survival ensemble model and Cox regression 7) Graf E.. et. al, Assessment and comparison of prognostic classification schemes for survival data. 16

17 Validation -- Churn prediction Model IBS7 Survival Ensemble Cox Regression Kaplan Meier bootstrap cross-validation error curves for the survival ensemble model and Cox regression 7) Graf E.. et. al, Assessment and comparison of prognostic classification schemes for survival data. 17

18 Summary and conclusion Application of state-of-the-art algorithm conditional inference survival ensembles to predict churn and survival probability of players in social games median survival time, i.e. time when the percentage of surviving in the game is 50%, can be used as a time threshold to categorize a player in the risk of churning Model able to make predictions every day in an operational environment Adapts to other game data: Democratizing Game Data Science YOKOZUNA data It does not require previous manipulation of the data It is able to deal efficiently with the temporary dimension It can be parallelized It not only outputs churn information but also variable importance 18

19 THANK YOU yokozunadata.com 19