Demand for Commitment in Online Gaming: A Large-Scale Field Experiment

Transcription

1 Demand for Commitment in Online Gaming: A Large-Scale Field Experiment Vinci Y.C. Chow and Dan Acland University of California, Berkeley April 15th Introduction Video gaming is now the leisure activity that people spend significant amount of time on. Although there are increasing concerns that some players play more than they would like to a priori, the phenomenon is as yet poorly studied. 1 Chow (2011) studied players of a popular type of online games called Massive Multi-player Online Role-playing Game (MMORPG), but the paper suffered from three shortcomings: a small sample size, a short duration and no within-subject variation in treatment assignments. This paper presents a second experiment designed to address those issues. In this experiment we had the advantage of having a very large subject pool, but with the drawback of only being able to measure game-playing statistics. We find that percent of subjects provided with a commitment device manifested a demand for it, depending on what threshold one uses to categorize a subject as having a demand. The availability of commitment device reduces the number of long sessions while increasing that of short sessions. Lastly, players with commitment devices stay longer with the game. This paper is organized as follows. Section 2 presents the experimental design. Section 3 summarizes the data. The demand for the commitment devices is discussed in Section 4, while Section 5 explores the commitment devices effects on gameplaying behavior. Section 6 concludes the paper Evans Hall #3880, Berkeley, California vincichow@berkeley.edu. We are grateful to my advisor Stefano DellaVigna, as well as Botond Kőszegi, Ulrike Malmendier, Robert MacCoun, Matthew Rabin and Mike Urbancic for their invaluable comments and discussions. Funding is generously provided by the Russell Sage Foundation and the Institute for Business Research. All errors are ours. 1 See Chow (2011) for an overview of the literature on videogaming. 1

2 Figure 1: Interface of Wordsplay 2 Experimental Design The experiment was based on an online multi-player word game WordSplay, whose underlying programming script we were granted access to. WordSplay is an online implementation of the board game Boggle, a game in which players compete on finding the highest number of English vocabularies from a board of alphabets. Each round of the game lasts 3 minutes each, which a 45-seconds break between each round. The game is synchronized for all players, so everyone starts and ends at the same time. The popularity of the game is such that we were able to gather data on over fifty thousand subjects. Figure 1 is a screenshot of the game interface. In contrast to the World of Warcraft implementation, this experiment solely provides the subjects with commitment devices, without the additional procedures of measuring any willingness-to-pay or prediction. This allows us to implement the commitment devices as a permanent feature of the game, without disclosing to the players that they were in fact provided as part of an experiment. As such, we do not expect any behavior alterations due to players realizing they are being observed. All players in the game are included in the experiment. Subjects assigned to the control group are monitored for game-playing statistics. Subjects assigned to the treatment group are additionally provided with the commitment devices from paper 1, implemented as follows: every time a player logged into the game server and before she played her first game, Device X would pop up a dialog asking if she would like to set a limit on how many games she could play (Figure 2). If the player would prefer not to set a limit she could close the dialog by clicking on a "No Thanks!" button. A link captioned "What s this all about?" led to a web 2

3 page explaining that the provider of game was testing new features. As for Device I, during the 45-seconds break between each game, a button with the caption "Log me off after one more game!" would show up immediately below the text box where players enter vocabularies in-game (Figure 3). Clicking on the button would lead to a dialog similar to Device X, with the limit fixed at one game (Figure 4). Once a limit is reached, the server would block the player s account for a duration of one hour (Figure 5). Treatment started on February 1st, Group assignments are implemented the following way: players who sign up on or after the start of the treatment ( new players ) are randomly assigned to either the treatment group or the control group. Players who signed up before then ( existing players ) were initially assigned to the control group. In the subsequent 20 weeks, 10 percent of these players were switched the treatment group every two weeks. This switches allows us to observe within-subject variation due to treatment assignments spread across time, avoiding the possibility that a single switch might coincide with some other factors affecting game playing. To ensure a sufficiently long observationary period, we limit our study on new players to those who signed up between February 1st and May 10th, Data from January 1st, 2009 to January 31st, 2010 was used as a passive observatory period, in which all subjects were monitored for game-playing statistics. For each game we have data on the date and time of the game, the player s score and the percentage of correct words the subjects the player entered. 3 Data Summary Table I summarizes the data. From January 1st, 2009 to March 18th, In the full sample there were recorded players. The average number of players in each game was Defining a session as gameplaying with no gaps longer than 30 minutes in between, the average and median number of session played were and 3, while the average and median session length, averaged across player, were 5.7 and 4.78 respectively. The latter translate to and minutes respectively. The average (median) lifespan of a player, defined as the date she first started playing minus the date she stopped playing in our data, was days. Figure 6 plots the survival rates of new players. Players in the treatment group are less likely to quit playing, resulting in the their longer average lifespan. The hazard ratio of being in the treatment group is estimated to be under the Cox Proportional Hazard model, which is statistically insignificant (χ 2 1 = 2.22, p = ). There were 3451 new players were assigned to the treatment group, while 1675 were assigned to the control group. The average and median lifespan of new players were much 3

4 Figure 2: Interface of Device X Figure 3: Interface of Device I Figure 4: Device I Pop-up Window Figure 5: Notification after Limit is Reached 4

5 Figure 6: Kaplan-Meier Survival Estimates - New Players Kaplan-Meier Survival Estimates - New Players Days since started playing Control Treatment Figure 6 plots the probability of a subject staying with the game as a function of the number of days since she started playing. lower than that in the full sample, suggesting that long-term players made up a significant fraction of existing players. The average session length of new players were longer than existing players because players play shorter sessions as it got further out from the date they started playing. Gameplaying occured more often in weekends and working hours. 2 The large number of sessions played during the day is suggestive evidence that some players might be playing from their workplace. Consistent with the theoretical model presented earlier, players in the treatment group played more but shorter sessions. This comparison is, however, potentially misleading rather than exerting its effect through the channel described in the model, the availability of commitment devices might have caused those changes by extending a player s lifespan. For the longer a player s lifespan, the more sessions she would have played and the shorter the average session length would be. Figure 7 plots three measures of duration of play by week since the new players started playing. The distribution of time spent playing was heavily skewed towards the top while the top 5 percent players played over 60 games (3.75 hours) in the first week, the median player played less than 10 games (37.5 minutes). By the third week the median player has stopped playing, while the top 5 percent played till the 30th week. The second and third plots show that the decrease in time spent playing was due to players becoming less likely to start playing. Condition on starting to play, the length of a session did not change significantly between the first week and the 30th week. 2 Dates and time of day are represented in the time zones of each player. 5

6 TABLE I SUMMARY STATISTICS Full Sample Players Joint between Feb 1st - May 10th, 2010 Total No. of Games Total No. of Sessions Treatment Control Pr( T > t ) No. of Players Avg. Players per Game (27.41) Median 41 Avg. Lifespan (in days) (259.80) (120.54) Median Avg. No. of Sessions All Dates and Time (243.51) (102.41) (75.14) Median A Day in Monday-Friday (38.49) 5.10 (16.62) 4.04 (11.57) A Day in Saturday-Sunday (42.57) 6.87 (18.35) 6.13 (14.88) Pr( T > t ) am-5pm Weekday (135.68) (52.70) (33.05) pm-1am Weekday (100.35) (41.69) (31.57) Pr( T > t ) Avg. Session Length (in games) - unweighted All Dates and Time 7.05 (7.08) 7.05 (7.08) 7.54 (8.92) Median A Day in Monday-Friday 6.69 (6.67) 6.94 (6.94) 7.46 (8.82) A Day in Saturday-Sunday 7.14 (7.21) 7.36 (7.45) 7.77 (9.16) Pr( T > t ) am-5pm Weekday 6.41 (6.54) 6.68 (6.78) 7.08 (9.11) pm-1am Weekday 7.21 (7.01) 7.51 (7.48) 8.06 (9.04) Pr( T > t ) Standard deviations in parentheses. 6

7 Figure 7: Measurements of Duration of Play among New Players - Per Week Games Per Week Weeks since started playing Treatment p50 Treatment p90 Treatment p95 Control p50 Control p90 Control p95 Sessions Per Week Weeks since started playing Treatment p50 Treatment p90 Treatment p95 Control p50 Control p90 Control p95 Mean Session Length (in games) Weeks since started playing Treatment p50 Treatment p90 Treatment p95 Control p50 Control p90 Control p95 7

8 Figure 8: Commitment Devices Usage among New Players - Per Week Fraction of Subjects who Used a Commitment Device Device X - New Players Device I - New Players Device X - Full Sample Device I - Full Sample Weeks since started playing This figure plots throughout time the usage of each device as a percentage of the total number of treatment subjects. 4 Demand for Commitment Devices 8 plots the fraction of new players who have used the devices each week. Rate of usage started at percent, dropping to below 3 percent by the fourth week. Among the 1776 new players in the treatment group, 429 (24.15 percent) used Device X at least once, setting a median of 1 limit and an average of The median limit was 5 games (18.75 minutes) while the average was games. The large number of users who used the device once suggests that many players were perhaps simply trying out the device as a new feature of the game, while the large average limit implies that many players were not intenting to set a real limit. Nevertheless, the majority of Device X usage was effective: the number of binding limits defined as the session length being equal to the limit set was 3002, representing percent of all limits set. 452 (25.45 percent) new players used Device I at least once, setting a median of 1 limit and an average of Since Device I blocks playing one round after activation, the effective limit is the length of the session, which median was 7 and average was The much higher take up of Device I in Wordsplay in comparsion with World of Warcraft is mostly likely due to two reasons: first, the device is more prominently shown in Wordsplay a button to activate the device is on the interface in Wordsplay, versus no visual cues in World of Warcraft after the first 45 seconds. Second, there is the 45-second delay between each game in Wordsplay, which serves as a natural moment in which a player might use to decide how much longer to play. There are no such delays in World of Warcraft. Because no measurement of willingness-to-pay is being taken, demand for commitment devices in this experiment is measured solely by usage. Given that the median number of 8

9 Full Sample TABLE II-A COMMITMENT DEVICES USAGE Pre-Game Device In-Game Device Overlapping Usage No. of Users Total w/ 3 or More Uses w/ 10 or More Uses Avg. No. of Uses per User (57.68) 8.62 (33.27) 2.15 (3.81) Median Avg. Effective Limit (in games) ( ) 8.72 (7.56) 5.69 (4.74) Median Total No. of Uses No. of Binding Uses No. of Users whose Median Limit was Binding 2501 Players Joint between Feb 1st - May 10th 2010 Pre-Game Device In-Game Device Overlapping Usage No. of Users Total w/ 3 or More Uses w/ 10 or More Uses Avg. No. of Uses per User 8.52 (32.97) 6.33 (20.55) 1.97 (4.59) Median Avg. Effective Limit (in games) ( ) 8.94 (7.67) 5.26 (4.47) Median Total No. of Uses No. of Binding Uses No. of Users whose Median Limit was Binding 374 Standard deviations in parentheses. 9

10 sessions played by new players was two, we consider the fraction of players who used the commitment devices up to three times. 617, 300 and 210 subjects used a device at least once, twice and thrice, representing 34.74, and percent of the treatment group respectively. 159 new players used Both devices. As in the World of Warcraft implementation, only a small fraction of players ever set up overlapping commitments: 91 (5.12 percent) players set up such commitments over a total of 179 sessions. The model in the previous paper proposes that these players might have believed that they were vulnerable to cues. Table II-B are conditional logit regressions of commitment usage, controlling for player fixed-effects. Controlling for the number of days a subject had been in the treatment group, length of the last session the subject played is estimated to have a small negative effect on the probability of using a commitment device. This negative effect can be explained by positive serial correlation in the underlying utility of playing, which we do not observe. Last session s rank has a large negative effect on commitment usage. This is consistent with subjects projecting their last session performance on to the current session, which would result in higher expected utility and thus lower demand for commitment devices. Subjects were also less likely to use a commitment device in the weekend, but there was no significant change in usage during working hours. Finally, as seen in 8, usage drops significantly with time. 5 Effects on Duration of Play 5.1 Basic Regressions Table III contains results of OLS regressions of total hours played, total number of sessions and mean session length on treatment group assignment. Because existing players were swtiched from the control group to the treatment group gradually, only new players are included. In contrast to the estimations in the World of Warcraft experiment, subjects in the treatment group played more games in total then those in the control group versus though the difference is not statistically significant. As predicted by the theoretical model in paper 1, this increase was due to treatment subjects playing more shorter sessions. The average session length for a treatment subject was 6.13 hours as opposed to 6.80 hours for a control subject, while the average total number of sessions was versus selectively. 10

11 First 5 Weeks TABLE II-B PROBABILITY OF USING A COMMITMENT DEVICE IN A SESSION - LOGIT New Players Full Sample Last Session Length ** ** (0.006) (0.014) (0.002) (0.004) Last Session Length ** Ln(1+Days in Treat.) (0.006) (0.002) Last Session Rank ** *** (0.150) (0.226) (0.105) (0.103) Last Session Rank * *** Ln(1+Days in Treat.) (0.506) (0.071) Weekend *** *** (0.080) (0.081) (0.022) (0.023) Work Hours (0.077) (0.079) (0.026) (0.027) Ln(1+Days in *** *** Treatment) (0.042) (0.017) Constant *** *** *** *** (0.000) (0.000) (0.000) (0.000) N First 10 Weeks New Players Full Sample Last Session Length *** *** (0.005) (0.012) (0.002) (0.004) Last Session Length *** Ln(1+Days in Treat.) (0.004) (0.001) Last Session Rank *** * (0.112) (1.233) (0.095) (0.150) Last Session Rank ** Ln(1+Days in Treat.) (0.189) (0.043) Weekend * ** *** *** (0.063) (0.065) (0.018) (0.019) Work Hours (0.065) (0.068) (0.020) (0.021) Ln(1+Days in *** *** Treatment) (0.035) (0.013) Constant *** *** *** *** (0.000) (0.000) (0.000) (0.000) N *Significant at 10%, **Significant at 5%, ***Significant at 1%. Estimates are odd ratios. Standard errors in parentheses. All regressions controlled for player fixed-effects. Table II-B are conditional logit regressions of commitment usage. Each observation is one session. Last Session Length is measured in games. Last session rank is normalized between 0 and 1, 1 being first. Weekend = 1 if the session started in a weekend, = 0 otherwise. Work Hour = 1 if the session started between 9am-5pm, = 0 otherwise. 11

12 TABLE III EFFECTS OF AVAILABILITY OF COMMITMENT ON DURATION-OF-PLAY OLS Total Games Played Total Session Count Mean Session Length Treatment Dummy * *** (28.263) (3.046) (0.172) Constant *** *** *** (19.616) (1.836) (0.132) N *Significant at 10%, **Significant at 5%, ***Significant at 1%. All regressions are ordinary least-square. Robust standard errors in parentheses. Table III regresses three measures of duration-of-play on group assignments. Each observation is one subject. Treatment Dummy = 1 if the subject was assigned to the treatment group, = 0 if assigned to control group. 5.2 Distribution of Session Length Compared to the simple estimations on the differences in means, a deeper analysis into the distribution of session length reveals a finer picture. Table IV regresses the fraction of sessions of a particular length each subject played on treatment group assignment. Consistent with the use of commitment devices, subjects in the treatment group were less likely to play long sessions and more likely to play short sessions. For example, the fraction of sessions of 30 minutes or longer is as much as 6.9 percent points lower in the treatment group compared to the control group, while the fraction of sessions of 30 minutes or shorter is higher by the same amount. Table V demonstrates the same effect among new players by regressing a player s probability of playing long and short sessions in a day on treatment group assignment. Again, subjects in the treatment group were less likely to play long sessions and more likely to play short sessions, an effect that is robust to different definitions of session lengths and the addition of various controls. For example, treatment subjects were 15 to 26 percent less likely than control subjects to play a session of 30 minutes or longer, but were 17 to 36 percent more likely to play a session of 30 minutes or shorter. The effect subsides as we get further away from the day a player started playing, as indicated by the positive odds ratio of the interaction between treatment assignment and days since started playing. This is the result of players in both groups playing less throughout their lifespan. The shift in the distribution of session length is consistent with the hypothesis that the use of commitment devices has an effect on the session length as well as with the hypothesis that the availability of the devices alone make players more aware of their time inconsistency. The shift is, on the other hand, inconsistent with the theory that the devices were simply a nuisance since, if that were true, the fact that the devices do not appear in game should result in the treatment subjects playing less across all session lengths. 12

13 TABLE IV FRACTION OF SESSIONS BY LENGTH - NEW PLAYERS OLS Over 1hr Over 30min Over 15min Below 30min Below 15min Treatment Dummy *** *** *** *** *** (0.006) (0.011) (0.011) (0.013) (0.012) Constant *** *** *** *** *** (0.005) (0.009) (0.009) (0.010) (0.008) N OLS w/ Time Zone Fixed Effects Over 1hr Over 30min Over 15min Below 30min Below 15min Treatment Dummy *** *** * ** ** (0.006) (0.010) (0.010) (0.018) (0.013) Constant * (0.003) (0.005) (0.005) (0.003) (0.003) N Quantile Over 1hr Over 30min Over 15min Below 30min Below 15min Treatment Dummy *** *** * ** * (0.000) (0.011) (0.011) (0.004) (0.021) Constant *** * * (0.000) (0.009) (0.009) (0.003) (0.015) N *Significant at 10%, **Significant at 5%, ***Significant at 1%. Robust standard errors in parentheses for OLS regressions. Table IV regresses fraction of sessions by length on group assignments. Each observation is one subject. Treatment Dummy = 1 if the subject was assigned to the treatment group, = 0 if assigned to control group. 13

14 TABLE V PROBABILITY OF PLAYING A SESSION OF A PARTICULAR LENGTH PER DAY - NEW PLAYERS Logit w/ Player Random Effects Over 1hr Over 30min Over 15min Below 30min Below 15min Treatment Dummy *** *** *** *** ** (0.087) (0.063) (0.052) (0.103) (0.078) Constant *** *** *** *** *** (0.000) (0.000) (0.000) (53.959) (14.776) N Logit w/ Time Zone, Day of Week and Month Fixed Effects and Player Random Effects Over 1hr Over 30min Over 15min Below 30min Below 15min Treatment Dummy *** * ** (0.089) (0.073) (0.065) (0.105) (0.073) Constant * * ** ** (0.054) (0.045) (0.040) ( ) ( ) N Logit w/ Time Zone, Day of Week and Month Fixed Effects and Player Random Effects Over 1hr Over 30min Over 15min Below 30min Below 15min Treatment Dummy *** * *** (0.087) (0.080) (0.075) (0.133) (0.088) Days since Started *** *** *** *** *** Playing (0.008) (0.005) (0.004) (0.034) (0.027) DSSP Treatment *** *** *** *** *** (0.023) (0.015) (0.013) (0.015) (0.012) Constant (2.311) (3.773) (4.689) (1.025) (0.928) N *Significant at 10%, **Significant at 5%, ***Significant at 1%. Estimates are odd ratios. Standard errors in parentheses. Table V regresses fraction of sessions by length in each day on group assignments and how long the subject has been playing. Each observation is one subject-day. Treatment Dummy = 1 if the subject was assigned to the treatment group, = 0 if assigned to control group. DSSP x Treatment is the interaction between the treatment dummy and days since started playing. 14

15 5.3 Within Subject Effect To investigate within subject effects of the commitment devices, Table VI regresses session length on treatment group assignment for subjects whom we switched from the control group to the treatment group. Controlling for player and time fixed effects, subjects played on average 49 percent less on the first session they have access to the commitment devices. As before, the effect subsides as we get further away from the day the player started playing. 6 Conclusion This paper presents a second experiment desgined to address the shortcomings of the World of Warcraft experiment, namely a small sample size, a short duration and no within-subject variation in treatment assignments. In this experiment we had the a subject count of over fifty thousand. We find that percent of subjects provided with a commitment device manifested a demand for it, depending on what threshold one uses to categorize a subject as having a demand. The availability of commitment device reduces the number of long sessions while increasing that of short sessions. Lastly, players with commitment devices stay longer with the game. These results imply that a significant fraction of online gameplayers overplay according to their own assessments, leading them to set limits on the amount of time their future selves can play. References Chow, Yan Chi Vinci, Demand for a Commitment Device in Online Gaming, Working Paper, March

16 TABLE VI SESSION LENGTH WITHIN SUBJECT FULL SAMPLE Session Length Ln(1+Session Length) (1) (2) (3) (1) (2) (3) Treatment Dummy *** Treatment Dummy *** *** *** (0.051) (0.303) (0.302) (0.005) (0.098) (0.097) Days since Started *** *** Ln(1+Days since *** *** Playing (0.001) (0.001) Started Playing) (0.003) (0.003) DSSP Treatment Ln(1+DSSP) *** *** (0.001) (0.001) x Treatment (0.015) (0.015) DSSP *** *** (0.000) (0.000) Last Session Rank *** Ln(1+Last Session *** (0.090) Rank) (0.013) Last Session No *** Ln(1+Last Session *** of Players (0.001) No.of Players) (0.004) Constant *** *** *** Constant *** *** *** (0.019) (0.157) (0.178) (0.000) (0.000) (0.000) N N *Significant at 10%, **Significant at 5%, ***Significant at 1%. All regressions are ordinary least-square with player, hour of day, day of week and month fixed effects. Clustered standard errors in parentheses. Table VI regresses session length on treatment group assignment for subjects who were switched over from the control group to the treatment group. Each observation is one session. Treatment Dummy = 1 if the subject was assigned to the treatment group, = 0 if assigned to control group. DSSP x Treatment is the interaction between the treatment dummy and days since started playing. Last Session Rank is the 16