Coding as a Game for Contests. Judith Bishop

Transcription

1 Coding as a Game for Contests Judith Bishop

2 Take aways 1. Scale of the data 2. Keeping players engaged 3. Different audiences 4. Analytics 5. Responsibility to players 6. Ask for collaborators

3 Audiences A.Coders: (enthusiasts, programmers at work, Microsoft) want continuously new puzzles that produce reliable and understandable scores for submitted solutions. B.Students: (K-12 to graduate) want to proceed through a sequence on puzzles that helps them learn and practice, and to get help when stuck. They also want A. C.Teachers: (K12 and university) want to easily set up universes tailored to specific needs that exercise different parts of a curriculum, and to track students progress and get results in a readable form. D.Contest Organizers want A and C as well a high degree of uptime and security. These organizers are both within Microsoft and outside. E.Researchers want access to the Azure data for mining how people code and learn, as well as hooks for adding new tools and APIs. At the moment, Code Hunt is delivering on B and is reliable and scalable. We are running at times on over 40 cores with 56% usage (see ), scaling fully automatically as needed depending on current user demand.

4 Code Hunt: A Game for Coding For individuals (K12, introductory courses, geeks) For competitions at any level, world-wide or in house Based on long-term research on symbolic program analysis (Pex, Z3) Works with Java and C# Runs in any modern browser Now working on tablets and phones

5 Code Hunt Usage Code Hunt has had several hundred thousands of users since launch in March 2014 Stats from Visual Studio Analytics over the period May 22-June 26 indicate 40,235 users Stickiness (loyalty) is very high

6

7 May and June July and August

8

9

10

11 Survey results (735 respondents) How much did the puzzle aspect of Code Hunt keep you interested in reaching a solution? In your opinion, were your final solutions well-structured code? We have many other statistics, but not so relevant to contests

12 The Hint System Line hints Look at line 4 to capture the code Positive recommendation hints: You may find a loop useful at this level Negative recommendation hints: The expression is rarely used to solve this level

13 Features that are identified loops (or LINQ) contains any loops or LINQ expressions nested loops (or LINQ) contains a loop that contains either a LINQ expression or another loop if statements contains any if statements booleans return type is boolean (wanted to distinguish from loop/if conditions) string contains expressions of type string string operations by character string and either indexes strings by character or contains expressions of type char[] string operations by substring uses string.join(), string.split() or string.substring(), regular expressions uses regular expressions arithmetic operators uses -, /, *, or % (note + is not included because it is so common, e.g., in loop increment) bitwise operators uses, &, or ^ multidimensional arrays contains expressions with a multidimensional array type (e.g. char[][] or int[,,]) array contains expressions with an array type arithmetic involving arrays array + loops (or LINQ) + arithmetic operators array manipulation array + loops (or LINQ) +not arithmetic operators

14 Features used by puzzles plain were used in many high scoring solutions (so they are likely to be essential to some solution); in parens were used in many lower scoring solutions so they may be only for less concise approaches; surrounded by **s were used in few unusual but high scoring solutions APCS-Loops-L 5422 arithmetic involving arrays arithmetic operators APCS-Loops-L 5082 array manipulation arrays APCS-Loops-L 4465 (loops (or LINQ)) arithmetic operators APCS-Loops-L 3932 (if statements) (loops (or LINQ))

15 Are hints useful?

16

17 Contests Microsoft has a demand for contests to help identify top coders to make its major competitions more fun 2,353 players 350 top players 41.0 average tries per level 7.6 average tries per level

18 Creating new contests Creating new puzzles Requires curation of a puzzle bank Original data about each puzzle Group numbers, arrays, strings, bools, binary Subjective difficulty Source who wrote the puzzle Features Each contest should have a sequence of sectors in increasing difficulty Avoid Bad puzzles early on those that fool users

19 Leaderboard and Dashboard Visible only to the organizer Publically visible, updated during the contest

20

21 Automatic Contest Designer Selects puzzles from the bank based on parameters. The contest can be customized by the Contest Designer.

22 Updating the puzzle bank statistics Updating the used field Modifying the difficulty rating based on user experience Options Score but the score is 1-3 and we know that 77% of users improve their code to get a 3 Tries a fairly objective reflection of how long it took to find the pattern and program a correct solution CAVEAT!!! Users in areas with poor internet are known to use the Capture Code button less

23 Players Players Code Hunt - the APCS (default) Zone Opened in March problems covering the Advanced Placement Computer Science course So far, over 45,000 users started APCS Zone, First three sectors, 45K to 1K Sector and Level APCS Players, Sectors 4 to 14, 1.3K to 110 Sector and Level

24 Early effect of difficulty on drop off rate Sector and level number Average tries for Winners Percentage drop off from previous puzzle

25 Formula for perceived difficulty For a particular puzzle a + b * tries + c * tries * distance a = 1, b = 0.05, c = 0.02, or for APCS c = tries are the average tries for all winners distance is the number of levels so far Examples Players Tries Sector.Level D = / * 29 / 1000 = 1.74 Original difficulty was 2 D= / * 69 / 1000 = 6.36 Original difficulty was Players Tries Sector.Level

26 How players perceive difficulty BoP China (same community) a=1, b=0.05, c=0.02 Subjective difficulty Perceived difficulty Starting players Ending players Levels with average tries over 10 BoPQuali out of 17 = 18% BoPPrelimA out of 6 = 50% BoPPrelimB out of 6 = 33% BoPSemi out of 10 = 20% CSTA and TEALS (identical contests) a=1, b=0.05, c=0.02 Subjective difficulty Perceived difficulty Starting players Ending players Levels with average tries over 10 TEALS (students) out of 23 = 22% CSTA (teachers) out of 23 = 30%

27 continued ICSE 2010 (five day contest) a=1, b=0.03, c=0.01 Subjective difficulty Perceived difficulty Starting players Ending players Levels with average tries over 10 ICSE ( grad students) out of 53 = 28% APCS (open zone) a=1, b=0.05, c=0.001 Subjective difficulty Perceived difficulty Starting players in Sector 1 Ending players Levels with average tries over 10 APCS out of 129 = 40%

28 Collaborators wanted Thank You!

29 Appendix Slides

30

31

32

33

34

35

36 Code Puzzle Bank Manager Code Puzzle Creator Code Puzzle Checker Contestant ACTORS Game Website Contest Designer CodeHunt Management Portal CodeHunt Backend API Services Code Hunt Cloud Contest Organizer Management Portal user Code Hunt Website user Portal Admin Contest Manager Zones Manager Contest Validator CodeHunt API service user

37 Planned Developments Technical enhancements Community engagement Catalog tool for puzzles Editing tool for universes Automatic testing tool for new universes Universe management tool Crowdsourcing puzzle creation Live feed showing active game play Website integrated dashboard Content testing of Office Mix plug in Maintaining the Java translator Support for user-defined types (objects) Management tool for data access Plug-in infrastructure for new APIs New content semi-annually Working with contest Organizers Building a research community Building a user community Collecting usage statistics and answering bug reports Some of these can be done by stakeholders/partners