Automatic Level Difficulty Adjustment in Platform Games Using Genetic Algorithm Based Methodology

Transcription

1 Automatc Level Dffculty Adjustment n Platform Games Usng Genetc Algorthm Based Methodology Nrach Watcharasatharornong Deartment of Comuter Engneerng Faculty of Engneerng Chulalongkorn Unversty Payatha Road, Patumwan Bangkok 10330, Thaland Tel notee@gmal.com Vshnu Kotrajaras Deartment of Comuter Engneerng Faculty of Engneerng Chulalongkorn Unversty Payatha Road, Patumwan Bangkok 10330, Thaland Tel vshnu@c.eng.chula.ac.th, ajarntoe@gmal.com Abstract In latform games, enemy behavor s not comlcated. Therefore, challenges n such games come from the rght mxture between enemes and envronments of each level. Platform games requre manual testng for tunng the game balance for mass audence. Ths s very tme consumng. In addton, the dffculty of each level obtaned s not guaranteed to sut ndvduals. Very few researches tackle how balanced levels can be generated automatcally for ndvduals. Ths aer rooses a new methodology for usng artfcal ntellgence to adjust games dffculty to sut layers by automatcally generatng levels n latform games. The method s nsred by genetc algorthm. It s much easer to mlement comared to an exstng renforcement learnng based method, whle stll mantans smlar gamelay qualty. The new methodology also consumes less memory. Keywords Level desgn, latform game, genetc algorthm, automatc dffculty adjustment 1. Introducton There are many researches that tackle the ssue of dffculty adjustments n games. Most of them concentrate on enemy behavor adjustment. However, for latform games, ther challenges come from learnng to overcome obstacles resented by fxed enemes and game envronments. Tunng the dffculty of latform games by adjustng the behavor of enemes wll smly destroy the mechanc of such games. An alternatve aroach for dffculty adjustment s to fne-tune the layout of game stages (ncludng the lacement of fxed behavor enemes). Kamnerdnond and Kotrajaras [1] roosed a model for automatcally generatng game envronments accordng to layers erformances for latform games. The model combned renforcement learnng wth desgn methodologes. Ther model, however, requred a lot of memory storage because data for ndvdual challenge lays and vote records from all revous challenges were needed durng lay.

2 In ths aer, we resent an alternatve model for latform games level generaton accordng to layers' erformances. We abandoned renforcement learnng aroach and oted for an aroach nsred by genetc algorthm. Ths aroach resulted n less memory usage whle stll allowed levels to be roduced effectvely. Our rototye was made after Suer Maro. We beleved that by utlzng a well-known game mechansm, we would be able to demonstrate our model more clearly. 2. Related Works Artfcal ntellgence alcatons for comuter games can be groued nto two categores. The frst category strves for the best ossble agent behavour. Genetc algorthm [2] and renforcement learnng [3] are rme examles of alcatons n ths category. Bakkes et al. [4] created a team based AI for Quake III by usng genetc algorthm to learn state-secfc behavor for the team. Cole et al. [5] used genetc algorthm to evolve sets of arameters for bots n Counter Strke. Genetc algorthm was able to tune arameters as good as a hghly exerenced layer could do n ffty generatons, whch was a relatvely short tme for tranng bots offlne. Graeel et al. [6] used renforcement learnng to tune a fghtng game AI character. Sronck et al. [7] ntroduced dynamc scrtng, a form of renforcement learnng that could adjust an AI to wn aganst ts oonent n a relatvely short tme. The second category of artfcal ntellgence alcatons ams to adat agents to sut layers. Sronck et al. [7] demonstrated that dynamc scrtng could be enhanced so that the game AI was able to scale ts dffculty level to match ts human oonent. Andrade et al. [8] aled renforcement learnng to match layers erformances wth those of agents. All these works concentrated on changng characters or agents behavor. For latform games, makng an enemy character adat or learn new behavor s not qute arorate because the dffculty of latform games comes manly from game envronments and obstacles, not from enemy characters alone. Therefore the adjustment should be aled to the game envronments nstead. Pagulayan et al. [9] roosed a method for desgnng game envronments to sut layers. Challenges were ut nto each game level accordng to ther dffculty. The am was to teach collectons of sklls to layers gradually. Players would then be able to mrove ther sklls n order to tackle more dffcult challenges and defeat game bosses. Björk and Holoanen [10] roosed that a game should have mechansms for smoothng layers learnng curve n order to rovde layers wth enough sklls to rogress whle reventng boredom. However, these works manly dscussed good ractce for manually desgnng game levels. For automatc level generaton of latform games, Kamnerdnond and Kotrajaras [1] used desgn methodologes from [9] and [10] together wth renforcement learnng to create each sutable level for layers to overcome. A level was formed from several challenges. Each challenge conssted of a sequence of contnuous actons (layers could not take a break whle erformng such acton sequence). Each ossble acton was derved from layers control sklls. Whle layng a generated game level, a layer s erformance was recorded. After the layer fnshed each game level, the collected data was used as feedback to calculate the robablty for each challenge to emerge n the

3 next level. Intally, the system generated all ossble challenges that dd not contan more than a certan number of actons. Each challenge had ts own dffculty score. All challenges were then dvded nto grous. Wthn each grou, challenges were sorted by ther dffculty score from low to hgh. After a layer fnshed layng a game level, the number of successful and unsuccessful lays for each challenge was gven to the renforcement learnng mechansm. A votng system was used. Each challenge could cast votes for more dffcult or less dffcult challenges of the same grou. The sread of the votng range was determned by the lay data. The robablty for a challenge to be selected for the next level ncreased accordng to ts votng score and decreased accordng to the number of tmes the layer cleared that challenge (to revent the challenges from beng selected too often). For each challenge grou, a certan number of challenges were chosen ths way to construct the next level. Ther system gave good exermental results. However, t consumed a lot of memory. Our aer resents an alternatve level generaton algorthm accordng to layers erformance, wth less memory usage. 3. Our Aroach We utlzed a crossover-lke mechansm to create new challenges n the next level, keeng them smlar to revous challenges. A challenge was created based on sklls we wanted layers to learn. The grah n the mddle of fgure 1 shows all ossble contnuous acton sequences n a Suer Marolke game. Acton F reresents the skll of throwng a freball. Acton M reresents runnng, whle E, J, and A reresent avodng enemes, jumng, and stomng on an enemy resectvely. Each acton can have varyng dffculty scores deendng on ts target. For examle, jumng to a small latform has hgher scores than jumng to a large latform. A challenge s a sequence of these actons, as shown on the rght of fgure 1. In our aroach, smlar challenges were groued. The dffculty of each challenge could be calculated from equaton (1). h d ( a ) d ( a ) d ( a )]*[ n ( a )] df a A j E e j [ (1) ; n ( a ) 1 a a Jum k M m k Where h df s the overall dffculty score of the challenge. a s a layer's acton. d s the dffculty score based on the layer's skll for the gven acton. d e s the dffculty score of the enemy nvolved n the layer s acton. d m s the dffculty score of the ma object used durng the layer s acton. n s the number of tmes the layer s acton s reeated contnuously. Currently, only jumng generates the score. Otherwse, t s 1. a j s the roerty of the enemy nvolved wth the layer's acton. a k s the roerty of the ma object used durng the layer's acton.

4 A E s the set of basc actons that can be carred out by the layer. s the set of enemes roertes. M s the set of ma objects' roerty. A chromosome reresents one challenge. A level can have any number of challenges from a challenge grou. In our rototye, a level conssted of two challenges from each challenge grou. At the frst level, chromosomes were created so that each of ts actons dd not exceed ther default dffcultes values. For each challenge grou, ts Challenge Rank score (CR) for a layer could be calculated from the layer s erformances durng lay. The value of CR for a grou of challenges was used to calculate the change n dffculty score for the challenges of that grou n the next level. Equaton (2) - (4) show how CR was calculated. PT 2 ST 1 RL (2) 2 RL 1 PR (3) CR PT PR (4) Where PT s the number of tmes the challenges n the grou were attemted. ST s the number of tmes the challenges n the grou were overcome. RL reresents Rank Level. It s the number of character used for the challenges n the grou over several lays. Its smallest value s 1, whch haens when the layer character dd not de at all. PR s Play Rank. It s actually RL rescaled n order to calculate CR. CR s Challenge Rank. It s transformed from RL so that scores are gven more to the number of successful lays. Fgure 1. Challenge Generaton Fgure 2. Chromosomes contan actons and ther dffculty scores

5 In our rototye, after a 5 th level was layed, PT and ST only counted the last three levels. A chromosome conssted of a strng of acton tyes and ther corresondng dffculty scores. Fgure 2 shows two chromosomes. Both were from the same challenge grou, JJ (ths grou contaned challenges that started wth two consecutve jums). 3.1 Chromosome Prearaton We roduced extra chromosomes to be used for crossover wth orgnal chromosomes. Each challenge grou was used to generate a number of extra chromosomes. In our rototye, the number of generated chromosomes was the same as the number of orgnal chromosomes. The CR value from a challenge grou was used to construct extra chromosomes for that grou. If CR >0, a newly roduced chromosome should have a hgher dffculty score than ts source. However, the dfference n scores should not exceed a factor of CR (such factor could be adjusted). If CR < 0, the new chromosome should have a lower dffculty score than ts source, but the dfference should not exceed the value of CR. For each challenge grou, extra chromosomes were roduced accordng to the followng stes. 1. The frst few elements n the chromosome that dentfed the grou were generated accordng to that grou s dentty. 2. Then a. If CR >0, a legal acton was randomly added to the chromosome. b. Else If CR <0, the new chromosome bult so far was comared wth ts source. If the dffculty score of the new chromosome was less than ts source, an acton would be added to the chromosome. 3. The chromosome bult so far was then checked to see f ts dffculty score matched the value that was needed. Equaton 6 and 7 were utlzed. Where h df h n h (6) df n df LF CL CR (7) h df n s the dffculty score of the chromosome bult so far. h df LF CL s the dffculty score of the chromosome used as source. s Learnng Factor. s Coeffcent of Learnng. It s a value used to scale the CR value. CR n s the dffculty level the current layer could overcome. If the challenge grou CR value of the current level s greater than 0, the value of CR n wll be between the CR value of the revous level and the CR value of the current level. If the challenge grou CR value of the current level s less than 0, the value of equal to the CR value of the current level. CR n wll be

6 Then a. If h df < LF, the new dffculty score had not reached the value sutable for the layer. Ste 2 was then revsted. b. If h df > LF, the new dffculty score was too hgh. The dffculty score of every acton n the chromosome was checked. If every acton had ts least ossble score, nothng would be done and the algorthm roceeded to the crossover. Otherwse, an acton that had a lower dffculty score and was stuated nearest to the end of the chromosome was chosen. Its dffculty score was then reduced. Then ste 3 was reeated agan. c. If h df = LF, the layer should be able to lay the new challenge and challenges generated from t. The crossover was erformed next. 3.2 Crossover Our crossover dffered slghtly from standard Unform crossover. Parts of the chromosomes whch dentfed ther challenge grous were not modfed. Furthermore, each resultng chromosome from our aroach needed to be checked for correct contnuous actons (see fgure 1). After the crossover, chromosomes were selected from the results. The chosen chromosomes would become the challenges of the next level. In our rototye, we selected chromosomes wth the value h df LF between 0 and 1. The reason we had to allow other values aart from 0 was because there mght not be any chromosome wth LF 0 h df at all after the crossover. We selected the ones wth h df LF nearest to 0 frst. If more than one chromosome had equal marks, ther order was randomly chosen. 4. Testng and Results Twelve testers were asked to lay our game twce. In the frst lay, the game utlzed Kamnerdnond s level generaton methodology. In the second lay, our level generaton technque was aled. A tester cleared 20 levels for each lay. Durng each layer s sesson, the dffculty score and the number of lves the layer sent for each challenge were recorded. The memory usage data for each level was also collected. After fnshng both games, each layer was asked about how he felt when layng each game and how the game dffculty changed durng lay. We need the followng model behavor. Frst, when a layer sent many lves overcomng a challenge, challenges of the same grou n the next level must become easer (but not too easy). Second, when a layer sent no lfe or very few lves overcomng a challenge, challenges of the same grou n the next level must become harder (but not too hard). Due to lmted sace, we cannot show results obtaned from every layer. However, all the layers results were very smlar. Fgure 3 shows the average dffculty score of each challenge grou for one

7 of the layers durng hs 20-level-lay of our model. Lves sent by the layer n fgure 3 are dslayed n fgure 4. Table 1 shows each challenge and lves sent to overcome t by the same layer. From the fgure 3, 4 and table 1, t can be seen that when a layer sent many lves for a grou of challenges n a sngle level or sent some lves for the same challenge grou over consecutve levels, the challenge dffculty score for that grou tended to go down n the next level. On the other hand, f the layer rarely ded, the challenge dffculty score for that grou tended to go u quckly. Only very few challenges dd not follow ths behavor. The layers onons suort ths concluson. Ten out of twelve layers (83.33%) felt that after they encountered very dffcult challenges n a level, the next level became easer. Eght out of twelve layers (66.67%) felt that after they layed a very easy level, the next level became more dffcult. All the results ndcate that our roosed model can effectvely adjust the game dffculty level accordng to layers erformances. Fgure 5 dslays the average memory usage of each level from Kamnerdnond s model and our model, collected from 12 testers, each layng 20 levels for each model. Usng the ared t-test, t was found that the two-taled P value was less than By conventonal crtera, ths dfference was consdered to be extremely statstcally sgnfcant. The dfference between the mean of Kamnerdnond s model and our model equaled The 95% confdence nterval of ths dfference was from to The ntermedate values used n calculatons ncluded t = , df = 19 and standard error of dfference = Therefore, t can be statstcally shown that our new model has better memory utlzaton. hdf Average hdf Level avg g1 avg g2 avg g3 avg g4 avg g5 avg g6 Fgure 3. Average level dffculty score of layer A for each challenge grou 12 Lfe use 10 Lfe use g1 g2 g3 g4 g5 g Level Fgure 4. Lves sent by layer A for each challenge grou

8 Table 1. Lves sent by layer A for each challenge n grou 3 Memory used ( byte ) Byte Level old model our model Fgure 5. Average memory usage Concluson Our man contrbuton was the new level generaton methodology for latform games nsred by genetc algorthm. Levels generated wth our methodology had ther dffculty that suted each layer s skll. The model also had lower memory usage comared to the renforcement learnng aroach. There was some roblem wth the random nature of crossover. Sometmes a crossover dd not roduce any good results. However, ths was very rare. References [1] Charya Kamnerdnond and Vshnu Kotrajaras. Automatc Level Dffculty Adjustment n Platform Games Based on Player s Performance: Suer Maro Case Study, In Proceedngs of the 11 th Natonal Comuter Scence and Engneerng Conference, Bangkok, Thaland. : [2] Tom M. Mtchell. Machne Learnng, McGraw Hll, Internatonal Edton 1997, ISBN: [3] Rchard S. Sutton and Andrew G. Barto. Renforcement Learnng (An Introducton), The MIT Press, Cambrdge, Massachusetts, London, England, 1998, ISBN: [4] Sander Bakkes, Peter Sronck, and Erc Postma. TEAM: The Team-orented Evolutonary Adatablty Mechansm, In Matthas Rauterberg, edtor, Entertanment Comutng - ICEC 2004, Lecture Notes n Comuter Scence, Srnger-Verlag, 3166: [5] Ncholas Cole, Sush J. Lous, and Chrs Mles. Usng a Genetc Algorthm to Tune Frst-Person Shooter Bots, Congress on Evolutonary Comutaton : [6] Thore Graeel, Ralf Herbrch, and Julan Gold. Learnng to Fght. In Proceedngs of the Internatonal Conference on Comuter Games: Artfcal Intellgence, Desgn and Educaton. : [7] Peter Sronck, Marc Ponsen, Ida Srnkhuzen-Kuyer, and Erc Postma. Adatve Game AI wth Dynamc Scrtng. Machne Learnng. 63(3): [8] Gustavo Andrade, Geber Ramalho, Hugo Santana, and Vncent Corruble. Challenge-Senstve Acton Selecton: an Alcaton to Game Balancng, In Proceedngs of the 2005 IEEE/WIC/ACM Internatonal Conference on Intellgent Agent Technology (IAT 05). : [9] Randy J. Pagulayan, Kevn Keeker, Denns Wxon, Ramon L. Romero, and Thomas Fuller. User-centered Desgn n Games, Handbook for Human-Comuter Interacton n Interactve Systems, Mcrosoft Cororaton, Mahwah, NJ: Lawrence Erlbaum Assocates, Inc [10] Staffan Björk and Juss Holoanen. Patterns n Game Desgn, Charles Rver Meda, Inc., 2005, ISBN: