Article Modified Adversarial Hierarchical Task Network Planning in Real-Time Strategy Games

Transcription

1 Article Modified Adversarial Hierarchical Task Network Plaig i Real-Time Strategy Games Li Su, Peg Jiao, Kai Xu, Quaju Yi * ad Yabig Zha College of Iformatio System ad Maagemet, Natioal Uiversity of Defese Techology, Chagsha , Chia; mksl163@udt.edu.c (L.S.); crocus201@163.com (P.J.); xukai09@udt.edu.c (K.X.); ybzha@udt.edu.c (Y.Z.) * Correspodece: yiquaju@udt.edu.c; Tel.: Received: 31 July 2017; Accepted: 20 August 2017; Published: 25 August 2017 Abstract: The applicatio of artificial itelligece (AI) to real-time strategy (RTS) games icludes cosiderable challeges due to the very large state spaces ad brachig factors, limited decisio times, ad dyamic adversarial eviromets ivolved. To address these challeges, hierarchical task etwork (HTN) plaig has bee exteded to develop a method deoted as adversarial HTN (AHTN), ad this method has achieved favorable performace. However, the HTN descriptio employed caot express complex relatioships amog tasks ad accommodate the impacts of the eviromet o tasks. Moreover, AHTN caot address task failures durig pla executio. Therefore, this paper proposes a modified AHTN plaig algorithm with failed task repair fuctioality deoted as AHTN-R. The algorithm exteds the HTN descriptio by itroducig three additioal elemets: essetial task, phase, ad exit coditio. If ay task fails durig pla executio, the AHTN-R algorithm idetifies ad termiates all affected tasks accordig to the exteded HTN descriptio, ad applies a ovel task repair strategy based o a prioritized listig of alterative plas to maitai the validity of the previous pla. I the plaig process, AHTN-R geerates the priorities of alterative plas by sortig all odes of the game search tree accordig to their primary features. Fially, empirical results are preseted based o a µrts game, ad the performace of AHTN-R is compared to that of AHTN ad to the performaces of other state-of-the-art search algorithms developed for RTS games. Keywords: HTN plaig; real-time strategy game; task repair 1. Itroductio The applicatio of artificial itelligece (AI) to real-time strategy (RTS) games icludes cosiderable challeges due to the very large state spaces ad brachig factors, limited decisio times, ad dyamic adversarial eviromets ivolved [1]. RTS games are regarded as a simplificatio of real-life eviromets, ad could therefore serve as a test bed for ivestigatig activities such as real-time adversarial plaig ad decisio makig uder ucertaity [2]. Moreover, task plaig techiques that have bee demostrated to be effective i RTS games could also be applied i real-world domais [2]. Compared with covetioal board games, RTS games have the followig primary differeces [2]: 1. Players ca pursue actios simultaeously with the actios of other players, ad eed ot take turs, as i games such as chess. Moreover, player actios ca be coducted over very short decisio times, allowig for rapid sequeces of actios. 2. Players ca pursue cocurret actios employig multiple cotrollable uits. This is much more complex tha covetioal board games, where oly a sigle actio is performed with each tur. Appl. Sci. 2017, 7, 872; doi: /app

2 Appl. Sci. 2017, 7, of Player actios are durative, i that a actio requires umerous time steps to be executed. This is also much more complex tha covetioal board games, where actios must be completed withi a sigle tur. 4. The state space ad brach factors are typically very large. For example, a typical map i StarCraft geerally icludes about 400 player cotrollable uits. Cosiderig oly the locatio of each uit, the umber of possible states is about , whereas the state space of chess is typically estimated to be aroud Of course, eve larger values are obtaied whe icludig the other factors i the RTS game. 5. The eviromet of a RTS game is dyamic. Ulike i covetioal board games such as chess, where the fudametal atures of the board ad game pieces ever chage, the eviromet of a RTS game may chage drastically i respose to player decisios, which ca ivalidate a geerated pla. Because of the differeces discussed above, stadard game tree search methods, which perform well for board games like alpha beta search [3], caot be directly applied i RTS games [4]. However, research has bee coducted to modify game tree search methods to address these differeces [4]. Chug et al. [5] ivestigated the applicability of game tree Mote Carlo simulatios i RTS games. Balla ad Fer [6] applied the upper cofidece boud for trees (UCT) algorithm i a RTS game to address the complicatios associated with durative actios. The UCT algorithm is a variat of Mote Carlo tree search (MCTS) that modifies the strategy of explorig child odes. The method is believed to adapt automatically to the effective smoothess of the tree. Churchill et al. [7] addressed the complicatios associated with simultaeous ad durative actios by extedig the alpha beta search process. However, the large brachig factors caused by idepedetly actig objects still remaied a substatial challege. Methods such as combiatorial multi-armed badits have attempted to address the challege of large brachig factors [8,9]. Otañó ad Buro [10] addressed very large state space ad brach factors by combiig the hierarchical task etwork (HTN) plaig approach with game tree search to develop what was deoted as adversarial HTN (AHTN). Here, rather tha explorig the etire combiatio of possible actios, HTN plaig could guide the search directio based o domai kowledge. AHTN also exteded the HTN plaig approach to accommodate simultaeous ad durative actios, ad applied HTN plaig directly ito the game search tree to take advatage of stadard optimizatios such as alpha beta pruig. Although AHTN has achieved good performace compared to other search algorithms [10], it still suffers from the followig weakesses. 1. The HTN descriptio used by AHTN caot express complex relatioships amog tasks ad accommodate the impact of the eviromet o tasks. For example, cosider a task ivolvig the forced occupatio of a fortified eemy emplacemet, deoted as the capture-blockhouse task, which cosists of two subtasks. The first subtask ivolves lurig the eemy away from the emplacemet (deoted as the lurig-eemy subtask), ad the secod subtask ivolves attackig the emplacemet (deoted as the attackig subtask). Here, the capture-blockhouse task will fail if the attackig subtask fails. However, failure of the lurig-eemy subtask would ot ecessarily lead to a overall failure of the capture-blockhouse task, but would rather ted to icrease the cost of completig the paret task because the attackig subtask ca still be executed eve though the lurig-eemy subtask fails. This type of relatio caot be expressed by the HTN descriptio employed i AHTN. Relatios related to coditios where subtasks are triggered by the eviromet or where the executio results of a paret task deped o both the eviromet ad the successful completio of its subtasks also caot be expressed by the HTN descriptio employed i AHTN. 2. AHTN plaig caot effectively address task failures that occur durig pla executio. The plaig process i AHTN geerates a ew pla at each frame of the game wheever idle uits which are ot assiged or execute actios exist. The ew pla processig caot cacel assiged actios of the previous plaig. If uits have bee assiged actios i previous plaig, they will keep executig those actios util those actios are failed or completed. I AHTN, each

3 Appl. Sci. 2017, 7, of 25 failed task of the previous pla is simply removed at the time of failure without cosiderig its impacts o remaiig executig tasks. The remaiig tasks will cotiue executig though these executios may be meaigless. To illustrate this poit, we ote that, if the failure of a task leads to the failure of its pla directly, the remaiig executig tasks of the pla should ot be cotiued executio because they caot affect the failure of the pla. The remaiig executig tasks should therefore be termiated so that the released resources ca be employed to repair failed tasks or formulate a ew pla. Thus, whe oe task fails ad caot be repaired, all related tasks i the pla should be termiated, ad the AI player should attempt to repair the task to maitai the validity of the origial pla. I this paper, we propose a modified AHTN algorithm with failed task repair fuctioality, deoted as AHTN-R, to address the two problems discussed above. First, AHTN-R employs a HTN descriptio exteded by addig the elemets essetial task, phase, ad exit coditio to ehace its capability for expressig complex relatioships amog tasks ad for accommodatig the impacts of the eviromet. Secod, AHTN-R employs a moitorig strategy based o the exteded HTN descriptio to idetify all tasks affected by a failed task. Fially, we develop a ovel strategy for repairig failed tasks based o a prioritized list of alterative plas. The priorities of alterative plas are geerated by sortig all odes of the game search tree accordig to their primary features, ad we employ the valid alterative pla with the highest priority to repair the failed task. This method ca reduce time cosumptio by takig advatage of historical iformatio. I each decisio cycle, the method attempts to repair failed tasks, followed by a ew plaig process. The remaider of this paper is orgaized as follows. Related work is preseted i Sectio 2. The exteded HTN descriptio is preseted i Sectio 3. The AHTN-R plaig algorithm is preseted i Sectio 4. Experimetal results idicative of the performace of the proposed AHTN-R algorithm are compared with the performaces of other algorithms i Sectio 5. Fially, we coclude the paper ad preset a idicatio of future work i Sectio Related Work Sice the foudatio of HTN plaig was first proposed by Sacerdoti [11] i 1975, may HTN plaers have bee proposed, such as the simple hierarchical ordered plaer (SHOP) [12] ad its successor SHOP2 [13]. Combiig HTN plaig with search methods guided by huma kowledge has bee demostrated to speed up plaig dramatically, ad this has led some researchers to attempt to apply this approach i typical video games [14]. Meif et al. [15] applied the Simple Hierarchical Plaig Egie (SHPE) to Steam Box, which is a type of first perso shooter game, usig a alterative ecodig of plaig data. Soemers ad Wiads [16] ivestigated the reuse of HTN plas i video games. Numerous examples exist of the successful use of HTN plaers i commercial video games [17]. However, to the best of our kowledge, oly a few attempts have bee made to apply HTN plaers to RTS games. Muñoz-Avila ad Aha [18] employed a HTN plaer to provide huma players with explaatios for the reasos leadig to curret states or evets accordig to user queries, ad for iterrogatig the behavior of autoomous players uder computer cotrol. Laaglad [19] preseted the desig, implemetatio, ad evaluatio of a HTN plaer i a ope source RTS game deoted as Sprig. The plaig of the RTS game was divided ito three levels with the HTN plaer beig employed i the highest strategic level. Laaglad [20] further summarized the advatages ad disadvatages of HTN plaig i RTS games. Naveed et al. [21] employed HTN plaers to reduce the size of the pathfidig search space i RTS games, ad tested their algorithm i games developed usig ORTS i Most recetly i 2015, Otañó ad Buro [10] developed the AHTN algorithm ad applied it to RTS games. Of the above studies, we ote that o research other tha that of Otañó ad Buro [10] has cosidered the adversarial feature of RTS games. Moreover, o research has adequately addressed task failures that occur durig pla executio i RTS games. However, studies focused i other fields have addressed task repair based o HTN. Garzó et al. [22] proposed a task repair strategy based o HTN plaig to adapt to failures durig the executio of treatmet plas geerated by therapy

4 Appl. Sci. 2017, 7, of 25 plaig systems. The strategy employed three methods for effectig task repair: applicatio of the kowledge base, search of alterative decompositios, ad operator otificatio. Gateau et al. [23] employed HTN to repair failed tasks to as localized a extet as possible i a high-level distributed architecture for multi-robot cooperatio. The core cocept of the strategy was to re-pla oly a subtree of the pla based o a hierarchical structure. Aya et al. [24] desiged a extesio of SHOP to repair failed tasks by itroducig a task-depedecy graph. Here, oly that portio of the origial pla related to the failed task was subjected to re-plaig. The task-depedecy graph was employed to record causal liks i the task etwork, ad was the applied to elimiate the effects of re-plaig o the other portios of the origial pla, ad to reduce the cost of re-plaig. The approach applied re-plaig oly to the affected parts of the task etwork usig HTN. However, may of these approaches failed to take advatage of the iformatio geerated i the previous plaig process to reduce the cost of task repair. 3. Exteded HTN Descriptio This sectio presets a exteded HTN descriptio to express complex relatioships amog tasks ad accommodate the impacts of the eviromet o tasks. We first aalyze the requiremets for extedig the HTN descriptio, ad the preset a formal defiitio of the exteded HTN descriptio Requiremets Aalysis The HTN plaig approach geerates plas by decomposig tasks recursively ito smaller subtasks [25]. HTN plaig icludes two types of tasks: primitive tasks ad compoud tasks. Primitive tasks correspod to actios that ca be executed by a aget directly i the game eviromet, such as a specific movemet. Compoud tasks represet goals that must be achieved by a high-order pla ad decomposed ito a task etwork Essetial Task Attribute I AHTN plaig, the result of a compoud task depeds o the results ad logical relatioships of its subtasks. Its task decompositio tree is a ad-or-tree with logical relatioships betwee subtasks that are either AND or OR. The AND relatioship idicates that the failure of ay subtask will lead to a failure of its paret task. The OR relatioship idicates that the success of ay subtask will lead to the success of its paret task. However, some domai kowledge regardig the completio of a task caot be represeted by a ad-or-tree. Returig to the capture-blockhouse task example discussed i the itroductio, we ote that the success of the lurig-eemy subtask may decrease the cost of completig the paret task, but its failure would ot lead to the failure of its paret task. However, the success ad failure of the capture-blockhouse task is completely depedet o the success ad failure of the attackig subtask. These more complex logical relatioships caot be represeted by the formatio of a ad-or-tree. To represet task decompositio reflectig more complex logical relatioships, we itroduce a attribute deoted as essetial task, whose value is either true or false, ad this attribute is applied to all tasks of the HTN. Here, the failure of ay essetial task will automatically lead to the failure of its paret task without cosideratio for the results of ay other subtasks. For two subtasks ad, their logical relatioships are represeted as follows: 1. if the essetial task attributes of both are true, the relatioship betwee ad is AND; 2. if the essetial task attributes of both are false, the relatioship betwee ad is OR; 3. if the essetial task attribute of oly oe is true, the relatioship is either AND or OR.

5 Appl. Sci. 2017, 7, of Phase ad Exit Coditio Attributes The result of a compoud task ot oly depeds o the relatioships ad results of its subtasks, but sometimes also depeds o the eviromet, which ca chage dyamically. These impacts from the eviromet ca be of the followig two types. 1. The eviromet ca cotrol the start of tasks oce previous tasks have bee completed. For example, the task associated with the sudde cocealed attack o a eemy, deoted as the ambushig-eemy task, cosists of two subtasks: the first is associated with movemet to the stagig area of the ambuscade, deoted as move-to-ambushig-area, ad the actual activity of attackig, deoted as attack-eemy. It is obvious that the attack-eemy subtask could ot be executed immediately upo completig the move-to-ambushig-area subtask because the attackeemy task caot be executed util the eemy resides i the ambuscade area. 2. The eviromet could ecessitate the failure of a task. For the ambushig-eemy task example, the move-to-ambushig-area subtask must be completed before the eemy has passed the ambuscade stagig area; otherwise, the ambushig-eemy task would fail. While the AHTN algorithm ca accommodate the first type of impact usig the precoditios of tasks, the secod type of evirometal impact caot be addressed. I this paper, we propose the exit coditio attribute to address these two types of evirometal impacts. The exit coditio attribute ca be divided ito sufficiet exit coditio ad ecessary exit coditio attributes. If a task has some ecessary exit coditios, its subsequet task caot be executed util all ecessary exit coditios are satisfied. If a task has some sufficiet exit coditios, its paret task will fail whe ay sufficiet exit coditio is satisfied but the task is ot completed. The exit coditio attribute is writte usig a Lisplike sytax like the precoditio defied i [13]. The phase attribute is proposed to describe the relatioships betwee subtasks that reflect the fact that the executio of a compoud task ca be divided ito several sequetial steps, ad each step may iclude parallel subtasks. Because tasks i the same phase are ofte affected by the same exit coditios, we add the set of exit coditios to the phase rather tha addig them to each task. For a example, we retur to the ambushig-eemy task, where we have modified the task to cosist of three subtasks: move-to-ambushig-area, surveil-eemy, ad attack-eemy. The modificatio icludes the subtask surveil-eemy reflectig the requiremet to first determie that the eemy resides i the ambuscade area prior to attackig. The first two subtasks are executed i parallel usig differet uits ad the attack-eemy ca be executed oly whe the first two subtasks are completed. We divide the executio of ambushig-eemy ito two phases. The subtasks move-to-ambushig-area ad surveil-eemy are icluded i the first phase, which is deoted as the preparatory-phase. The attackeemy subtask is icluded i the secod phase, deoted as the attack-phase. The preparatory-phase has both sufficiet ad ecessary exit coditios, where the ambushig-eemy task will fail if the subtasks of the preparatory-phase have ot bee completed by the poit at which the eemy has passed the ambushig-area (i.e., the sufficiet exit coditio is satisfied), ad the attack-eemy task will ot be iitiated if the subtasks of the preparatory-phase have bee completed while the eemy has ot passed the ambushig-area (i.e., the ecessary exit coditio is ot satisfied). The data structures employed i the example are show i Figure 1, where the uits deoted as?uitid1 ad?uitid2 will ambush the eemy deoted as?eemy at the ambushig-area deoted as?place. I the preparatory-phase,?uitid1 is allocated to move to?place ad?uitid2 surveils?eemy. If the preparatory-phase is completed,?uitid1 will attack?eemy. Additioal defiitios are provided i the followig subsectios.

6 Appl. Sci. 2017, 7, of 25 Figure 1. A example of the ambushig-eemy task. Icludig the phase attribute durig the decompositio of a compoud task is the mai differece betwee the AHTN used i this paper ad that i [10]. The defiitio of the phase attribute is as follows. Defiitio 1. (phase): The phase attribute is defied as a tuple:, h,,,, where the followig defiitios apply. is the ame of the phase. h cosists of the ame of the method to which the phase belogs ad a list of parameters for the phase. is the set of sufficiet exit coditios. Each elemet is a logic expressio of literals which cosist of the ame ad a list of parameters. is the set of ecessary exit coditios. Each elemet is a logic expressio of literals which cosist of the ame ad a list of parameters. Here, subsequet phases ca be executed oly whe all ecessary exit coditios are satisfied. is the set of subtasks that should be completed to accomplish a compoud task. The subtasks ca be either compoud tasks or primitive tasks. Each elemet i cosists of the subtask s ame ad a list of parameters. Because a phase is executed sequetially, ay failure of a phase will lead to the failure of the method associated with that phase. The result of a phase depeds o two factors: tasks ad sufficiet exit coditios. Accordigly, the followig four coditios lead to the failure of a phase: 1. Ay essetial task of the phase has failed. 2. Ay sufficiet exit coditio is satisfied, with oe or more essetial tasks still executig. 3. Ay sufficiet exit coditio is satisfied, ad o subtasks have bee completed. 4. All subtasks have failed. For 1, the failure of ay essetial task meas that the paret task has failed, ad its phase is also cosidered as failed. For 2, ucompleted essetial tasks caot cotiue executig whe a sufficiet exit coditio is satisfied, ad the paret task also fails. I additio, accordig to 3, the phase will fail whe o subtasks are completed ad a sufficiet exit coditio is satisfied. Fially, a task obviously fails if all its subtasks have failed Defiitio of the Exteded HTN Descriptio

7 Appl. Sci. 2017, 7, of 25 A HTN problem ca be defied as a tuple:,,,,, where the followig defiitios apply: is the set of curret world states, ad cosists of all iformatio that is relevat to the plaig process. is the set of task decompositio methods. Each method ca be applied to decompose a task ito a set of subtasks. is the set of operators. Each operator is a executio of a primitive task. is the curret task etwork. It is a tree whose odes are tasks, methods, or phases. is the state trasform fuctio. Give, (, ) defies the trasitio of the state whe a primitive task is executed by a aget. If (, ), the operator is ot applicable i. The defiitio of a method i the proposed exteded descriptio is similar to the defiitio employed i previous HTN descriptios. However, our method uses the list of phases rather tha sets of subtasks, ad adds the attribute deoted as essetial. Defiitio 2. (method): A method is defied as a tuple:,,, h,, where the followig defiitios apply. is the ame of the method. is the ame of the task to which the method is applied. is the logical precoditios for the method, which should be satisfied whe the method is applied. h is the list of phases. The sequece of the elemets i h is the executio sequece. is a Boolea value, ad is true to idicate that a task to which the method is applied is a essetial task. The defiitio of a operator is also similar with that i previous HTN descriptios except for addig the essetial attribute. Defiitio 3. (operator): The operator is defied as a tuple: h,,,,, where the followig defiitios apply. h is the primitive task that ca be applied by this operator. is the precoditio that must be satisfied before task executio. are the delete effects. are the add effects. is a Boolea value, ad is true to idicate that the task to which the operator is applied is a essetial task. 4. AHTN-R Plaig Algorithm The overall framework of the AHTN-R algorithm is provided firstly, ad the fuctios of its differet compoets are explaied. The, the moitorig strategy employig the exteded HTN descriptio is proposed to address the issue of failed tasks. Fially, we discuss the proposed failed task repair strategy.

8 Appl. Sci. 2017, 7, of ATHNR Framework The overall framework of the AHTN-R algorithm is illustrated i Figure 2. The process flow cosists of three mai compoets: pla geeratio, pla executio, ad task repair. The pla geeratio compoet provides the best pla to the pla executio compoet, ad provides a prioritized list of alterative plas to the task repair compoet. The pla executio compoet processes the failed tasks resposible for pla failure ad seds the task eedig repair to the task repair compoet. The task repair compoet attempts to repair failed tasks at the begiig of each decisio cycle. Figure 2. Overall framework of the adversarial hierarchical task etwork (AHTN)-R algorithm. At each decisio cycle, whe there are idle uits, the pla geeratio compoet will search the best pla usig the AHTN algorithm with the exteded HTN descriptio. This paper modified the part of the AHTN algorithm obtaiig the ext actio. I AHTN, the subtasks ca be obtaied if a method is applied. However, with the itroductio of phase i AHTN-R, the phases will be obtaied rather tha the subtasks if a method is applied. The AHTN-R algorithm accesses the geerated phases to obtai the ext primitive task. Algorithm 1 shows the method employed to obtai the ext executable actio based o the HTN pla N ad executio poiter t, which keeps track of which parts of N have already bee executed. The result of Algorithm 1 is a primitive task or a empty set, idicatig a absece of a available primitive task. Lies 1 4 show that, if o tasks have yet bee executed, the algorithm returs the first primitive task i N. Lies 7 19 show that, if all tasks i the same phase with t have bee executed, the algorithm searches the first primitive task from the ext phase. If o primitive tasks are foud i the ext task, a empty set is retured. Lies show that the algorithm searches for a primitive task from the remaiig tasks i the same phase that have ot bee executed.

9 Appl. Sci. 2017, 7, of 25 Algorithm 1 Returs (, ) 1. If = the 2. Get the root of N, deoted as the 3. Retur (, ) 4. Ed If 5. Acquire the phase p to which t belogs 6. Get all tasks deoted as of 7. If all tasks i have bee executed the 8. Get the ext phase of 9. If = the 10. Retur 11. Ed If 12. Get all tasks deoted as of 13. For all do 14. If (, ) the 15. Retur (, ) 16. Ed If 17. Ed For 18. Retur 19. Ed If 20. For all that have ot bee executed do 21. If is a primitive task the 22. Retur 23. Ed If 24. Get the first phase of the method applied to 25. Get all tasks deoted as of 26. For all do 27. If (, ) the 28. Retur (, ) 29. Ed If 30. Ed For 31. Ed For 32. Retur Oce the best pla is geerated, the AHTN-R will obtai the prioritized listig of alterative plas geerated by AHTN through sortig the odes of the game search tree. The method of sortig plas is discussed i Sectio 4.3. Durig pla executio, the task may fail because of the dyamics of the RTS game eviromet. Oce a task fails, AHTN-R will process the failed task accordig to the moitorig strategy discussed i Sectio 4.2, which seeks to idetify ad termiate the smallest possible set of tasks affected by the failed task. I this way, the effects of a failed task ca be limited to the smallest scope possible. At the begiig of each decisio cycle, AHTN-R first attempts to repair the failed task by selectig a suitable pla from the list of alterative plas accordig to their priorities. AHTN-R will the execute the tasks of the alterative pla correspodig to the tasks affected by the failed task. If AHTN-R is uable to fid a suitable alterative pla, it will geerate a altogether ew pla for the failed task by AHTN. This process is discussed at the ed of Sectio Failed Task Moitorig Strategy Usig the Exteded HTN Descriptio Oce a task fails, a AI player must idetify ad termiate all tasks affected by the failed task. To accomplish this, the sources of the pla failure are aalyzed first. Pla failure ca be attributed to two sources: the failure of actios ad the state of the sufficiet exit coditios attribute. A actio will fail whe its uit is destroyed or whe available resources caot be obtaied to fulfill the actio. A pla may also fail if the sufficiet exit coditio is satisfied, as discussed i Sectio Task failure

10 Appl. Sci. 2017, 7, of 25 triggers the moitorig strategy. Algorithm 2 shows the mai process of the strategy employed to moitor failed tasks. The iput of Algorithm 2 is the failed task t, ad the result of Algorithm 2 is a list deoted as RepairTaskList, which cotais the tasks requirig repair. Lies 1 4 show the differet strategies for addressig essetial ad uessetial tasks. Lies 5 15 show how failed compoud tasks are addressed. Lies 7 13 show that, if a failed compoud task has available methods, it will be saved ad repaired i the ext decisio cycle. Oly whe a task has o available methods will it be cosidered as the failed task. I this way, the umber of affected tasks will be as small as possible, which will reduce the extet of repair required. Lie 15 shows the strategy for addressig primitive ad compoud tasks without available methods. Algorithm 2 Retur TaskFailed(t) 1. If ( ) = the 2. SubTaskFailed(t) 3. Retur 4. Ed If 5. If t is a compoud task the 6. Acquire all methods m of t 7. For each method ( )! =failed 8. Get the precoditio set of 9. If all is satisfied the 10. Add t ito RepairTaskList 11. Retur 12. Ed if 13. Ed For 14. Ed if 15. SubTaskFailed(t) Accordig to the exteded HTN descriptio, each task, except the root ode task, belogs to a sigle phase, ad whether the phase correspodig to a failed task has also failed must be determied. Algorithm 3 shows the strategy for addressig the correspodig phase accordig to the HTN descriptio preseted i Sectio 3.2. The iput of Algorithm 3 is a task t. Lie 2 shows that, if the failed task is the root ode task, it eed ot be repaired. If the phase is failed, Algorithm 4 ad Algorithm 5 are called to termiate the affected tasks ad process compoud tasks. The failed phase serves as the iput of Algorithms 4 ad 5. Algorithm 4 shows the strategy for processig the failed phase. If the task belogig to the failed phase is a compoud task, its subtasks will be processed, ad, if it is a primitive task, it will be termiated directly. Algorithm 5 shows the strategy for processig the method to which the failed phase belogs. Lies 2 6 show that all ufiished phases of the method are failed because they are sequetial.

11 Appl. Sci. 2017, 7, of 25 Algorithm 3 SubTaskFailed(t) 1. Acquire the phase correspodig to t 2. If = the retur 3. Ed If 4. If oe of the sufficiet coditios of is met the 5. If ay essetial tasks of are failed the 6. PhaseFailed( ); 7. MethodFailed( ); 8. Else If has o essetial task 9. (all tasks of were executed or did ot start) 10. PhaseFailed( ); 11. MethodFailed( ); 12. Ed If 13. Ed If 14. Else If ay essetial tasks of fail 15. ( has o essetial task all tasks of fail) 16. PhaseFailed( ); 17. MethodFailed( ); 18. Ed If 19. Ed If If ay sufficiet exit coditio is satisfied, the AI player will determie whether the phase to which the sufficiet exit coditio belogs has failed. The processig ivolved is similar to that give by Algorithm 3 except that its iput parameter is a sufficiet exit coditio rather tha a failed task. Algorithm 4 PhaseFailed( ) 1. Get all tasks deoted as of 2. For all tasks 3. If status( ) = h status( ) = the 4. cotiue; 5. Ed If 6. If t is a compoud task the 7. Get the method m applied to t 8. Get all phases withi h of m 9. For all phases h 10. PhaseFailed(q) 11. Ed For 12. Else 13. Cacel (t) 14. Ed If 15. Ed For

12 Appl. Sci. 2017, 7, of 25 Algorithm 5 MethodFailed (p) 1. Get the method m to which p belogs 2. For all phases h of m 3. If h ( )! = h the 4. PhaseFailed(q) 5. Ed If 6. Ed For 7. Get task t to which m is applied 8. TaskFailed(t) 4.3. Task Repair Strategy Based o the Priorities of the Alterative Plas At each decisio cycle, the AI player attempts to fid a alterative pla for each task i RepairTaskList based o the iformatio icluded i previous plaig. Here, if all plas geerated durig previous plaig could be saved, the AI player could geerate a repair pla by searchig the previously geerated plas rather tha costructig a ew pla for the failed task directly. This will reduce the time cosumed by the task repair process. To accomplish this, the AI player geerates some alterative plas with differet evaluatios durig each task plaig period. These evaluatios establish the priorities of the alterative plas. The evaluatios are based o two primary features of the alterative plas: their task decompositios ad task parameters. For example, a task T ca be decomposed ito tasks sub-t1 ad sub-t2, or sub-t3 ad sub-t4. Each costructs a pla. The plaer will compute the evaluatios for executig T accordig to the two plas, respectively. If oe fails durig executio, the plaer will employ the other to repair T. Better repair performace will be obtaied by coductig a search accordig to the priorities of the alterative plas, begiig with the alterative pla with the highest priority. However, aside from the best pla, the plas geerated by the AI player are usually out of order. Therefore, the geerated alterative plas must be recorded i descedig order accordig to their evaluatios. At the same time, because the alterative plas form the leaves of the game search tree i the AHTN algorithm, the reorderig of plas is equivalet to the reorderig of leaf odes. To accomplish the above processig, we propose Theorem 1 ad prove the validity of the orderig thereby obtaied as follows. Theorem 1: Give a m-level mi max search tree, the ith level has Ni odes, where each ode is represeted by i,j (i m, j Ni). The sub-odes of i,j are ordered accordig to the followig two coditios. 1. if i,j (i.e., the paret ode) is a max ode, its subodes (or child odes) are ordered decreasigly from left to right accordig to their evaluatios; 2. if i,j is a mi ode, its subodes are ordered icreasigly from left to right accordig to their evaluatios. The above orderig of all odes of the mi-max search tree represet the proposed prioritizig of plas, where the pla of the left ode will better tha the pla of the right ode. Proof: We employ the mi max search tree illustrated i Figure 3 as a example, where max odes ad mi odes are represeted by rectagular ad circular boxes, respectively. Here,,,,,,,,,, are subodes of, ad,,,,,,,,, are subodes of,,,1. We assume that, is a max ode. The evaluatios are deoted by the variable v. For example, the evaluatio of,, is represeted by,,, where the subode evaluatios of, are ordered decreasigly from left to right as,,,,,,. I additio, we assume that,, is a mi ode, so that its subode evaluatios are ordered icreasigly from left to right as,,,,,,. With, beig a max ode, we obtai the relatioships, =,, =,, ad,,,,,,,,,,,,. Whe the best pla ode,, is removed, the value,, of,, will become,,, ad the values of the other subodes of, will ot chage. Therefore, the value of, will become,,, ad the best pla ode is,,.

13 Appl. Sci. 2017, 7, of 25 If we cotiue the process of removig the best pla ode, the best pla ode will become,,,,,,,,,. Whe all subodes of,, have bee removed, the best pla ode will become the first subode of,,. For,, the order of best plas is the same as the order of,,. If, is a mi ode, the coclusio will ot chage., v i 1,1 i 1,1 v i,1,1 i,1,1, v i,1,2 i,1,2, v i,1, N i,1, N i i, v 1,1,1 1,1,1, v 1,1,2 1,1,2, v 1,1, k 1 1,1, k 1, v 1,2,1 1,2,1, v 1,2,2 1,2,2, v 1,2, k2 1,2, k2, v 1, j,1 1, j,1, v 1, j,2 1, j,2, v 1, j, k 1, j, k, Figure 3. A mi max search tree is employed as a example for proof of Theorem 1. The rectagular odes are max odes, which are ordered icreasigly from left to right accordig to their evaluatios, ad the circular odes are mi odes, which are ordered decreasigly from left to right accordig to their evaluatios. Algorithm 6 shows the mai process of sortig plas accordig to their evaluatios. The iput of Algorithm 6 is a ode of the game search tree. The sortig process begis with the root ode of the game search tree. The algorithm sorts all odes of the game search tree from the top dow. Lies 2 4 check whether the iput ode is a leaf ode. Lies 5 9 sort all subodes of the iput ode accordig to Theorem 1. Lies process each subode of the iput ode iteratively. Algorithm 6 SortedPla(root) 1. Get the list of all subodes of root 2. If = the 3. Retur 4. Ed If 5. If root is a max ode the 6. Sort decreasigly accordig to subode evaluatios 7. Else 8. Sort icreasigly accordig to subode evaluatios 9. Ed If 10. For each 11. SortedPla( ) 12. Ed For After coductig the above process, the geerated alterative plas are provided to the AI player for repairig failed tasks. However, prior to coductig task repair, we must determie the locatio of the failed task withi the alterative pla, ad execute that part of the alterative pla i place of the failed task. Because the same task ca be used as a subtask of differet compoud tasks, it is ecessary to compare both the ame ad decompositio path to cofirm the locatio of the failed task withi the alterative plas. Whe the failed task has bee foud i a alterative pla, its leaf odes will be cosidered as the ew actios to be executed.

14 Appl. Sci. 2017, 7, of 25 A example is give i Figure 4, where Figure 4a shows a pla that has failed owig to the failure of essetial task T6 (give i bold fot). The failure of T6 requires the repair of T3, which is a subtask of T1, ad does ot affect T2. AHTN will oly remove T6 ad cotiue executig remaiig tasks eve though the pla has failed. However, AHTN-R ca select a alterative pla, which applies aother method to T3. AHTN-R executes ew subtasks T11 ad T12 i place of subtasks T6 ad T7, which is marked by the dashed-lie box i Figure 3b. (a) The Failed Pla (b) The Selected Pla Figure 4. A Example of repairig a failed pla (a) owig to the failure of subtask T6 by employig a alterative pla (b). 5. Experimetal 5.1. Experimetal Eviromet ad Settigs We compared the performace of the proposed AHTN-R algorithm with other algorithms [26] usig free µrts software [27], which has bee used i the past to evaluate various algorithms applied i RTS games [4,9,10]. A screeshot of the µrts game eviromet is show i Figure 5. Here, the uits of two players (deoted by the blue ad red outlies) compete to destroy the uits of their oppoet. Each player has the same types of uits. The small gray circles are workers that ca attack eemies, build bases, ad harvest ad trasport resources. The gree squares are the limited resources that ca be harvested. The orage circles are light attackers, ad the yellow circles are heavy attackers. The heavy ad light attackers both have larger hit ad health poits tha the workers, makig them more effective ad resiliet attackers, but they caot harvest or trasport resources. The white squares are bases that ca produce ew uits. The µrts game eviromet provides six types of actios for uits: move ito ay empty space i the four cardial directios, attack ay eemy i rage, harvest mierals from resource uits, retur mierals to bases, produce ew uits i ay empty spaces i oe of the four cardial directios (for workers ad bases), ad remai idle, where

15 Appl. Sci. 2017, 7, of 25 the uit takes o actio. Although µrts provides simplified RTS games, the games are sufficietly complex, ad exhibit the stadard challeges of RTS games, such as cocurret player activity, durative ad simultaeous actios, real-time limitatio, ad large brachig factors. The origial µrts is determiistic ad fully observable. Figure 5. A screeshot of the µrts game eviromet. The two players are distiguished accordig to the blue ad red outlie colors. The gree squares are resources. The white squares are bases. The gray circles are workers, the yellow circles are heavy attackers, ad the orage circles are light attackers. The compariso evaluatio cosidered the stadard AHTN algorithm i additio to the followig collectio of search algorithms. 1. Radombiased: A AI player employig a radom biased strategy that executes actios radomly. 2. LightRush: A hard-coded strategy. This AI player produces light attackers, ad commads them to attack the eemy immediately. 3. HeavyRush: A hard-coded strategy. This AI player produces heavy attackers, ad commads them to attack the eemy immediately. 4. UCT: We employ a implemetatio with the extesio for accommodatig simultaeous ad durative actios [10]. The followig parameters were employed i the testig. CPU time: The limited amout of CPU time allowed for a AI player per game frame. I our experimets, we employ differet CPU time settigs from 20 to 200 ms to test the performaces of the algorithms. Playout policy: The Radombiased playout policy is employed for the AHTN-R, AHTN, ad UCT algorithms i our experimets [10]. Playout time: The maximum ruig time of a playout. The playout time is 100 cycles. Pathfidig algorithm: I our experimets, the AI player employs the A* pathfidig algorithm to obtai the path from a curret locatio to a destiatio locatio.

16 Appl. Sci. 2017, 7, of 25 Evaluatio fuctio: I our experimets, we employ a evaluatio fuctio derived from [10] to compute a reward value to obtai the best pla. The evaluatio fuctio is a variat of LTD2 [27,28] that ot oly cosiders the hit-poits of a uit, but also its costs. Maximum game time: The maximum game time is limited to 3000 cycles. This meas that, if both players have livig uits at the 3000th cycle, the game is declared a tie. Maps: The three maps used i our experimets are M1 (8 8 tiles), M2 (12 12 tiles), ad M3 (16 16 tiles). To evaluate each algorithm, we coduct a roud-robi touramet, i which each algorithm plays 50 games (with various startig positios) agaist all other algorithms i each of 3 differet maps ( = 9600 games i total). The method used to compute the score of each algorithm is as follows: the wier of each game was awarded 1 poit, ad both algorithms were awarded 0.5 poits i the evet of a tie. Each of the two AI players i all competitios bega with a sigle base, a equivalet resource value, ad a sigle worker. To compare the performace of our AHTN-R algorithm with the performace of the stadard AHTN algorithm, we crafted two differet HTNs for the µrts domai, which are defied as follows. 1. Low Level: cotais 12 operators (primitive tasks) ad 9 methods for 3 types of tasks. 2. Flexible: cotais the 12 operators of the Low Level, but provides 49 methods ad 9 types of tasks. This fuctioality allows methods to employ parallel executio tasks. The employmet of the Low Level ad Flexible HTNs i the AHTN ad AHTN-R algorithms are deoted subsequetly as, e.g., AHTN-R-LL ad AHTN-R-F, respectively. Naturally, the HTNs used by the AHTN-R algorithm employ the exteded HTN descriptio. All tasks except for wait-forfree-uit ad wait i HTNs are defied as essetial tasks. Some of the phases iclude exit coditios to reflect the ifluece of the eviromet. Figure 6 shows a example task from the Low Level HTN. The task amed destroy-player-iteral ca be decomposed ito two subtasks uit-order ad destroyplayer-iteral by applyig the method dpi-extuit whe the precoditio is satisfied. Because the two subtasks are parallel, they belog to the same phase, deoted as dpi-extuit-phase. Because oe subtask is the same as its paret task, we will focus o subtask uit-order. If the precoditios of its method uo-attack are satisfied, this task will be decomposed ito the subtasks move-ito-attack-rage ad attack. The subtasks belog to the differet phases: preparatory-phase ad attack-phase because they must be performed sequetially, ad both are essetial tasks. We ote that attack-phase-1 has a ecessary exit coditio, idicatig that subtask attack i the subsequet phase caot be executed util the ecessary exit coditio is satisfied. Because the essetial attributes are all true i Figure 6, the failure of move-ito-attack-rage or attack will lead to repair uit-order. If uit-order caot be repaired, the plaer will termiate all tasks i dpi-extuit-phase ad try to repair its paret task.

17 Appl. Sci. 2017, 7, of 25 Figure 6. A example of the AHTN-R process of executig the compoud task destroy-player-iteral from the Low-Level fuctioality Experimetal Results ad Aalysis I this sectio, we compare the performace of AHTN-R with the other algorithms i terms of the average score, average decisio time, ad average failed task repair rate for the three maps. Each algorithm will play 700 games i total agaist the other algorithms as player 1 or player 2 ad 50 games agaist itself. Because data are collected from both players, the average score of each algorithm show i Figures 7 9 is obtaied effectively over 800 games. The average decisio time for each algorithm show i Figures is effectively the average value of 100 games agaist eight types of AI players, respectively. The average failed task repair rate of AHTN-R is also obtaied i the same way.

18 Appl. Sci. 2017, 7, of Average Score uder Differet CPU Time Settigs for the Three Maps Figures 7 9 preset the comparisos of the average scores obtaied for each algorithm durig the roud-robi competitio with respect to the CPU time from 20 to 200 ms for the three maps. Accordig to the results, we ca see that the AHTN-R-F player outperforms all other AI players o all three maps uder differet CPU times. I additio, the performaces of AHTN-R-F vary little with the icreasig scale of the maps, ad are very stable with respect to CPU time. AHTN-R-F outperforms AHTN-F because of the greater flexibility of AHTN-R tha AHTN i a dyamic eviromet. Here, AHTN does ot cosider the effects of failed tasks, while AHTN-R idetifies ad termiates all tasks affected by the failed task, which allows AHTN-R to make more efficiet use of available uits ad resources. AHTN-R also attempts to repair failed tasks i as local a extet as possible i a effort to maitai the validity of the previous pla. AHTN-R-F performs best because it uses a improved HTN domai kowledge for guidig the game tree search, which yields a good pla i a relatively short time. With respect to the other algorithms, we ote that AHTN-R-LL ad AHTN-LL have similar performaces o the smallest map. However, the performace of both AHTN-R-LL ad AHTN-LL deteriorate as the map size icreases, ad AHTN-R-LL geerally performs worse tha AHTN-LL o the largest map. The performaces of both players deteriorate because more uits are produced o the larger maps ad the domai kowledge obtaied by AHTN-R-LL ad AHTN-LL is too simple to adapt to such a complex game. For AHTN-R, the poor domai kowledge meas that the previous pla may ot be suitable for the curret coditio, ad it would be better to costruct a ew pla. The relative performaces of the scripted methods HeavyRush ad LightRush are also observed to improve with icreasig map size, which is owig to the uderperformace of AHTN-R-LL ad AHTN-LL o larger maps. With respect to CPU time, the performaces of all AI players chage very little for all three maps. This is because all AI players require little time to make decisios. However, if the CPU time is very small, the performace will deteriorate. Because the performaces of all AI players were similar uder differet CPU time settigs, we employed a sigle CPU time settig of 100 ms i subsequet experimets. Figure 7. The average score of each algorithm obtaied over the 800 games of the roud-robi competitio for map M1 (8 8 tiles) with respect to the CPU time from 20 to 200 ms. The playout time is 100 cycles.

19 Appl. Sci. 2017, 7, of 25 Figure 8. The average score of each algorithm obtaied over the 800 games of the roud-robi competitio for map M2 (12 12 tiles) with respect to the CPU time from 20 to 200 ms. The playout time is 100 cycles. Figure 9. The average score of each algorithm obtaied over the 800 games of the roud-robi competitio for map M3 (16 16 tiles) with respect to the CPU time from 20 to 200 ms. The playout time is 100 cycles Average Decisio Time for the Three Maps Figures show the average plaig times of the AHTN-R ad AHTN players for oe decisio process of 100 games for the three maps, respectively. We fid that AHTN-R usually requires more decisio time tha AHTN whe they have similar domai kowledge. This is caused by the added time of task repair. Both the repair of failed tasks at the begiig of each decisio cycle ad the sortig of all alterative plas accordig to their evaluatios requires greater plaig time for AHTN-R tha AHTN. However, as show i Tables 1 ad 2, the time required for repairig failed tasks ad savig alterative plas is short compared with the total time required by the decisio process. Therefore, while AHTN-R requires greater decisio time tha AHTN, it is withi a acceptable rage. We ote that the decisio times for AHTN-LL ad AHTN-R-LL icrease for M2 ad M3 whe playig agaist Radombiased or UCT. Compared to the other algorithms, Radombiased ad UCT have greater opportuity to create complex game eviromets due to the

20 Appl. Sci. 2017, 7, of 25 greater radomess of their actios. Here, the poor domai kowledge of AHTN-LL ad AHTN-R- LL relative to that of AHTN-F ad AHTN-R-F makes it more difficult for AHTN-LL ad AHTN-R- LL to geerate plas i respose to the radom actios of Radombiased ad UCT. Therefore, AHTN-LL ad AHTN-R-LL must utilize more recursive tasks i HTN to obtai better solutios. This requires that the AI player search more tree odes, ad the time spet o some odes also icreases with icreasig HTN depth. Figure 10. The average decisio times of AHTN-R ad AHTN with the two types of HTNs obtaied over 100 games agaist each of the 8 algorithms, where the CPU time is 100 ms for map M1 (8 8 tiles). The playout time is 100 cycles. Figure 11. The average decisio times of AHTN-R ad AHTN with the two types of HTNs obtaied over 100 games agaist each of the 8 algorithms, where the CPU time is 100 ms for map M2 (12 12 tiles). The playout time is 100 cycles.

21 Appl. Sci. 2017, 7, of 25 Figure 12. The average decisio times of AHTN-R ad AHTN with the two types of HTNs obtaied over 100 games agaist each of the 8 algorithms, where the CPU time is 100 ms for map M3 (16 16 tiles). The playout time is 100 cycles. Table 1. The percetage of extra time required for repairig failed tasks ad savig alterative plas relative to the total time for decisio processig. The games were betwee AHTN-R-LL ad the 8 algorithms for the three map sizes (M1 is 8 8 tiles, M2 is tiles, ad M3 is tiles). AI Player M1 M2 M3 Radombiased LightRush HeavyRush UCT AHTN-R-LL AHTN-R-F AHTN-LL AHTN-F Table 2. The percetage of extra time required for repairig failed tasks ad savig alterative plas relative to the total time for decisio processig. The games were betwee AHTN-R-F ad the 8 algorithms for the three map sizes. AI Player M1 M2 M3 Radombiased LightRush HeavyRush UCT AHTN-R-LL AHTN-R-F AHTN-LL AHTN-F Average Failed Task Repair Rate for the Three Maps The average repair rates of AHTN-R agaist the eight algorithms over 100 games for the three maps are show i Tables 3 ad 4. We ca see that AHTN-R-F has a higher repair rate tha AHTN- R-LL agaist all AI players for all three maps. It is the result of the differet levels of domai kowledge used by AHTN-R-LL ad AHTN-R-F, where the poor domai kowledge used by AHTN-R-LL was ot able to provide good alterative plas for task repair. Here, whe a player attempted to repair a failed task, o alterative plas were available due to a lack of uits. A greater umber of alterative plas with greater differeces ca icrease the repair rate. The improved