The Significance of Temporal-Difference Learning in Self-Play Training TD-rummy versus EVO-rummy

Transcription

1 The Significance of Temporal-Difference Learning in Self-Play Training TD-rummy versus EVO-rummy Clifford Konik Jugal Kalia Universiy of Colorado a Colorado Springs, Colorado Springs, Colorado CLKOTNIK@ATT.NET KALITA@PIKESPEAK.UCCS.EDU Absrac Reinforcemen learning has been used for raining game playing agens. The value funcion for a complex game mus be approximaed wih a coninuous funcion because he number of saes becomes oo large o enumerae. Temporal-difference learning wih self-play is one mehod successfully used o derive he value approximaion funcion. Coevoluion of he value funcion is also claimed o yield good resuls. This paper repors on a direc comparison beween an agen rained o play gin rummy using emporal difference learning, and he same agen rained wih co-evoluion. Coevoluion produced superior resuls. 1. Inroducion The success of TD-gammon is well known (Tesauro ). Tesauro rained an arificial neural nework (ANN) o approximae he value funcion for he game of backgammon wihou explici exper advice programmed ino he agen. Wih only he definiion of legal moves and a reward when he game was won, emporal difference (TD) learning and self-play allowed he ANN o be rained well ino he level of experienced human play. Furher refinemens allowed TD-gammon o reach exper level (Tesauro 1995). TD-gammon was developed based on some of he early work on TD learning ha has more recenly been formalized and expanded. See, for example, Suon and Baro (1998). The major challenge in deriving he value funcion is ha here are many seps he agen mus ake before he game is won and a reward can be assigned. TD learning provides a mehod o assign credi from he reward o seps leading up o i. This is done in such a way ha he value funcion can be adjused in incremenal seps as he game progresses. Combined wih an ANN, his approach provides an error signal ha is backpropagaed a each sep of he game o incremenally rain he nework. The algorihm differs from normal backpropagaion in ha he hisory of weigh changes over he course of he game is used a each sep. Suon and Baro refer o his hisory as he eligibiliy race. There are hose who quesion he significance of TD learning claimed based on he success of experimens such as TD-gammon. Pollack, Blair and Land (1996) argue ha a simple co-evoluionary approach o deriving he weighs for an ANN ha approximaes he value funcion works quie well. They argue TD learning is no he major reason for TD-gammon's success, and hey sugges i is due o he self-play approach and specific feaures of he game of backgammon. Pollack, Blair and Land describe an experimen designed o mimic TDgammon, bu wih he weighs of he ANN derived by a simple evoluionary approach. However, he acual configuraion for TD-gammon was no available o hem, making direc comparison impossible. They esed agains a publicly available version of Tesauro's backgammon player and repored encouraging resuls. For his experimen, TD and evoluionary echniques are compared direcly on he same agen wih only he mehod of deriving he weighs for he value approximaion funcion differing. This allows he resuling players o play agains each oher. In addiion, he cos of raining can be direcly compared. 2. Problem Definiion 2.1 Game Definiion The problem is o rain an agen o play he game of gin rummy. Gin rummy is a wo-handed card game ha can be summarized as follows (Gibson 1974): Deck: sandard 52 card deck Rank: King=high, Ace=low Poins: King, Queen, Jack=10; Ace=1; all ohers=face value Deal: 10 cards o each player; nex card forms discard pile; remaining cards form he draw pile; discard pile is always face-up; draw pile is face-down; winner of each hand deals he nex Goal: form meld from ses of 3 o 4 cards of same value or sequences of 3 or more cards of same sui and wih he oal of he face value Proceedings of he Twenieh Inernaional Conference on Machine Learning (ICML-2003), Washingon DC, 2003.

2 of remaining cards no so formed (called deadwood) less han or equal o 10; a single card canno form par of a se and a sequence in he same hand Turn: during each urn a player can ake he op card from he discard or draw pile, mus discard one card face-up on he op of he discard pile and, if he goal sae is reached, may lay down meld and deadwood (called knocking) Play: players alernae urns saring wih he dealer s opponen unil one player knocks Laying off: afer one player knocks, he opponen may exend any of he knocking player s ses or sequences (called laying off) wih any of his/her deadwood. Score: player who knocks scores he difference beween he oher player s deadwood poins and his/her own. If he player who knocks has no deadwood, he oher player is no allowed o lay off, and he player knocking receives a score of 25 plus he oher player s deadwood poins. If, afer laying off, he opposing player s deadwood poins are equal or less han he player knocking, he opponen scores 25 plus he difference in poins insead of he player knocking. A couple of simplificaions o he game have been made for his experimen. We decided no o incorporae laying off in his experimen. Play usually coninues unil one player reaches 100 poins. This porion of he game and oher deails of assigning bonus poins beyond he descripion of a single hand are ignored for his analysis. 2.2 Reinforcemen Learning Problem The learning agen represens a gin rummy player, hereafer called simply "he player". The environmen consiss of he opposing player, he random sequence of cards on he draw pile, and he known sequence of cards in he discard pile. The game sae is represened by he locaion of each card from he poin of view of he agen. The sae may be in-player s-hand (IPH), in-opponen shand (IOH), in-discard-pile (IDP) or unknown (UNK). A card is only considered o be IOH if i has been drawn from he discard pile. All oher cards in he opponen s hand are considered o be UNK. Gin rummy represens a moderaely complex game ha cerainly canno have all sae-acion combinaions enumeraed. Wih 52 cards in one of four possible saes here are 4 52 or approximaely 2 X possible saes. On he oher hand i has a simple se of rules and small se of acions a each urn. The acions ha he agen can perform are o exchange any card in is hand for he op card of he discard pile, o exchange any card for he op of he draw pile or o ake eiher of he preceding acions followed by knocking. The immediae reward is he score following a knock. I will be posiive if he player scores, and zero if he opposing player scores. The problem is episodic; each hand is an episode. The deails of scoring muliple hands ino a game oal are ignored. A he conclusion of each urn, a player has 10 cards. The value for his sae is approximaed wih a funcion, as described below. During each urn, he player mus decide wheher o draw from he discard pile or he draw pile. Then he decision mus be made as o which card o discard. Finally, he player mus decide wheher o knock. The policy used is as follows: For he op card on he discard pile and for each card whose locaion is unknown (i.e., possible cards on he op of he discard pile) he maximum of he value funcion is deermined. This is accomplished by evaluaing he hand resuling from exchanging each possible new card for each of he 10 cards currenly in he player s hand. The value funcion is used o approximae each of hese saes. If he value funcion for drawing from he discard pile is greaer han 50% of he possible cards on he draw pile, he player will pick from he discard pile, oherwise from he draw pile. The card o be discarded will be he one ha leaves he maximum expeced value for he remaining 10 cards. Noe ha his is one of he calculaions already complee. If he oal of he player s remaining deadwood is 10 or less, he player will knock. The ask is o learn he value funcion based only on he resuls of self-play. Excep for he value funcion, he deails of he above policy are implemened in discree program logic. The value funcion has only he inpu of he game sae and generaes a single numeric evaluaion of i. There is no knowledge of he rules of he game buil ino he value funcion. There is no noion of sequences, ses or deadwood. 3. Implemenaion The implemenaion and experimenaion phases were consrained o a fixed period of ime. A limied number of machines were available o run he raining. Given he CPU inensive naure of his sor of raining, cerain compromises were made o limi he raining ime. These will be idenified in he following descripion. An ANN is used o esimae he value funcion. The learning ask is hen o deermine he weighs for his nework. The ANN is a feed-forward, muli-layer nework wih 52 inpus (one inpu for each card), 26 hidden unis and a single oupu represening he value of ha sae. All layers of he nework are compleely

3 conneced. Values for he four card saes (IPH, IOH, IDP and UNK) are chosen wih consideraion for he nework's use of he Euclidean disance beween he saes. IPH=2, IOH=-2, IDP=-1 and UNK=0. The acivaion funcion for he hidden and oupu unis is he sigmoid. The game score is scaled o fall wihin [0,1] o correspond wih he sigmoid values. The approach used for inpus represens he accessibiliy of he card o he player. A posiive value represens possession. Negaive values represen increasing levels of inaccessibiliy. Zero is used for unknown. An alernae choice of ANN inpus for he card saes is o have four binary or bipolar inpus for each card represening he saes IPH, IOH, IDP or UNK. Only one of hese inpus would be acive a a ime. While four inpus per card may provide a more accurae represenaion, i will also increase he nework size by a facor of four and he run ime will increase accordingly. We decided o use he faser represenaion for hese ess. The learning approach uilizes self-play. The opponen is implemened as a second copy of he player. The opponen has full knowledge of he cards in is hand and parial knowledge of he cards in he player s hand. Thus he player and opponen compee on an equal fooing. This approach provides an endless se of raining daa, and i allows direc comparison of he TD and evoluionary players. Similar o he approach aken by Pollack e al. (1996), where he backgammon board was reversed, each game is played wice. Firs he cards are shuffled and deal wih one player going firs. Then he same saring order is used and he oher player goes firs. An epsilon-greedy approach is no implemened. I is assumed ha he random order of he cards from he draw pile will generae enough exploraion. The Sugar Neural Nework Simulaor (SNNS) sofware package is used for ANN processing. The value funcion approximaion is obained by a forward pass hrough he SNNS nework. A cusom back-propagaion algorihm buil on op of he SNNS framework accomplishes raining of he weighs. 3.1 Temporal Difference Learning: TD-rummy The approach used is aken from he TD-Gammon experimen as described in Tesauro (1992) and furher explained in Suon and Baro (1998). During each game, he player s value funcion is approximaed by a forward pass of he game sae hrough he player s nework. Once he player makes is decision, he nework is rained wih he cusom back-propagaion algorihm developed for TD-rummy. Training coninues incremenally afer each urn he player akes. The formula used for back-propagaion raining of he ANN weighs, Q, is Q + 1 = Q + a( r g V ( s + 1) -V ( s )) e where e is he vecor of eligibiliy races (Suon and Baro 1998) ha is buil up over he course of each game based on he formula e = g l e Q V ( s ) The immediae reward, r +1, is zero excep when he player wins he game. In his case i is he score scaled o be in [0,1]. Like TD-Gammon, here is no discouning of rewards. g = 1 Boh he player and is opponen have heir own nework ha is rained simulaneously wih his algorihm. Afer an epoch of six games, he nework corresponding o he player wih he mos wins is duplicaed and rained for boh players during he nex epoch. The six games are acually hree pairs of games wih an idenical saring sae, bu wih he player and opponen reversed. 3.2 Evoluionary Learning: EVO-rummy The second learning algorihm is he simple evoluionary approach described in Pollack and Blair (1996). The player and opponen sar ou using wo random neworks. They play an epoch, consising of wo pairs of games. If he opponen wins hree of he games, he weighs of he player s and opponen s neworks are crossed by moving he player s nework weigh 5% in he direcion of he opponen. If he opponen wins wo or fewer games, he player s nework is lef unchanged. In eiher case, he opponen s nework weighs are muaed by adding Gaussian noise o hem. The noise has a sandard deviaion of 0.1. This approach implemens a simple hill climbing. While his is called evoluionary, here is no a populaion of individuals ha compee and go hrough selecion for a new generaion. There are only wo individuals. When he opponen is measured o be superior, he evolving nework is moved in ha direcion. Like Pollack and Blair (1996), I chose no o move oo aggressively oward he opponen, moving jus 5% of he way. The idea is o implemen a simple approach. If a simple evoluionary approach can mee or exceed he emporal-difference echnique, i provides more reason o quesion he significance of he laer. Some experimenaion wih he raining parameers is included in he resuls ha follow. One nework was rained wih a crossover of 10%. Anoher nework was swiched o a higher hreshold for crossover five ou of six games. More deails on boh implemenaions will be available laer his year (Konik 2003). 4. Resuls Boh learning algorihms were run on hree o five differen self-play configuraions for a period of hree weeks. Five wo-processor Inel sysems were used ha conain GHz class CPUs. Boh algorihms are CPU

4 inensive, wih he TD algorihm using more CPU per game, as expeced. In oal more han 89,000 raining games were played using he evoluionary approach, and 58,000 wih he TD approach. Name Algorihm Training Games TD1 emp diff 9,484 TD2 emp diff 16,200 TD4 emp diff 16,243 TD5 emp diff 20,698 TD6 emp diff 1,800 EVO2 evoluion 23,762 EVO3 evoluion 18,407 EVO4 evoluion 41,154 Decripion alpha=0.1, lambda=0.3 lambda=0.7 lambda=0.2 lambda=0.9 lambda=0.9 crossover=5%, muaion=0.1 crossover=5%, muaion=0.1 crossover=10%, muaion=0.1 Figure 1. Summary of he raining parameer for he players whose performance is furher analyzed. The evoluionary algorihms were compleed firs, primarily due o he simpler sofware involved. Training commenced wih he evoluionary algorihms while he emporal-difference sofware was coded. The number of urns required o reach a winning sae wih he iniial random saring weighs is very high. Games lasing 10,000 urns or more are no uncommon. Based on he iniial ess wih he evoluionary algorihms, a limi on he number of urns was inroduced ino boh algorihms. Once reached, he game was considered a draw. For he majoriy of games, he limi was 5,000 urns. Figure 1 summarizes he raining parameers for he players whose performance is furher analyzed. The learning rae, a, is in he range [0.1,0.3]. The emporal difference sep weighing facor, l, used is in he range [0.2,1.0]. Tesauro (1992) used 0.1 for a and 0.7 for l. The larger number of games for evoluionary raining is a resul of he earlier compleion of ha sofware and hence a longer ime o rain. However, as will be described below, TD5 and EVO3 were he wo bes of breed players. Here he emporal-difference agen had more raining games han he evoluionary agen. The average number of urns per game urned ou o be a useful approximaion of he raining progress. Figure 2 shows he progression of raining for wo players wih each algorihm. This meric is only an early approximaion as indicaed by player EVO3, whose urns per game increased afer 15,000 raining games. On he surface, he value is raher disappoining since i never drops below 200. As anyone who plays gin rummy knows, he deck seldom has o be urned over o complee a game Turns / Game TD1 TD2 EVO3 EVO Training Game Nbr Figure 2. Training urns per game.

5 The real measure of performance of a player is wheher i wins agains compeiors. A ournamen where each player played every oher player for 10 games was conduced. The 10 games were five pairs of games wih he deal reversed. Figure 3 conains he resuls of he hree evolved players and four TD players. The RANDUMB player is a nework of random weighs such as ha used o sar all he raining algorihms. Thus each player played 80 games. The 18 games no accouned for were draws. Games Won Loser Winner EVO2 EVO3 EVO4 TD1 TD2 TD4 TD5 TD6 EVO EVO EVO TD TD TD TD TD RANDUMB Toal Wins beer on he whole han he TD players. The bes overall player is EVO3 ha won 87.5% of is games. This is he same player ha showed he upswing in urns per game during raining. The bes TD rained player, TD5, had he larges value of l = TD6 used l = 1. 0, bu was only rained for a shor ime. One sligh asymmeric aspec of he wo raining approaches is ha of game score versus games won. The emporal-difference approach rained he nework o esimae he game score. However, he evoluionary approach deermined he bes fi based on he games won. Therefore, figure 3 shows boh merics. Measuring performance base on score or games won produced he same ranking of players. Loser Winner EVO2 EVO3 EVO4 TD1 TD2 TD4 TD5 TD6 EVO EVO EVO TD TD TD TD TD Game Score Winner RANDUMB Loser EVO2 EVO3 EVO4 TD1 TD2 TD4 TD5 TD6 EVO EVO EVO TD TD TD TD TD RANDUMB Toal Score Figure 3. Tournamen resuls of 10 games beween players. Upper able shows he games won. Lower able shows he score. As expeced, he random weigh los every ime. The oher obvious fac is ha he evolved players did much Figure 4. Average urns per game in ournamen. Figure 4 conains he average urns per game for he ournamen. These resuls indicae ha he similarly rained players play he same and end o prolong he game. The unexpeced resuls ha EVO3 ouperforms he oher evolved players ha have longer raining cycles may indicae over-raining. To es his, a ournamen was conduced no only wih he final version of he players, bu also wih inermediae resuls. Figure 5 shows he resuls for hree players from each algorihm afer various regimes of raining. These regimes correspond o roughly 5-10,000 games. EVO4 encounered a serious problem afer regime 5. TD4 has sagnaed. The oher players do no show definie signs of over-raining. However, more daa are needed o reach any firm conclusion. A version of he ournamen program presens he sae of he game on an animaed display showing each player's acions and he cards afer each urn. Observing he play of a number of games, one echnique he evolved players

6 developed ha he TD players did no was o rapidly dispose of high cards in order o be able o knock quickly. This causes EVO-rummy players o inerfere wih each oher in obaining ses and sequences of smaller cards. From observing games beween EVO3 and TD5, i is clear ha EVO3 is superior. Boh players make obvious misakes, bu EVO3 makes far fewer. While neiher player has reached a human level of play, you have only o wach he nearly endless progression of a game beween players using random neworks o see how far he raining has come. To furher invesigae he difference in sraegy learned by hese players, he cards in each player's hand a he end of each game in he final ournamen were analyzed. The number of ses and sequences were summarized as were he number of cards of each sui and each face value. To quanify he affiniy of he players for cards of cerain face values or suis, a chi-squared es of he hypoheses ha he players' final hands conained a even disribuion of suis and face values was calculaed. The resuls are shown in figure 6. The EVO players' hands conained approximaely wice as many ses as sequences. RANDUMB's hands conained he reverse, or wice as many sequences as ses. In addiion o learning o go afer ses and sequences, he EVO players also appear o favor cards of lower face value, which makes sense. The chi-squared ess are consisen wih his, indicaing a high probabiliy he evoluionary players do no have a sui preference and a low probabiliy hey do no have a face value preference. The TD players' hands conained almos all sequences. The TD players developed a srong affiniy for a single sui. This seems o indicae he TD players jus learn o go afer one sui and he sequences jus happen. The chisquared ess suppor his. Name Chi-squared Sui Chi-squared Face EVO2 72% 30% EVO3 50% 21% EVO4 80% 29% TD1 3% 100% TD2 1% 100% TD4 2% 100% TD5 1% 100% TD6 1% 100% RANDUMB 31% 97% Figure 6. Chi-squared es of he hypoheses ha he players choose suis and face values evenly. 5. Conclusions Self-play coupled wih an arificial neural nework o evaluae he value funcion for a game is an effecive raining mehod. This experimen has demonsraed ha when self-play is coupled wih a simple, evoluionary echnique he resul can ouperform he more guided emporal difference learning echnique. Boh echniques, when combined wih self-play, can rain agens on a se of rules wihou explici human raining % Tournamen Games Won EVO2 EVO3 EVO4 TD2 TD4 TD Training Regime Figure 5. Overraining es.

7 In hese experimens, he agens rained wih he evoluionary approach developed a much more balanced sraegy, going afer ses and sequences and righly favoring cards wih lower face value. The emporaldifference raining generaed agens ha simply colleced a single sui. I appears ha some mechanism is required o inroduce more exploraion in he emporal-difference raining. The general approach is no wihou difficulies. The whole process is exremely compue inensive. In addiion, here are a number of parameers he experimener mus se ha need o be compared a various levels. This only makes he long raining imes more problemaic. Self-play can lead he agens o be rained on a specific ype of play ha may no generalize, as wih he single sui approach for TD. 5.1 Furher Research The assumpion ha he TD raining approach for his ask does no require explici exploraion may no be valid. Furher esing wih an epsilon-greedy or similar approach added o he raining will deermine if addiional exploraion can improve performance. Cerainly an exended period of experimenaion will be worhwhile. The basic impac of he emporal-difference parameers needs o be sudied in a more sysemaic way. Tesauro (1992) even suggess ha he learning rae be gradually decreased during he raining process in somehing akin o simulaed annealing. I has been suggesed ha gamma values slighly less han one migh help. The impac of he basic parameers for he evoluionary approach can be sudied. However, here are a number of more sophisicaed algorihms ha can be ried as well. Boh approaches may benefi from several refinemens. Changing he ANN opology wih four inpus for each card may produce beer resuls. In addiion, more hidden unis in he ANN may help. The basic raining and measuremen approach should be consisenly se for eiher bes oal score or mos games won. In he policy for deermining wheher o selec from he draw or discard pile, i would be beer o use he mean raher han median expeced value. Coninued experimenaion can cerainly benefi from faser raining imes. The maximum urns before a draw is called migh be lowered. Opimizaion of code o play he game and perform he back-propagaion faser will help o speed he abiliy o experimen. Wih he number of players rained, here is only a hin of wha impac he various parameers have. A more opimized raining program can allow hese o be explored. More raining is also needed o deermine if over-raining is an issue. I will be ineresing o add he feaure allowing a player o lay off cards when is opponen knocks. This will make he game more realisic and challenge he player o pay more aenion o is opponen's cards. References Gibson, Waler. (1974). Hoyle s Modern Encyclopedia of Card Games. New York: Doubleday. Konik, Clifford. (2003). Training Techniques for Sequenial Decision Problems. Forhcoming Maser of Science Thesis. Universiy of Colorado a Colorado Springs. Pollack J. B., Blair A. D. & and Land M. (1996). Coevoluion of a Backgammon Player. Proceedings of he Fifh Inernaional Conference on Arificial Life. SNNS Sugar Neural Nework Simulaor, Universiy of Sugar and Universiy of Tübingen, hp://www-ra.informaik.uni-uebingen.de/snns/ Suon, Richard and Andrew Baro. (1998). Reinforcemen Learning, An Inroducion. Cambridge: MIT Press. Tesauro, Gerald. (1995). Temporal Difference Learning and TD-Gammon. Communicaions of he ACM. Vol 38:3, Tesauro, Gerald. (1992). Pracical Issues in Temporal Difference Learning. Machine Learning. Vol 8,