SEARCH VS KNOWLEDGE: EMPIRICAL STUDY OF MINIMAX ON KRK ENDGAME

Transcription

1 SEARCH VS KNOWLEDGE: EMPIRICAL STUDY OF MINIMAX ON KRK ENDGAME Aleksander Sadikov, Ivan Bratko, Igor Kononenko University of Ljubljana, Faculty of Computer and Information Science, Tržaška 25, 1000 Ljubljana, Slovenia, Abstract: Key words: This article presents the results of an empirical experiment designed to gain insight into what is the effect of the minimax on the evaluation function. The experiment s simulations were performed upon the KRK chess endgame. Main result is that dependencies between evaluations of sibling nodes in a game tree and an abundance of possibilities to commit blunders present in the KRK endgame are not enough to explain the success of the minimax in practical game-playing as was previously believed. The article argues that correlation between a high branching factor in the game tree and having a winning move available is something that minimax implicitly takes into account and that this is at least partially responsible for its success. Second part of the article presents weighted minimax, an attempt at exploiting dependencies between evaluations of sibling nodes in a game tree and shows that its results could be promising. minimax principle, KRK chess endgame, weighted minimax 1. INTRODUCTION Over twenty years ago Beal (1980) set out to analyze whether and why values backed up from minimax search are more trustworthy than the heuristic values themselves. He constructed a simple mathematical model to analyze the minimax algorithm. To his surprise the analysis of the model showed that backed-up values are actually somewhat less trustworthy than heuristic values themselves. He writes: This result is disappointing. It was hoped that the analysis would show that the probability of error reduced with

2 2 Aleksander Sadikov, Ivan Bratko, Igor Kononenko backing-up. A couple of years later two articles (Beal, 1982; Bratko and Gams, 1982) simultaneously conducted further analysis into why minimax does yield good results in practical game-playing while apparently backedup values seem less reliable and reached the same conclusion. They argued that the true values of sibling nodes in a game tree are not independent of one another. This clustering of similar values is a major feature in practical games and it was this phenomenon that Beal s mathematical model did not account for. The problem with the minimax paradigm under sibling values independence was also confirmed by Nau (1982, 1983), who called this a search-depth pathology in game trees. His simulation (Nau, 1982) where strong dependencies between sibling nodes were introduced concurs that this can cause search-depth pathology to disappear. However, Pearl (1984) partly disagrees with the conclusion reached by Beal, Bratko, Gams and Nau, and claims that while strong dependencies between sibling nodes in a game tree can eliminate the pathology, practical games like chess do not possess dependencies of sufficient strength. He points out that there are few chess positions so strong that they cannot be spoiled abruptly if one really tries hard to do so. He concludes that the success of minimax in game-playing programs is based on the fact that common games do not possess a uniform structure but are riddled with early terminal positions, colloquially named blunders, pitfalls or traps. Close ancestors of such traps carry more reliable evaluations than the rest of the nodes, and when more of these ancestors are exposed by the search, the decisions become more valid. Moreover, Schrüfer (1986) and its follow-up (Althöfer, 1989) did some further analysis of pathology in game trees. Especially interesting is their observation that to avoid pathology, an evaluation function must, among other things, have negligible probability of underestimating a position from the perspective of the player to move. All of the above studies have two things in common: (a) they accept the empirical evidence that minimax principle works in practical game-playing programs and (b) they try to mathematically model the minimax algorithm and theoretically deduce what happens when heuristic values assigned to leaves are backed-up towards the root of the game tree. To make such mathematical analysis feasible the researchers are forced to make certain assumptions about the game they model and to make simplifications in their model. Thus, the results of these models are always to be viewed with the acknowledgement of this assumptions and simplifications in the back of one s mind. In contrast to that, our approach in this article is to take (part of) a real game with a real evaluation function and observe empirically what is going on when we change the search depth and the quality of evaluation function. We have at our disposal an absolutely correct evaluation function which we can corrupt in a controlled way. We also have a minimax search

3 Search vs Knowledge: Empirical Study of Minimax on KRK Endgame 3 engine that is capable of searching right down to the terminal nodes of the game tree. The next chapter describes our choice of the game, the evaluation function and its artificial corruption, as well as the search engine. Chapter 3 presents the results for various settings of our simulation parameters. Chapter 4 unveils the outcome of our attempt to improve the minimax search by letting it acknowledge the dependencies between sibling nodes in a game tree. At the end we give our conclusions and some ideas for further work. 2. EXPERIMENTAL DESIGN We have decided to center our simulations on a simple subset of chess: the KRK endgame. In this endgame White has a king and a rook, while Black has only his king. The goal for White is to mate his opponent, striving to do so in as little moves as possible. There are two possible outcomes of this endgame: a win for White or a draw. While the KRK endgame is very simple, it still possesses all the interesting attributes: positions are of various difficulties (measured in the number of moves to mate), there surely exist dependencies between the values of sibling nodes in a game tree, and there is a possibility of blunders and early termination for both sides (stalemate or losing a rook for White; premature mate for Black). We are interested in the quality of play for White under different conditions. Therefore, unless stated otherwise, we always look at things from the White player s perspective. For the KRK endgame we have at our disposal an absolutely correct evaluation function. It is in the form of a database that consists of all possible legal positions and their values. The database can be obtained from UCI Machine Learning Repository (Blake and Merz, 1998). The value for each position tells us how many moves are needed to reach mate in the case that both players play optimally. The positions in the database always assume it is Black s turn to move. There are two special cases: value 0 means Black is mated and value 255 means that Black has a draw (either the position is a stalemate or Black can capture the white rook). The database consists of 28,056 positions. There are actually over 200,000 legal KRK positions, however board symmetries allow for such a reduction. Detailed description of the database and board symmetries is given in (Bain, 1992). Our version of the database is implemented as an array of 28,056 cells and can be viewed as a sort of transposition table. Apart from positions having special evaluations of 0 or 255, there are 25,233 positions divided into 16 levels of difficulty. Positions from level 1 require one move (2 plies) to mate (assuming optimal play); positions from level 2

4 4 Aleksander Sadikov, Ivan Bratko, Igor Kononenko require two moves to mate, and so on. Positions from the most difficult level require 16 moves (32 plies) to mate. Different levels have different number of positions. There are 4,553 positions of level 14 and only 390 positions of level 16. Figure 1 shows how many cases (positions) are left unsolved if we applied searches of various depths without any knowledge apart from the rules of the game. The term unsolved in this context means that White has to make a move without knowing at that time the complete move-tree that guarantees a mate. The curve starts to fall significantly between depths of 14 and 20 plies and after ply 20 it steeply drops towards zero. Unsolved cases Number of cases Search depth Figure 1. Number of unsolved cases as a function of search depth For the purpose of our experiments we corrupted the ideal evaluation function in a controlled manner. Our method of doing this is as follows. We take a position value and add to it a certain amount of Gaussian noise. Formula (and a plot): P 2 2 ( x µ ) / ( ) ( 2 σ x = e )

5 Search vs Knowledge: Empirical Study of Minimax on KRK Endgame 5 gives the probability P(x)dx that given the correct evaluation and standard deviation the new (corrupted) evaluation x will take on a value in the range [x, x + dx], which is a real number. The error of new evaluation is x. We do this for all positions in the database. The corruption is symmetrical, meaning that there is practically equal chance that the new evaluation will be optimistic or pessimistic. Exceptions are positions closer to mate, where there is a higher probability of pessimistic evaluations; especially with higher degrees of corruption. This is due to the fact that we do not allow x to become negative (we redo the corruption of the position until it is not negative). The level of corruption is controlled by the parameter, which is in fact standard deviation and controls how dispersed are the corrupted values x around the correct values (the width of the hill on the plot above). Standard deviation is measured in moves. For example, if equals 0.5, this means that approximately two thirds of corrupted evaluations are within 0.5 moves around the true evaluation and over 95% of corrupted evaluations are within 1.0 move (two standard deviations) around the true evaluation. To be able to compare the quality of initial knowledge (evaluation function) to the quality of knowledge after backing up the values with the minimax algorithm, we have to be able to calculate the standard deviation after minimaxing. This is easy, because our search algorithm returns the backed-up values from a fixed search depth for every unique KRK position in an array exactly the same as our initial database. This array is in fact our backed-up evaluation function. We thus have one such array for every search depth from 0 (initial database) to 32 ply. After obtaining such an array, we calculate with the formula: σ = 1 N ( x i µ i ) N i= 1 2 where x i is the backed-up corrupted value and i is the true value for position i. N is the number of positions in the array. This gives us a tool to directly monitor how minimax affects the quality of evaluation function. Our search engine is the standard fixed-depth minimax search. We were able to search up to the maximal needed search depth of 32 plies (after 32 plies or 16 moves all the positions are solved by search alone) by exploiting the fact that the KRK endgame only has a comparatively small number of unique (under symmetries) positions (28,056) which we can all remember in a sort of transposition table. We start at depth 0 by loading the values from a (corrupted) database, then move on to depth 2, perform a 2-ply minimax search and use the results of the previous depth as evaluations of the leaves, store results of depth 2 search, move on to depth 4 and so on. For terminal

6 6 Aleksander Sadikov, Ivan Bratko, Igor Kononenko nodes (mate or draw) we use their true game values and not the evaluation from the database / transposition table. 3. THE RESULTS OF THE EXPERIMENTS Figure 2 shows what happens to the quality of the evaluation function when we change the search depth. The x-axis represents the search depth measured in plies and the y-axis represents the standard deviation measured in moves. Each curve in the graph represents a different evaluation function they differ in the level of their corruption (the initial ). The legend marks these different evaluation functions with the size of their initial. The best way to separate the curves is to look at their initial corruption. The last three functions, marked with rand, are added for comparison their initial values are random real numbers between 0 and 50. They represent zero knowledge evaluation functions. Search vs Knowledge Knowledge corruption level Search depth 0.25f 0.50f 0.75f 1.00f 1.50f 2.00f 3.00f 5.00f 10.00f 15.00f 20.00f 25.00f rand 1 rand 2 rand 3 Figure 2. Influence of search depth on quality of evaluation function It seems that we have to divide the evaluation functions into two groups: in the first group we have evaluation functions with (relatively) low initial error and in the second group those with a high initial error. The first group is a realistic model of real-life evaluation functions, while the second group contains evaluation functions with (almost) zero knowledge. We can observe that evaluation functions with initial less than 3.0 do not exhibit the

7 Search vs Knowledge: Empirical Study of Minimax on KRK Endgame 7 tendency to speedily drop towards 0 (perfect knowledge). They drop slightly or remain on the same level until very deep search is invoked. One has to bear in mind that when search depth reaches 20 ply a large portion of the positions is solved by the search alone and that heavily influences the assessment of for very deep searches. That is probably the only reason why they eventually do drop. On the other hand, we can see that evaluation functions with initial of 10 (even 5) or more do exhibit the expected behaviour of speedily dropping with increased search depth. As the graph shows they behave basically the same as random evaluation functions. From this we can conclude that with corruption they lost just about all their knowledge and are in fact random evaluation functions themselves. To circumvent the effect of deep searches, where many of the positions are solved by the search alone, we have made another analysis in which we disregarded the effect of the solved positions. Figure 3 shows the results of this analysis. Everything is done exactly the same as before, the only exception is in calculating the new standard deviation. Positions, which are solved by the search, are omitted from the calculation, thus reducing the number of positions N. If we take a closer look at Figure 3 we can see that there is not much change in comparison with Figure 2. The evaluation functions from group two behave more or less the same, while evaluation functions from group one decline to drop for a little longer. When looking at Figure 3, one must bear in mind that from search depth 28 onward the curves are unreliable due to small number of positions that are left unsolved by the search and are used to calculate. The end result is that for reasonable evaluation functions (group one) searching deeper does not (significantly) improve the quality of the evaluation function for playing the KRK endgame. The endgame undoubtedly contains dependencies between the values of sibling nodes in a game tree. It is also full of possibilities for blunders on the part of White player. White can, after all, lose his rook in at most two moves if Black plays normally. This means that the two reasons why minimax is effective in practice are present and yet the pathology is still present. On the other hand, evaluation functions from group two, that are (practically) random, benefit a lot from searching deeper. How can all this be explained?

8 8 Aleksander Sadikov, Ivan Bratko, Igor Kononenko Search vs Knowledge (unsolved cases only) Knowledge corruption level Search depth 0.25f 0.50f 0.75f 1.00f 1.50f 2.00f 3.00f 5.00f 10.00f 15.00f 20.00f 25.00f rand 1 rand 2 rand 3 Figure 3. Influence of search depth on quality of evaluation function (disregarding search-solved cases) The answer why random evaluation functions from group two benefit a lot from searching deeper is given in (Beal and Smith, 1994). The authors have conducted an experiment where they pitched a chess playing program using a random evaluation function against a program using a random evaluation function and a fixed-depth minimax search. The latter program completely dominated. We can confirm this result because our simulation of the first program returned a constant standard deviation of around 20 for all search depths up to 26 ply. The explanation for this phenomenon is that there exists a correlation between a high branching factor in the game tree and having a winning move available. Minimax implicitly takes into account the mobility parameter at all depths of the game tree (which is not the same as using an evaluation function explicitly counting available moves). Further theoretical investigation of this phenomenon is given in (Levene and Fenner, 2001). This phenomenon might also be another part of answer to the question why minimax is successful in practice. It is possible that it has some effect on all evaluation functions (not just random ones) and it is possible that it prevents the error of the evaluation function to increase unreasonably. Could it be that with the initial knowledge of the evaluation function at least preserved, the minimax s ability to prevent falling into traps might be enough for the minimax principle to be successful?

9 Search vs Knowledge: Empirical Study of Minimax on KRK Endgame 9 Up to this point, we did not say anything about how well would a computer program using one of our corrupted evaluation functions actually play. The answer is given in Figure 4. We have played out all unique KRK positions except the ones with special values of 0 or 255, in total 25,233 positions. White player was guided by a corrupted evaluation function. Black player was always playing optimally. We measured the quality of play as the average number of moves above what an optimal White player (using a non-corrupted database) would need. This statistic is computed as the difference between number of moves spent by White for all positions and number of moves needed for all positions using optimal play, divided with the number of positions (25,233). Since it is hard to separate different curves in Figure 4: curve 0.25 is practically on the x-axis and therefore hardly visible, other curves then follow from left to right by increasing level of corruption. The exception is curve which is actually the last one on the right. Noise effect on actual play Average moves above optimum Search depth 0.25f 0.50f 0.75f 1.00f 1.50f 2.00f 3.00f 5.00f 10.00f 15.00f 20.00f 25.00f Figure 4. Quality of play using corrupted evaluation functions In Figure 4 we can see that evaluation function with initial corruption level of 0.25 moves provides practically optimal play starting already at search depth 0. Other evaluation functions from group one play well (spending no more than 1 move more than optimal play) from search depth 6-8 onward. Evaluation functions from group two play well only after reaching search depth 14.

10 10 Aleksander Sadikov, Ivan Bratko, Igor Kononenko 4. WEIGHTED MINIMAX Since there are dependencies between evaluations of the sibling nodes in a game tree we were curious if it is conceivable to exploit them. We introduced the idea of a weighted minimax. It works just like a normal minimax search, except that the evaluation of a node takes into account the values of its sibling nodes. The closer the sibling s evaluation is to the evaluation of the node under investigation, the bigger weight is assigned to it, which in turn means that it has more effect on the evaluation. If a terminal node is reached by the search, its evaluation remains unchanged. Only the best of all the siblings needs to have its value modified as described. Figure 5 explains how weighted minimax computes the evaluation of a node. It starts by ordering the siblings by their evaluations from best to worst. Then it calculates the weighted minimax evaluation weval for the best sibling using the following formula: weval ( Pos) 1 eval 3 i i = 1 3 i i ( Pos ) where eval(pos i ) is the evaluation of sibling i. i Figure 5. Calculation of weighted minimax The results of the simulation using weighted minimax in the form described above are presented in Figure 6. Group two evaluation functions drop as with regular minimax, they even seem to drop a little faster. On the other hand, group one evaluation functions exhibit an interesting behaviour. At first (up to ply 4-8) they drop significantly, which means that weighting is useful for these search depths. The exception is the initially least corrupted evaluation function (0.25f). Then they begin to rise back up, well past the point of their initial corruption. For larger search depths, weighting clearly is harmful. In general, it seems that the worst the initial knowledge, the more

11 Search vs Knowledge: Empirical Study of Minimax on KRK Endgame 11 beneficial weighting is at low search depths and vice versa. If this is true, weighting could serve either to improve the search evaluations of a gameplaying program or to confirm that the existing evaluation is good enough. Weighted minimax (version 1) 6.0 Knowledge corruption level f 0.50f 0.75f 1.00f 2.00f 5.00f 20.00f Search depth Figure 6. Weighted minimax, version 1 We have done one further experiment with weighted minimax. We were afraid that bad moves which are present in practically all positions get too much influence despite the fact that they are given low weights. We wondered what would happen if we were to fully disregard their influence on the evaluation. Second version of our weighted minimax algorithm works exactly the same as the first version, except that it disregards those siblings with evaluations worse by a certain threshold than the evaluation of the node under investigation. In our experiment we decided to set the threshold at 3.0 [moves]. The results of the experiment with second version of weighted minimax algorithm are presented in Figure 7. We can see that they do not differ a lot from the results of the first experiment. Main difference is that at low search depths there is slightly more benefit from using the weighted minimax. We believe that the gains could be further increased by fine tuning the threshold value as well as the function that assigns weights to sibling nodes.

12 12 Aleksander Sadikov, Ivan Bratko, Igor Kononenko Weighted minimax (version 2) 6.0 Knowledge corruption level f 0.50f 0.75f 1.00f 2.00f 5.00f 20.00f Search depth Figure 7. Weighted minimax, version 2 At the moment we are unable to give a credible explanation why weighted minimax behaves as it does. Our hypothesis is the following. Weighting helps an erroneous evaluation function up to the point where the function becomes quite good (this point seems to be around = 0.4). When the evaluation function is good, further weighting becomes harmful much like it is harmful for terminal positions where we already know the correct evaluation. There is no doubt that the correct evaluation can only be harmed, it cannot be improved. For the evaluation function with initial = 0.25, it is harmful from the start because it is below 0.4 already. Evaluation function with initial = 2 is no exception, it falls until it reaches 0.4, then it rises again. Situations that do not conform to this hypothesis only occur at depths of 26 ply and beyond (evaluation functions with initial = 5 in Figure 6; = 2 and = 20 in Figure 7). However, graphs in Figures 6 and 7 are disregarding the effect of solved positions (just like the graph in Figure 3) and we believe them to be quite unreliable for very high depths (26 ply and beyond). From the results of these experiments we suggest using the weighted minimax might be useful at low search depths (in the top part of the game tree). The idea should be tried in a serious game-playing program.

13 Search vs Knowledge: Empirical Study of Minimax on KRK Endgame CONCLUSIONS AND FURTHER WORK Theoretical studies of the minimax principle in the past have shown that it has a negative effect on the quality of the evaluation function. As the answer why it is nevertheless successful in practice they suggested two reasons: (a) dependencies between evaluations of sibling nodes in a game tree and (b) existence of traps that cause early terminations of the game. We have taken the opposite approach to the problem; we tried to empirically check these conclusions using the KRK chess endgame. We can confirm that minimax algorithm is in fact poor preserver of the knowledge built into the evaluation function. Yet, regardless of that, it proved to still be successful in actual play (Figure 4). It turns out that even with dependencies between evaluations of sibling nodes in a game tree and an abundance of possibilities to commit blunders present in our endgame, the anomaly still existed. Our belief is that the result reached by Beal and Smith (1994) is of great importance; we are convinced that it is part of the answer why minimax is successful. It the second part of the article we presented weighted minimax, an attempt to exploit dependencies between evaluations of sibling nodes in a game tree. It seems to work, however we need to know when it reaches the point where it becomes harmful. Fine tuning of its parameters might further increase its benefits. In view of the presented results further work should focus on testing whether our hypothesis regarding the behaviour of weighed minimax stands. It would also be very interesting to recheck our results using a more complex game perhaps a KRKN or KRKP chess endgame. REFERENCES Althöfer, I. (1989) Generalized minimax algorithms are no better error correctors than minimax itself, Advances in Computer Chess, 5 (ed. D.F. Beal), pp , Elsevier Science Publishers. Bain, M. (1992) Learning optimal chess strategies, Proc. Intl. Workshop on Inductive Logic Programming (ed. S. Muggleton), Institute for New Generation Computer Technology, Tokyo, Japan. Beal, D.F. (1980) An Analysis of Minimax, Advances in Computer Chess, 2 (ed. M.R.B. Clarke), pp , Edinburgh University Press. Beal, D.F. (1982) Benefits of minimax search, Advances in Computer Chess, 3 (ed. M.R.B. Clarke), pp. 1-15, Pergamon Press. Beal, D.F. and Smith, M.C. (1994) Random Evaluations in Chess, ICCA Journal, Vol. 17(1), pp Blake, C.L. and Merz, C.J. (1998) UCI Repository of machine learning databases [ Irvine, CA: University of California, Department of Information and Computer Science.

14 14 Aleksander Sadikov, Ivan Bratko, Igor Kononenko Bratko, I. and Gams, M. (1982) Error analysis of the minimax principle, Advances in Computer Chess, 3 (ed. M.R.B. Clarke), pp. 1-15, Pergamon Press. Levene, M. and Fenner, T.I. (2001) The Effect of Mobility on Minimaxing of Game Trees with Random Leaf Values, Artificial Intelligence, 130, pp Nau, D.S. (1983) Pathology on Game Trees Revisited, and an Alternative to Minimaxing, Artificial Intelligence, 21(1, 2), pp Nau, D.S. (1982) An Investigation of the Causes of Pathology in Games, Artificial Intelligence, 19, pp Pearl, J. (1984) Heuristics: intelligent search strategies for computer problem solving, Addison-Wesley Publishing Company. Schrüfer, G. (1986) Presence and absence of pathology on game trees, Advances in Computer Chess, 4 (ed. D.F. Beal), pp , Pergamon Press.