Optimizing the State Evaluation Heuristic of Abalone using Evolutionary Algorithms

Transcription

1 Optimizing the State Evaluation Heuristic of Abalone using Evolutionary Algorithms Benjamin Rhew December 1, Introduction Heuristics are used in many applications today, from speech recognition technology to filters used in fighting spam. They are so prevalent that everyone comes into contact with them during everyday life. Yet, there is still room for improvement. The problem with heuristics is that they are usually either able to find guaranteed good solutions with a poor running time, or run fast but find poor solutions. Optimizing heuristics can help with these inherent problems, and make applications using them more effective. However, the optimization of a heuristic is difficult. Hand-tuning of a heuristic can provide some benefits, but often the heuristic being optimized is complex, and a human guessing at optimal parameters can produce sub-optimal performance. Also, non-obvious parameters can have a drastic effect on the performance of a heuristic. Because of these limitations, an automated method of testing different tunings of a heuristic is sought. One such method of automation is using an evolutionary algorithm. This allows many different possible sets of heuristics to be tested, especially ones with non-obvious parameter sets. This, however, can take a long time, depending on how complex the original problem is, and how long it takes to evaluate possible heuristics. To prove the concept, one must start with a simple problem. One such possible problem is the optimization of the evaluation function of abalone, a computerized version of a boardgame. It already has a partially hand-tuned heuristic to start with, so the evolutionary algorithm will only modify the preexisting heuristic. The main goal is to have the EA-tuned heuristic be proven to be better than the evaluation function it is based on, using the exact same tree search algorithm. This means that the EA-tuned heuristic needs to win against the original heuristic playing both sides. A secondary goal would be to beat a separate, better heuristic and tree-search implementation, such as Christopher Walker s winning abalone program. The rest of this paper is organized as follows. Section 2 will go over prior research that has been done. Section 3 will explain the game of abalone. Sec- 1

2 tion 5 will explain the design and implementation of the evolutionary algorithm. Section 6 will cover the experimental setup used for testing purposes. 2 Prior Research There has been a good amount of research in trying to optimize heuristics for chess playing programs. This is more complex than abalone, but it provides a good starting point. [GAKB02] talks about using a hybrid GP/ES system to evolve better heuristics for chess playing programs. They show that it is possible to evolve better heuristics than plain algorithms. However, they do not have the individuals playing among themselves, which is something that will be implemented. [KW01] leans more toward the approach taken in this paper, because they are trying to fine-tune evaluation function parameters. However, they don t use standard ways of selecting parents. They are trying to use population dynamics to get more fit individuals. They do show there is quite a bit of improvement from the original to the fine-tuned parameter set. These are just a few examples of what can be found. Other papers include information on using neural networks ([Thr95]), temporal differences ([Sut88]), and other topics in combination with chess, which provide good background for methods of developing or optimizing a heuristic. 3 Backgrond on Abalone Abalone is a board game that was developed in the 1990s. It can be thought of as sumo wrestling with marbles, since the point of the game is to push six of the opponent s marbles off the board. The board is a hexagonal board, with 5 spaces on each edge. Each side starts with 14 marbles. Up to three marbles in a line can be moved in any of the six possible directions, as long as there is nothing blocking the line of marbles. Exceptions to this are when pushing opponent marbles, which can only be done when one has numerical superiority. For example, white is trying to push black, when black has two marbles in a line. White must have three marbles to push black. But if black has only one marble, either two or three marbles can be used to push black. If black has three marbles, white cannot push black, because there is no numerical superiority. 4 Two-Pool EAs One of the important notes about the design of the Evolutionary Algorithm is that it uses what is called a two-pool EA. These are EAs that have two separate population pools for varying reasons. One reason to have two pools is to have two genders, which can have different fitness functions. Parents would be drawn from both pools, and after reproduc- 2

3 tion, children would be sent to either pool, depending on certain factors. Having two genders can help certain types of problems. Another reason would be to model the predator-prey relationship. Again, the the two pools would have different fitness functions, but in this case the two pools would not be able to interbreed. The fitness of both pools would be a moving target, since the predators might evolve a better way to catch prey, with the prey evolving ways to evade predators. A third reason is the reason the EA design is using a two pool EA. This reason is to have children and adults. With this method, the fitness function is generally the same between both pools, but children cannot reproduce or die until they reach a certain age, when they are moved into the adult pool. This helps to provide the children a way to get a good measure of their fitness before being moved to the adult pool, whichs helps prevent poor children from killing bad adults, and bad adults from killing good children. 5 The EA Design There are three main methods that can be used for representation of individuals. Each new one is an extension of the previous method. The first gene method is very simple. Basically, for each separate heuristic in the evaluation function, a single floating point gene is used. Each heuristic would be multipled by the value stored in the gene during evaluation. The second gene method is a bit more complex. For the board heuristics, each ring of the board would have a separate floating point gene. The rest of the heuristics would still have a single gene, since they are based on single numbers. This method should provide better utilization of the board heuristics. Finally, the third method is to have each of the board heuristics fully realized as floating point genes in the genetics of the possible solutions. The rest of the heuristics would still be based on a single gene, as in the second method. This method should provide the best utilization of the board heuristics. What is chosen for this particular experiment is the second method. This provides a compromise between the compactness and complexity of the individual. The more compact an individual is, the better the memory footprint. The more complex an individual is, on the other hand, the better the heuristic that is developed, at least presumably. For the evolutionary process, what is known as a two-pool EA will be used, which is described in a bit more detail in Section 4. Child individuals will first go through a process where they can t be killed, thereby separating them from the rest of the population. They will not reproduce during this period - this is to get a good estimate of the fitness of the individuals. This is to prevent possible good solutions from being overridden by previous good individuals, or new bad solutions overriding old, good solutions. Fitness can be done in two methods. The first method is based on the number of wins an individual has. If an individual wins a game, then the fitness will be increased by one. Similarly, if one loses a game, then the fitness will be 3

4 decreased by one. If a game goes through a certain number of moves, it will be considered a stalemate and each solution will have its fitness increased by 0.5. Each individual will play a single random opponent twice, once for each side. Initial fitness prior to generation 1 will be determined by playing the original evaluation heuristic. A second fitness function is very similar. It uses the above fitness measure, but it also divides the value by the number of games played. That way, the fitness is limited in maximum value, allowing a chance for more competition. The second method of determining fitness of an individual is what is used. This is because it avoids the ever-climbing fitness of method one, which helps in other portions of the EA (such as parent selection and survival selection). Parent selection is rank-based, with the rank being based on fitness. Higher fitness individuals will therefore have a better chance of reproduction, but lowfitness individuals will also have a chance to contribute to the gene pool. Mutation is based on a gaussian random variable. If a gene of an individual is chosen to be mutated, the gene s value will be added to by the variable, centered around zero. Survival selection is stochastic, with the worst individuals having the highest chance of dying off. However, the best individual will survive no matter what, so it will be also elitist. Recombination is done by uniform crossover, which is just n-point crossover with n being the size of the individual. This allows the most diverse method of transmitting genetic material to the children. Implementation of this is the same as any generic EA, except for the fitness function. The fitness depends on using an Abalone engine that will take the individuals and use them in the heuristic function. 6 Experimental Setup The experimental setup is just as important as choosing good design elements. For the testing phase, the following parameters are used: Population Size - This is set to 20, to give a good set of random individuals, without being too large in size. This saves on computational time. Length of Time as a Child- This is set to 10 generations, so that each child individual will have played 20 games before being considered an adult. This should give a good measure of fitness. Number of Parents - This is set to 4, since a steady-state EA is what is desired. Number of Children - This is set to 2, with one child being produced per pair of parents. Initialization of Individuals - This will be done randomly, using small floating point values. 4

5 Recombination Percentage - This is set to 100%, since recombination should be done every time. Mutation Percentage - This is set to 1/n, where n is the size of an individual. This should provide a good chance of mutation per gene. Gaussian Mutation Value - This is set to 2, so that the mutation process is a bit more controlled. 7 Results Unfortunately, there still hasn t been time to finish results. This will be changed in the future. 8 Conclusion Currently there is nothing to conclude upon. References [GAKB02] Roderich Groß, Keno Albrecht, Wolfgang Kantschik, and Wolfgang Banzhaf. Evolving chess playing programs. In W. B. Langdon, E. Cantú-Paz, K. Mathias, R. Roy, D. Davis, R. Poli, K. Balakrishnan, V. Honavar, G. Rudolph, J. Wegener, L. Bull, M. A. Potter, A. C. Schultz, J. F. Miller, E. Burke, and N. Jonoska, editors, GECCO 2000: Proceedings of the Genetic and Evolutionary Computation Conference, pages , San Francisco, California, Morgan Kaufmann Publishers. [KW01] Graham Kendall and Glenn Whitwell. An evolutionary approach for the tuning of a chess evaluation function using population dynamics. In Proceedings of the 2001 Congress on Evolutionary Computation CEC2001, pages , COEX, World Trade Center, 159 Samseong-dong, Gangnam-gu, Seoul, Korea, IEEE Press. [Sut88] Richard S. Sutton. Learning to predict by the methods of temporal differences. Machine Learning, 3:9 44, [Thr95] Sebastian Thrun. Learning to play the game of chess. In G. Tesauro, D. Touretzky, and T. Leen, editors, Advances in Neural Information Processing Systems 7, pages The MIT Press, Cambridge, MA,