Theory of Computer Games: Concluding Remarks

Transcription

1 Theory of Computer Games: Concluding Remarks Tsan-sheng Hsu 1

2 Practical issues. The open book. The endgame database. Smart usage of resources. Time Memory Coding efforts Debugging efforts Putting everything together. Software tools Fine tuning Abstract How to know one version is better than the other? Concluding remarks TCG: Concluding remarks, , Tsan-sheng Hsu c 2

3 The open book (1/2) During the open game, it is frequently the case branching factor is huge; it is difficult to write a good evaluation function; the number of possible distinct positions up to a limited length is small as compared to the number of possible positions encountered during middle game search. Acquire game logs from books; games between masters; games between computers; Use off-line computation to find out the value of a position for a given depth that cannot be computed online during a game due to resource constraints. TCG: Concluding remarks, , Tsan-sheng Hsu c 3

4 The open book (2/2) Assume you have collected r games. For each position in the r games, compute the following 3 values: win: the number of games reaching this position and then wins. loss: the number of games reaching this position and then loss. draw: the number of games reaching this position and then draw. When r is large and the games are trustful, then use the 3 values to compute an estimated goodness for this position. Comments: Pure statistically. Can build a static open book. You program may not be able to take over when the open book is over. It is difficult to acquire large amount of trustful game logs. Automatically analysis of game logs written by human experts. [Chen et. al. 2006] Using high-level meta-knowledge to guide the way in searching: Dark chess: adjacent attack of the opponent s Cannon. [Chen and Hsu 2013] TCG: Concluding remarks, , Tsan-sheng Hsu c 4

5 Example: Chinese chess open book (1/3) A total of 28,591 (Red win)+21,072 (Red lose)+55,930 (draw) games. TCG: Concluding remarks, , Tsan-sheng Hsu c 5

6 Example: Chinese chess open book (2/3) Can be sorted using different criteria. Win-lose winning rates... TCG: Concluding remarks, , Tsan-sheng Hsu c 6

7 Example: Chinese chess open book (3/3) A tree-like structure. TCG: Concluding remarks, , Tsan-sheng Hsu c 7

8 Endgame Entering the endgame, it is frequently the case the number of remaining pieces is small; special strategies or heuristics differ from the one used in other phases of the game exist. Solving the endgame by implementing heuristics; systematically enumeration of all possible combinations. TCG: Concluding remarks, , Tsan-sheng Hsu c 8

9 Endgame Databases Chinese chess endgame database: Indexed by a sublist of pieces S, including both Kings. K G M R N C P King Guard Minister Rook Knight Cannon Pawn KCPGGMMKGGMM ( vs. ): the database consisting of RED Cannon and Pawn, and Guards and Ministers from both sides. A position in a database S: A legal arrangement of pieces in S on the board and an indication of who the next player is. Perfect information of a position: What is the best possible outcome, i.e. win/loss/draw, that the player can achieve starting from this position? What is a strategy to achieve the best possible outcome? Given S, to be able to give the perfect information of all legal positions formed by placing pieces in S on the board. Partial information of a position: win/loss/draw; DTC; DTZ; DTR. TCG: Concluding remarks, , Tsan-sheng Hsu c 9

10 Usage of Endgame Databases Improve the skill of Chinese chess computer programs. KNPKGGMM ( vs. ) Educational: Teach people to master endgames. Recreational. TCG: Concluding remarks, , Tsan-sheng Hsu c 10

11 An Endgame Book TCG: Concluding remarks, , Tsan-sheng Hsu c 11

12 Books TCG: Concluding remarks, , Tsan-sheng Hsu c 12

13 TCG: Concluding remarks, , Tsan-sheng Hsu c 13

14 TCG: Concluding remarks, , Tsan-sheng Hsu c 14

15 Definitions State graph for an endgame H: Vertex: each legal placement of pieces in H and the indication of who the current player (Red/Black) is. Each vertex is called a position. May want to remove symmetry positions. Edge: directed, from a position x to a position y if x can reach y in one ply. Characteristics: Bipartite. Huge number of vertices and edges for non-trivial endgames. Example: KCPGGMMKGGMM has positions and about edges. TCG: Concluding remarks, , Tsan-sheng Hsu c 15

16 Overview of Algorithms Forward searching: doesn t work for non-trivial endgames. AND-OR game tree search. Need to search to the terminal positions to reach a conclusion. Runs in exponential time not to mention the amount of main memory. Heuristics: A, transposition table, move ordering, iterative deepening... OR search... AND search TCG: Concluding remarks, , Tsan-sheng Hsu c 16

17 Retrograde Analysis (1/2) First systematic study by Ken Thompson in 1986 for Western chess. Retrograde analysis ( 回溯分析 ) Algorithm: List all positions. Find all positions that are initially stable, i.e., solved. Propagate the values of stable positions backward to the positions that can reach the stable positions in one ply. Watch out the and-or rules. Repeat this process until no more changes is found. TCG: Concluding remarks, , Tsan-sheng Hsu c 17

18 Retrograde Analysis (2/2) Critical issues: time and space trade off. Information stored in each vertex can be compressed. Store only vertices, generate the edges on demand. Try not to propagate the same information. backward propagation terminal positions TCG: Concluding remarks, , Tsan-sheng Hsu c 18

19 Stable Positions Another critical issue: how to find stable positions? Checkmate, stalemate, King facing King. It maybe the case the best move is to capture an opponent s piece and then win. so called distance-to-capture (DTC); the traditional metric is distance-to-mate (DTM). Need to access values of positions in other endgames. For example, KCPKGGMM needs to access KCKGGMM KPKGGMM KCPKGMM, KCPKGGM A lattice structure for endgame accesses. Need to access lots of huge databases at the same time. [Hsu & Liu, 2002] uses a simple graph partitioning scheme to solve this problem with good practical results. TCG: Concluding remarks, , Tsan-sheng Hsu c 19

20 An Example of the Lattice Structure KCPKGGMM KCKGGMM KPKGGMM KCPKGMM KCPKGGM KKGGMM KC KGMM KCKGGM... KKGMM KC KMM KC KGM KKMM KC KM KKGM KC KG KKM KC K KKG KC K... KK TCG: Concluding remarks, , Tsan-sheng Hsu c 20

21 Cycles in the State Graph (1/2) Yet another critical issue: cycles in the state graph. Can never be stable. In terms of graph theory, a stable position is a pendant in the current state graph; a propagated position is removed from the sate graph; no vertex in a cycle can be a pendant. cycle in the state graph TCG: Concluding remarks, , Tsan-sheng Hsu c 21

22 Cycles in the State Graph (2/2) For most games, a cyclic sequence of moves means draw. Positions in cycles are stable. Only need to propagate positions in cycles once. For Chinese chess, a cyclic sequence of moves can mean win/loss/draw. Special cases: only one side has attacking pieces. Threaten the opponent and fall into a repeated sequence is illegal. You can threaten the opponent only if you have attacking pieces. The stronger side does not need to threaten an opponent without attacking pieces. All positions in cycles are draws. General cases: very complicated. TCG: Concluding remarks, , Tsan-sheng Hsu c 22

23 Previous Results Retrograde Analysis Western chess: general approach. Complete 3- to 5-piece, pawn-less 6-piece endgames are built. Selected 6-piece endgames, e.g., KQQKQP. Roughly positions per endgame. Perfect information bytes for all 3- to 6-piece endgames. Awari: machine and game dependent approach. Solved in the year positions in an endgame. Using parallel machines. Win/loss/draw. Checkers: game dependent approach positions in an endgame. Currently the largest endgame database of any games using a sequential machine. Win/loss/draw. Solved in the year 2007 with a total endgame size of Many other games. TCG: Concluding remarks, , Tsan-sheng Hsu c 23

24 Results Chinese Chess Earlier work by Prof. S. C. Hsu ( some other researchers in Taiwan. ) and his students, and KRKGGMM ( vs. ) [Fang 1997; master thesis] About positions; Perfect information. Memory-efficient implementation: general approach. KCPGMKGGMM ( vs. ) [Wu & Beal 2001] About positions; Perfect information. KCPGGMMKGGMM ( vs. ) [Wu, Liu & Hsu 2006] About positions; seconds per position; Perfect information. The largest single endgame database and the largest collection reported. Verification [Hsu & Liu 2002] Special rules: larger. more likely to be affected when endgames get TCG: Concluding remarks, , Tsan-sheng Hsu c 24

25 Problems and Solutions Need to solve the problem of finding cycles in a graph. Modeling using graph theory. Using previous knowledge from graph theory. Need to solve the problem of requiring a huge space o store the database being constructed. Carefully partition the database into disjoint portions so that only only the needed parts are loaded into the memory. Using combinatorial properties to do the partition. TCG: Concluding remarks, , Tsan-sheng Hsu c 25

26 Using resources: time and others Time is the most critical resource [Hyatt 1984] [Šolak and Vučković 2009]. Watch out different timing rules. An upper bound on the total amount of time can be used. It is hard to predict the total number of moves in a game in advance. However, you can have some rough ideas. Fixed amount of time per ply. An upper bound T 1 on the total amount of time is given, and then you need to play X plys every T 2 amount of time. System issues: CPU time usage or wall clock? Thinking style of human players. Using almost no time while you are in the open book. More time is spent in the beginning immediately after the program is out of the book. Stop searching a path further when you think the position is stable in the middle game. In the endgame phase, use more time in critical positions or when you try to initiate an attack. Do not think at all if you have only one possible logical move left. TCG: Concluding remarks, , Tsan-sheng Hsu c 26

27 Pondering Pondering: Use the time when your opponent is thinking. Guessing and then pondering. System issues. How interrupt is handled? Polling every now and then or triggered by events? How pondering is done. In your turn, keep the first 2 plys m 1 and m 2 in the PV you obtained. You choose to play m 1, and then it s the opponent s turn to think. In pondering, you can assume the opponent plays m 2. Then you continue to think at the same time your opponent thinks. If the opponent plays m 2, then you can continue the pondering search in your turn. If the opponent plays other moves, then you restart a new search. TCG: Concluding remarks, , Tsan-sheng Hsu c 27

28 Using other resources Memory Using a large transposition table occupies a large space and thus slows down the program. A large number of positions are not visited too often. Using no transposition table makes you to search a position more than once. CPU Do not fork a process to search branches that have little hope of finding the PV even you have more than enough hardware. Other resources. TCG: Concluding remarks, , Tsan-sheng Hsu c 28

29 Putting everything together Game playing system GUI Use some sorts of open books. Middle-game searching: usage of a search engine. Evaluation function: knowledge. Main search algorithm. Enhancements: transposition tables, Quiescent search and possible others. Use some sorts of endgame databases. Debugging and testing TCG: Concluding remarks, , Tsan-sheng Hsu c 29

30 Software tools Using make to do a better software project management. Using svn or other version control tools to do code maintaining. Using compiler optimization switches to speed up your execution code. gcc -O2 main.c gcc -O3 main.c Object code may not be stable using aggresive optimization flags. Using gdb or other debugging tools to debug. Using gprof or any other profiling bottleneck of your code execution. tools to find out the gcc -pg coins.c a.out gprof a.out gmon.out TCG: Concluding remarks, , Tsan-sheng Hsu c 30

31 Profiling TCG: Concluding remarks, , Tsan-sheng Hsu c 31

32 Call graph TCG: Concluding remarks, , Tsan-sheng Hsu c 32

33 Testing You have two versions P 1 and P 2. You make the 2 programs play against each other using the same resource constraints. To make it fair, during a round of testing, the numbers of a program plays first and second are equal. After a few rounds of testing, how do you know P 1 is better or worse than P 2? TCG: Concluding remarks, , Tsan-sheng Hsu c 33

34 How to know you are successful Assume during a self-play experiment, two copies of the same program are playing against each other. Since two copies of the same program are playing against each other, the outcome of each game is an independent random trial and can be modeled as a trinomial random variable. Assume for a copy playing first, P r(game first ) = { p if win q 1 p q if draw if lose Hence for a copy playing second, P r(game last ) = { 1 p q if win q p if draw if lose TCG: Concluding remarks, , Tsan-sheng Hsu c 34

35 Outcome of self-play games Assume 2n games, g 1, g 2,..., g 2n are played. In order to offset the initiative, namely first player s advantage, each copy plays first for n games. We also assume each copy alternatives in playing first. Let g 2i 1 and g 2i be the ith pair of games. Let the outcome of the ith pair of games be a random variable X i from the prospective of the copy who plays g 2i 1. Assume we assign a score of x for a game won, a score of 0 for a game drawn and a score of x for a game lost. The outcome of X i and its occurrence probability is thus P r(x i ) = p(1 p q) if X i = 2x pq + (1 p q)q if X i = x p 2 + (1 p q) 2 + q 2 if X i = 0 pq + (1 p q)q if X i = x (1 p q)p if X i = 2x TCG: Concluding remarks, , Tsan-sheng Hsu c 35

36 How good we are against the baseline? Properties of X i. The mean E(X i ) = 0. The standard deviation of X i is E(X 2 i ) = x 2pq + (2q + 8p)(1 p q), and it is a multi-nominally distributed random variable. When you have played n pairs of games, what is the probability of getting a score of s, s > 0? Let X[n] = n i=1 X i. The mean of X[n], E(X[n]), is 0. The standard deviation of X[n], σ n, is x n 2pq + (2q + 8p)(1 p q), If s > 0, we can calculate the probability of P r( X[n] s) using well known techniques from calculating multi-nominal distributions. TCG: Concluding remarks, , Tsan-sheng Hsu c 36

37 Practical setup Parameters that are usually used. x = 1. For Chinese chess, q is about , p = and 1 p q is Data source: 63,548 games played among masters recorded at This means the first player has a better chance of winning. The mean of X[n], E(X[n]), is 0. The standard deviation of X[n], σ n, is x n 2pq + (2q + 8p)(1 p q) = 1.16n. TCG: Concluding remarks, , Tsan-sheng Hsu c 37

38 Results (1/3) P r( X[n] s) s = 0 s = 1 s = 2 s = 3 s = 4 s = 5 s = 6 n = 10, σ 10 = n = 20, σ 20 = n = 30, σ 30 = n = 40, σ 40 = n = 50, σ 50 = TCG: Concluding remarks, , Tsan-sheng Hsu c 38

39 Results (2/3) P r( X[n] s) s = 7 s = 8 s = 9 s = 10 s = 11 s = 12 s = 13 n = 10, σ 10 = n = 20, σ 20 = n = 30, σ 30 = n = 40, σ 40 = n = 50, σ 50 = TCG: Concluding remarks, , Tsan-sheng Hsu c 39

40 Results (3/3) P r( X[n] s) s = 14 s = 15 s = 16 s = 17 s = 18 s = 19 s = 20 n = 10, σ 10 = n = 20, σ 20 = n = 30, σ 30 = n = 40, σ 40 = n = 50, σ 50 = TCG: Concluding remarks, , Tsan-sheng Hsu c 40

41 Statistical behaviors Hence assume you have two programs that are playing against each other and have obtained a score of s + 1, s > 0, after trying n pairs of games. Assume P r( X[n] s) is say Then this result is meaningful, that is a program is better than the other, because it only happens with a low probability of Assume P r( X[n] s) is say Then this result is not very meaningful, because it happens with a high probability of In general, it is a very rare case, e.g., less than 5% of chance that it will happen, that your score is more than 2σ n. For our setting, if you perform n pairs of games, and your net score is more than n n, then it means something statistically. You can also decide your definition of a rare case. TCG: Concluding remarks, , Tsan-sheng Hsu c 41

42 Concluding remarks Consider your purpose of studying a game: It is good to solve a game completely. You can only solve a game once! It is better to acquire the knowledge about why the game wins, draws or loses. You can learn lots of knowledge. It is even better to discover knowledge in the game and then use it to make the world a better place. Fun! Try to use the techniques learned from this course in other areas! TCG: Concluding remarks, , Tsan-sheng Hsu c 42

43 References and further readings M. Buro. Toward opening book learning. International Computer Game Association (ICGA) Journal, 22(2):98 102, R. M. Hyatt. Using time wisely. International Computer Game Association (ICGA) Journal, pages 4 9, R. Šolak and R. Vučković Time management during a chess game, ICGA Journal, no. 4, vol. 32, pp , T.-s. Hsu and P.-Y. Liu. Verification of endgame databases. International Computer Game Association (ICGA) Journal, 25(3): , P.-s. Wu, P.-Y. Liu, and T.-s Hsu. An external-memory retrograde analysis algorithm. In H. Jaap van den Herik, Y. Björnsson, and N. S. Netanyahu, editors, Lecture Notes in Computer Science 3846: Proceedings of the 4th International Conference on Computers and Games, pages Springer-Verlag, New York, NY, TCG: Concluding remarks, , Tsan-sheng Hsu c 43