Computing optimal strategy for finite two-player games. Simon Taylor

Transcription

1 Simon Taylor Bachelor of Science in Computer Science with Honours The University of Bath April 2009

2 This dissertation may be made available for consultation within the University Library and may be photocopied or lent to other libraries for the purposes of consultation. Signed: I

3 Computing optimal strategy for finite two-player games Submitted by: Simon Taylor COPYRIGHT Attention is drawn to the fact that copyright of this dissertation rests with its author. The Intellectual Property Rights of the products produced as part of the project belong to the University of Bath (see This copy of the dissertation has been supplied on condition that anyone who consults it is understood to recognise that its copyright rests with its author and that no quotation from the dissertation and no information derived from it may be published without the prior written consent of the author. Declaration This dissertation is submitted to the University of Bath in accordance with the requirements of the degree of Batchelor of Science in the Department of Computer Science. No portion of the work in this dissertation has been submitted in support of an application for any other degree or qualification of this or any other university or institution of learning. Except where specifically acknowledged, it is the work of the author. Signed: II

4 Abstract Game theory is the study of strategic situations in which one or more players make decisions to grant them success whilst also depending on the decision of others in the game. Game theory is used in many areas of research. Very commonly used in Economics it is also used in other areas such as Biology and Political science. It is also popular in Computer science with its use in artificial intelligence. Although game theory had been used since the 18th century is wasn't until 1928 when John von Neumann published a series of papers that it became a unique field of research. The aim of this project is to create a system that solves two-player strategic games. III

5 Contents CONTENTS...I LIST OF FIGURES... II LIST OF TABLES...III ACKNOWLEDGEMENTS...IV INTRODUCTION... 1 LITERATURE SURVEY PRISONERS DILEMMA NASH EQUILIBRIUM MIXED STRATEGIES COMPUTING MIXED STRATEGIES Computing mixed strategy for 2x2 games GAME VALUE DOMINATING STRATEGIES OTHER METHODS SIMPLEX METHOD OTHER IMPLEMENTATIONS REQUIREMENTS FUNCTIONAL REQUIREMENTS NON-FUNCTIONAL REQUIREMENTS CONCLUSIONS REFERENCES BIBLIOGRAPHY i

6 List of Figures FIGURE 1.1 NORMAL AND EXTENSIVE FORM OF A STRATEGIC GAME... 2 FIGURE 2.1 PRISONERS DILEMMA IN NORMAL FORM... 4 FIGURE 2.2 CAMPERS... 5 FIGURE 2.3 STRATEGIC GAME WITH NO NASH EQUILIBRIUM... 6 FIGURE 2.4 MATCHING PENNIES... 7 FIGURE 2.5 A 2X2 GAME... 8 FIGURE 2.6 SELECTIONS FOR EACH PLAYER... 8 FIGURE 2.7 GAME WITH DOMINATING STRATEGIES FIGURE 2.8 EXAMPLE GAME FIGURE 2.9 GAME AFTER ADDITION OF CONSTANT FIGURE 2.10 FIRST SCHEMA FIGURE 2.11 PIVOT CRITERION FIGURE 2.12 SECOND SCHEMA FIGURE 2.13 FINAL SCHEMA ii

7 List of Tables TABLE 1 AN EXAMPLE TABLE... ERROR! BOOKMARK NOT DEFINED. iii

8 Acknowledgements Add any acknowledgements here. iv

9 Chapter 1 Introduction Games theory is the study of strategic situations in which one or more players make decisions to grant them success whilst also depending on the decisions of the other players. It is widely used in Social sciences, notably in Economics but it is also used in areas such as Biology, Political science and Computer science. Players in a game can represent a variety of different things including people, animals, plants or governments. A strategic game consists of players and for each player a set of actions and a preference relation over this set of actions. The number of players in a game affects how difficult is can be to calculate optimal strategies for each player. In one-player games it is just a case of selecting the action that will give greatest reward. For games with two players finding the optimal strategy for each player means looking at the highest pay off that can be achieved given the other players strategy. With games involving more than two players it becomes more difficult to find optimal strategies as there is the possibility of cooperation between players among other things. Games can be represented in one of two forms, these being extensive and normal form. In extensive form the game is shown as a tree with nodes giving the state of the game and each branch giving actions available from the current state. Playoffs are given at the bottom of the tree. This representation is easier to understand and view when learning game theory, however as games get more complex with more players and actions the tree grows to sizes that can be difficult to draw. In normal form the game is represented as a matrix containing each player s payoffs. One player s actions and payoffs are given as the rows in the matrix and the other player as the columns. Figure 1.1 shows a strategic game in both extensive and normal form. 1

10 Figure 1.1 normal and extensive form of a strategic game As is show, although the extensive form can be read more easily even with this small game it is large and looks a little cluttered, where as the normal form game is compact and as games become more complex with additional actions this will become a lot easier to use than extensive form. The payoffs given in the game relate to the type of game being player. In zero-sum games the payoffs for each player sum to zero, so one player gains what the other losses and so the values sum to zero. Games where this is not the case is called non zero-sum games. Non zero-sum games can be transformed into zero-sum games but adding an additional dummy player. The losses of this player compensate for the total gains of the other players. Some of the additional factors of non zero-sum make it more difficult to work with and so will be worked with only is success is achieved with zero-sum games first. One of the most famous strategic games is the Prisoner s dilemma. For most people being introduced to game theory this will be one of the first examples they may come across. It will be used as an example to cover the basics of game theory 2

11 Chapter 2 Literature Survey 3.11 Prisoners dilemma Two suspects are arrested by the police. There is insufficient evidence for a conviction of either suspect. Having separated the prisoners, they are each visited and offered the same deal. If one testifies (betray) against the other and the other stays silent then the betrayer will be set free and the silent suspect will receive a 10-year sentence. If both remain silent, both will be sentenced but will only receive six months for a minor charge. If both testify against each other, they each receive a five-year sentence. Each prisoner must chose to testify or stay silent. Neither prisoner will know what the other chose. With this information we can construct a strategic game for the situation. One assumption made during this is that both prisoners want to receive the shortest sentence possible. In this game the players are each of the suspects. In any game all players have a set of actions available to them. In this game both players have the set of actions {silent, betray}. An action profile is created when the players act, an example of this would be where both players stay silent, (silent, silent), where the active players choice is first and the other players second. A preference relation is a ranking of all possible action sets available to the player in order of their highest payoff first and lowest payoff last. This however does not show how much each action set is favoured over another. In the case of the prisoners dilemma each player will have the preference ordering: (betray, silent) > (silent, silent) > (betray, betray) > (silent, betray) It is normally more convenient to represent a preference relation wait a payoff function. This is a function with values that correspond to the rankings in the preference relation. For this example we can take the values of jail time as the payoff for each players action, however as the preference relation is ordinal we could simple assign values to each action profile in order. 3

12 This can be show in normal form in figure 2.1 Suspect 2 Silent Betray Suspect 1 Silent 2, 2 0, 3 Betray 3, 0 1, 1 Figure 2.1 Prisoners Dilemma in normal form The rows give the actions of the first suspect and the columns for the second suspect. The numbers in this matrix are each of the player s payoffs, with the first players being the first value. A zero-sum game can be represented differently due to the link between the playoffs. For a zero-sum game it is only necessary to show the first player s payoffs as the second player s payoffs are found through negation of the first. By looking at the payoff matrix it would appear that the obvious choice for both players would be to stay silent as they would each receive the shortest sentence. However this turns out not to be the case. Both players have a greater incentive to betray the other because if the other remains silent they will receive no sentence. This leads to the decision that if one player betrays then to receive the highest payoff then the second player will also have to betray. From this it shows that although there is a high payoff for both players if they remain silent, the incentive to betray in the hopes that the other stays silent out weight the cooperation and so (betray, betray) becomes the action set chosen. This outcome is known as Nash equilibrium. 4

13 3.12 Nash equilibrium Named after John Nash who proposed it, is a solution concept of a game involving two or more players. This involves the assumptions that each player knows the equilibrium strategies of the other players and no one player has anything to gain by changing their strategy. If each player has chosen a strategy and no player can benefit by changing their strategy while the other keep theirs unchanged, then this current set of strategy choices and the corresponding payoffs constitute a Nash equilibrium (Wikipedia, Nash equilibrium). For the example of the Prisoners dilemma the action profile (betray, betray) is a Nash equilibrium. This is because there is no action that either player could change to, to give a greater payoff while the other remained the same. However not every strategic game has a Nash equilibrium and this will be discussed later on with mixed strategy. This idea of an equilibrium point was not new when Nash proposed it but due to his work in the field it has become recognized as this. Before this time it was known as a saddle point. The method used for finding Nash Equilibrium is known as the minimax method. The idea of the minimax method is that player 1 will choose the action that s minimum payoff, given all of player 2 s possible actions, is highest. Player 2 will act in the same way. Player Player Figure 2.2 Campers As is shown in figure 2.2 (J.D.Williams, 1966), player 1 will select action 3 as the minimum payoff is 3 which is better than the minimum payoff for any other action. Player 2 will select action 2 as this will give them the lowest maximum loss for any action of -3. This gives the action profile (3, 2) as this games Nash equilibrium. It should be noted that in this game player 2 can never win unless there are some additional side rules added that cannot be represented in the table. In this project side rules will not be implemented. It isn t uncommon to find games that have no Nash equilibrium, figure

14 Player2 1 2 Player Figure 2.3 Strategic game with no Nash equilibrium In this game using the minimax method will return 3 and 4 for each player respectively. Therefore no Nash equilibrium exists. This leads us on to mixed strategies. 6

15 3.13 Mixed strategies So far all games discussed have seen each player picking the same action, the Nash equilibrium. This however as just seen doesn t always happen and this leads to a mixed strategy equilibrium where each player doesn t know what action the other will choose but instead the probability with which they will choose certain actions. This can be shown with the matching pennies game in figure 2.4. Player Player Figure 2.4 Matching pennies In this game, player 1 must match the face of the pennies while player 2 must make them different. The winner receives an extra coin. Using the minimax method it can be seen that there is no Nash equilibrium. However there are mixed strategies for the game. It needs to be assumed that both players would prefer to win more money than lose out. If player 1 was to choose heads with a probability greater than a half then player 2 will choose tails with a probability 1, this would cause player 1 to switch his strategy from heads to tails. This would continue until both players realize that they can do no better than choosing an action with probability greater than a half, this leads to the mixed strategy equilibrium for the game. When the game matrix is mirrored and negated through the diagonal from the first to last element, like in figure 2.4 both players will have the same strategy. This doesn t help with calculating the strategy but if the value in the columns and rows sum to zero then the value of the game is zero and the optimal strategies involve all actions equally. 7

16 3.14 Computing mixed strategies One method for computing mixed strategies uses the players expected utility to find an optimal strategy. The idea is to select a strategy that will win no matter what the other player chooses Computing mixed strategy for 2x2 games One of the best strategies for winning with mixed strategies is to make the other player indifferent as to which action you choose. For 2x2 zero-sum games there is an equation that can be derived from this, that can be used to solve the mixed strategies without any need to use simultaneous equations. The expected utility for player 1 for action 1 will be ap + d(1-p) and for action 2 will be bp + c(1-p). Making these to equations equal and then rearranging them will give us, p = c d / a b + c d, where p is the probability of choosing the first action and 1 p is the probability of the second action. Putting in the values for the game in figure 2.5 will give us p = 3 4 / = ¼. Player Player Figure 2.5 a 2x2 game This is the probability in which player one should choose action 1. leaving us with the probability of ¾ for action 2. The matrix is read differently for player 2 as is shown in figure 2.6. Player Player 1 1 A B 1 A D This will give p = 3 5 / = ½ 2 D C 2 B C Figure 2.6 selections for each player There are also other methods for calculating the mixed strategies of a 2x2 game. One of them is done by finding the odds of the game. Firstly Nash equilibrium need to be looked 8

17 for, if one exists then the odds will not need to be looked of as they will be 1 for the row and column of the equilibrium point. One problem with this method is that it will normally give incorrect strategies if there is a Nash equilibrium. If no Nash equilibrium exists then the mixed strategy will be found. The method for finding the odds for a player is to subtract the 2 nd value for each action from the 1 st. The signs of these odds are not important. The odds of each action are the result of the subtraction from the opposite action. In the example from figure 2.5 the odds for player 1 s first action will be 4 3 = 1 and for the second action 2 5 = -3. This gives the odds for this game for player 1 as 1:3, which is equivalent to action 1 being a probability of ¼. The same can be done for player 2 where action 1 will be 5 3 = 2 and for action 2 will be 2 4 = -2. This giving the odds or 2:2 which can be reduced to 1:1, which is the equivalent of action 1 being probability ½. So for this game the mixed strategies would be for player 1: ¼ A1 + ¾ A2 And for player 2: ½ A1 + ½ A2 Working the values out will give player 1 s average payoff as 4.5 and player 2 s average payoff as This shows that the strategies are optimal as both values are equivalent with signs being ignored; this gives the value of the game. 9

18 3.15 Game value The value of any game is the payoff that can be won when both players are playing their optimal strategies. If this value is positive then on average player 1 will win, if negative player 2 will win and if it is zero then the game is considered fair and both players have the same chances of winning. If a game has a Nash equilibrium then this value will be the games value, however not all games have this and so the mixed strategies will give average payoffs for the players and this will be the value. Sometimes a game can be changed by either adding a constant to the payoffs or by multiplying them. This will not affect game play but will affect game value. The change in game value will be the change made to the payoffs. For example if 3 is added then the new game value would be 3 higher than its original value. Adding constants can change the fairness of the game while multiplication can represent something like a change in currency. 10

19 3.16 Dominating strategies In 2xm games where m is larger than 2 finding dominating strategies is important. If no Nash equilibrium exists then reducing the size of the game makes calculating its mixed strategies easier. A strategy can be dominated by another if, for every action chosen by the other player the payoff is greater in the other. Player Player Figure 2.7 Game with dominating strategies In this game there exist a number of dominant strategies for player 1. The 1 st and 3 rd action being compared shows that action 3 dominates action 1 for both player 2 s actions. Also action 4 dominates action 2 and 5. This allows for the game to be reduced to a 2x2 game. This can now be solved as a 2x2 game with the probability of the removed actions being 0. This however is not the case for all 2xm games, some will not reduce to 2x2 and others will not reduce at all. 11

20 3.17 Other Methods There are other methods for solving or helping to solve matrix games. Some of these are Graphical which can help reduce the game. Trial and error, which again is useful in reducing a game. Gaussian elimination is a more complex method for solving games and makes use of 3 matrices. However for the purpose of this project the Simplex method will be used 3.18 Simplex method The following algorithm for solving finite games is called the simplex method and is used for solving zero-sum games, mxn games (J.D.Williams, 1966) 1. Add a constant to all elements of the game matrix, if necessary, to insure that the value is positive. After computation this constant must be subtracted from the value of the new matrix game to get the value of the original matrix game. The strategies are not affected. 2. Create the first schema by augmenting the game matrix with a new column of -1s on the lower edge, a new row containing 1s on the right edge and zero in the lower right corner. Label Player 1 s strategies on the left from x1 to xm and Player 2 s strategies on the top from y1 to yn. 3. Select any entry from the interior of the schema to be the pivot, say row p column q, subject to the following properties: 3.1. The number in the final row of the potential pivot column, r, must be negative The potential pivot, p, must be positive Compute for each potential pivot the quantity r x c / p, where c is the final value in the column containing the potential pivot. This value is known as the pivot criterion Find the smallest criterion in each valid column (Those containing a negative value in the final row) Find the largest criterion from the set of smallest column criterion. The element corresponding to this criterion is the pivot. 4. The numbers for the next schema are found as follows: 4.1. Number corresponding to the pivot is the value, D, of the preceding schema Numbers that correspond to those of the preceding pivot row stay the same The numbers that correspond to those of the preceding pivot column are the same value negated, except the pivot All other values, except the pivot, are computed from N x p pr x pc / D, where N is the original number and pr and pc are the numbers that have rows and columns in common with p and N. 12

21 4.5. The next value of D is the pivot value, p, of the preceding schema. N.B. D is 1 for the first schema. 5. Player 1's strategies are represented by the rows from the original game matrix and Player 2's strategies are, similarly, represented by the columns from the original game matrix. The two strategies represented by the row and column containing the pivot are part of the optimal strategy for each player respectively. However, when pivoting the strategies swap the action they represent. Example, p = a[1][4], This means that Player 1's action, represented by row 1 is part of his optimal strategy but is now represented by the value in the final element of column 4. Similarly Player 2's action that is represented by column 4 is now represented by the value in the final element of row Check the final row of the new schemata for negative numbers If one exists, return to step If not, the computation is complete and the strategies can be found The numbers in the final column that are part of the optimal strategy for Player 1 are the odds for the action they represent. The numbers in the final row that are part of the optimal strategy for Player 2 are the odds for the action they represent. The value of the probabilities is easily calculated by dividing each individual odd by the sum of the odds. Example, if Player 1's optimal strategy is represented by 0:1:2, the probability for action 1 is 0, for action 2 is 1/3 and for action 3 is 2/3. The game value calculated from D divided by the final element in the matrix. The game in figure 2.8 (J.D.Williams, 1966) will be used as a worked example of the simplex method Player Player Figure 2.8 Example game To begin with the matrix must be cleared of any negative numbers by adding a constant equal to the lowest negative value. This must be added to every element in the matrix. In this example the lowest value is -2 and so 2 will be added to every element. This gives: 13

22 Player Player Figure 2.9 Game after addition of constant This matrix is now augmented with a column of 1s, a row of 1s and a 0 where the two meet. Player Player Figure 2.10 First schema The next step is to find the pivot point. This involves finding the largest of the smallest column pivot criterion. As is shown in figure 2.11 the largest pivot criterion from the set of smallest pivot criterion is 1/7 in the final row and column, this will be the initial pivot point. 1/8 1/2 1/5 1/10 0 1/5 1/6 1/8 1/7 Figure 2.11 Pivot criterion The pivot column values are then negated and the pivot row is left as they are. The pivot point itself takes a new value D, which for the first pivot is 1. After this D takes the value of the previous pivot. 14

23 Player Player Figure 2.12 Second schema Figure 2.12 shows the completed second schema. All values that are not part of the pivots row or column are found using the formula, N x P R x C / D, where N is the value of the pervious schema, p is the pivot value and C and D are the values that share the same row and column as N and P. For the first element this would be 8 x 7 6 x 5 / 1. This method is repeated until there are no negative numbers left in the final row. As there is still a negative in the final row in the example, another pivot point needs to be found from any column that still has a negative in the final row. In this example that is only the first column. Player Player Figure 2.13 Final Schema Once all negatives in the final row have been eliminated the final schema will have been found. Now strategies are ready to be taken from the matrix. Player 1 s mixed strategy is action 2 and action 3 in the ratio 1:5 or 0:1:5. Player 2 s mixed strategy is action1 and action 15

24 3. in the ratio 1:2 or 1:0:2. The game value if found by dividing D, which is 40 by the final element in the matrix, which is 6 giving us 6 2/ Other implementations Software that finds equilibrium is not widely available even though Game theory has its usefulness in many areas of academic research. Some programming languages had a simplex function for them. There is also a set of tools known as Gambit which can be found online. There are a few applications that can be placed on certain calculators. 16

25 Chapter 3 Requirements In this section I will outline the requirements for the system to be created from. These requirements will define the constraints on the implementation and operation of the system. It will be important to keep an up to date set of requirements to make sure that the system has a structure to follow. In small projects it can be easy to create a set of requirements and then not make use of it. This can allow for alterations to be made to the requirements at the end of the project so that it fits the final system, however this is bad practice and will be avoided. This system has one function and that is to compute the optimal strategies for each player in a finite zero-sum game given a set of payoffs from the user. This systems main use is going to be as an aid in the teaching of game theory and as research as it will be too simple to be of use in industry. After research it was decided that the simplex method was to be implemented in this system and so the requirements incorporate that. 3.1 Functional requirements 1. User input 1.1. Game data is separated by whitespace 1.2. Any nxm matrix can be input 1.3. Payoff values can be signed 1.4. Payoffs can be fractions 1.5. Selected output file 2. System operation 17

26 2.1. Look for Nash equilibrium If Nash equilibrium exists then return and end program 2.2. Get size of the game matrix If size is 2x2 then use method for computing 2x2 games 2.3. If any other size use simplex method Make matrix elements positive if needed Create first schema Find pivot and perform calculations search for negatives in final row repeat from if a negative is found otherwise return solutions 3. System output 3.1. results will be written to a user specified output file and to the command line 3.2 Non-functional requirements 1. The system will be written in java 2. The system should run efficiently 3. The interface should be easy to learn and use 4. The system must be reliable 4.1. If an error occurs the system should exit and an error message displayed 4.2. Error messages must be understandable 18

27 Chapter 4 Conclusions During this project I did not manage my time properly and as such I did not give enough time to complete the code. I am not a very competent coder and so when problems arose I had difficulty finding and fixing the error. This was not helped by not going to get help with some of it and leaving it so late that it couldn t be finished. Had I been able to complete the code then I may have been able to complete this document to a much better level. Although I have increased my capabilities as a coder it is still not sufficient at the moment to be of use to myself and so I will need to work harder to achieve this. Had I managed my time properly and began the project when I should have, I could have finished the code and been able to create a high standard project. Working on this project has improved my knowledge on game theory and helped with my research abilities. I am interested in the field of game theory and with the outcome of this project I am disappointed with myself for not working my hardest to get a working system and to have it work successfully. In the future as well as making the system function fully and work, there would be some other functionality that would have been nice to add. A graphical interface would allow for easier understanding of the system and could allow for the games to be represented in both normal and extensive form. Allowing for more than 2 players. There are examples in game theory where a game may include more than just 2 players. Ability to solve non zero-sum games. 19

28 References WIKIPEDIA, The Free Online Encyclopaedia, GARG, R Online Game Theory Text, available at OSBORNE, MARTIN J (1997), An Introduction to Game Theory WILLIAMS, J.D (1966), McGraw Hill Book Company The Compleat Strategyst Bibliography MCKELVEY, RICHARD, MCLENNAN, ANDREW, TUROCY, THEODORE Gambit: Software tools for Game Theory, available at PORTER, NUDELMAN, SHOHAM (2004) Simple Search Methods for Finding a Nash Equilibrium EPPERSON, JAMES (2002) An Introduction to Numerical Methods and Analysis 20