3 Game Theory II: Sequential-Move and Repeated Games

Transcription

1 3 Game Theory II: Sequential-Move and Repeated Games Recognizing that the contributions you make to a shared computer cluster today will be known to other participants tomorrow, you wonder how that affects how many resources you should contribute. What if other participants condition their future contributions on your current contributions? In this chapter we consider game theoretic settings in which participants act in sequence. The main new consideration is the ability of one participant to condition his action on the earlier actions of others. Knowing that this conditioning is possible can change the set of equilibria, with the possibility of responses in the future changing incentives in the present. We start with a simple example of a two player bargaining game (Section 3.1) and introduce the extensive-form representation of a sequential-move game (Section 3.2). In Section 3.3, we define a subgame-perfect equilibrium, which modifies a Nash equilibrium to insist that players model each other as behaving rationally after every possible sequence of play. This is important in ruling out non-credible beliefs about how the other player will act. A special case of a game with sequential moves is a repeated game, where the same strategic situation takes place over and over again (Section 3.4). Repeated games model many practical settings, such as reputation systems and access to shared resources. In Section 3.6, we discuss folk theorem results in repeated games. These highlight the very different outcomes that can occur when a game is played over and again. One challenge with sequential-move games is that the size of an explicit representation of a strategy grows exponentially in the number of rounds of interaction. Recognizing this, we adopt finite-state automata to provide succinct representations of strategies in repeated games (Section 3.5). Automaton strategies also allow for a simple model of bounded rationality, in which a preference is given to strategies that can be represented with a small number of states (Section 3.7). 3.1 Introduction In a game with sequential moves, the strategy of a player needs to describe the action this player will take after every possible sequence of observations of the actions of other players. Let s first study the normal-form representation of a sequential-move game. Rather than enumerate payoffs for combinations of actions, as in the simultaneous-move games of Chapter 2, the normal form must now enumerate payoffs for all combinations of strategies; i.e., all 43

2 3 Game Theory II: Sequential-Move and Repeated Games Figure 3.1: The bargaining game: Player 1 determines a split of $4 and player 2 either accepts or declines. Edges are annotated with actions (y for accept and n for decline), decision nodes with the player who has the turn, and leaves with payoffs. combinations of complete descriptions of play. bargaining game. Let s consider the following example of a Example 3.1 (Bargaining Game). Two agents are bargaining over how to divide $4. If they can t agree on a division neither player will receive any payment. Player 1 goes first and can propose one of three possible splits, (x 1,x 2 ) 2{(3, 1), (2, 2), (1, 3)}, which we refer to as {me, even, you}, where x 1 is the amount he should receive and x 2 is the amount player 2 should receive. Player 2 then accepts (y) or declines (n), with zero payoff to both players if she declines. Player 1 s strategy determines the split. Player 2 s strategy determines whether the split is accepted or whether both players receive nothing. See Figure 3.1, which illustrates the game as a tree, with the play of player 1 at the root and the play of player 2 at the intermediate nodes. Player 1 only makes one decision, and his only possible strategies are {me, even, you}. Player 2 s strategy must define an action after every possible move of player 1. We denote player 2 s possible strategies as {nnn, nny, nyn,..., yyy}. For example, strategy yny corresponds to accepting the (3,1) and (1,3) splits, but declining the (2,2) split. The normal-form representation of the game is provided in Figure 3.2. Inspection of the payoff matrix reveals many pure-strategy Nash equilibria; for example, (you,nny), (even,nyy) and (me,yyy). But some of these equilibria do not seem very sensible. In (you,nny), player 2 s strategy nny corresponds to declining a split of (2, 2) and (3, 1), which is not utility-maximizing in these situations. Similarly, in equilibrium (even,nyy), player 2 declines a split of (3, 1). In this sense, the strategies nny and nyy include non-credible threats. The Nash equilibrium (me,yyy) does not suffer from these difficulties. Informally, a strategy contains a non-credible threat when the action is not rational because it would not be utility-maximizing for the player if chosen. Because the action is not rational, 44 Copyright 2014 Parkes & Seuken. Draft: Do not distribute without permission.

3 3.2 The Extensive-Form Representation Player 1 Player 2 nnn nny nyn nyy ynn yny yyn yyy me 0, 0 0, 0 0, 0 0, 0 3, 1 3, 1 3, 1 3, 1 even 0, 0 0, 0 2, 2 2, 2 0, 0 0, 0 2, 2 2, 2 you 0, 0 1, 3 0, 0 1, 3 0, 0 1, 3 0, 0 1, 3 Figure 3.2: The normal-form representation of the bargaining game. it should not be considered credible by the other player. Such an action can exist in a Nash equilibrium when it occurs off the equilibrium path. The normal form does not retain information about the sequential structure of the game. Nothing special is implied by writing nyy against a column in the normal form. The represented game would be identical if the columns were simply labeled by something arbitrary, such as W 1, W 2, and so on. In particular, given that player 1 plays you, there is no way to decide whether player 2 will respond with yyy or nny---both have the same payoff entries in the normal form. It is incorrect to say that (even,y) is a Nash equilibrium of the bargaining game. By only describing the action y that player 2 takes when player 1 plays even, we leave the behavior of player 2 partially specified. In order to check that even is a best-response of player 1, we need to specify how player 2 would act after every possible action of player 1. We can say that (even,y) are actions on the equilibrium path, and the split of payoffs (2, 2) occurs in an equilibrium outcome. 3.2 The Extensive-Form Representation The extensive-form representation of a sequential-move game retains information about the sequential structure of the game. In doing this, the extensive form has to specify: the players, who moves when (the order of moves), what a player can do when he can move, what a player knows when he moves, and the payoffs. For this, the extensive form associates a history with each node in the tree of possible sequences of moves (see Figure 3.1). The history of moves is the sequence of actions that lead to the node. Some histories are terminal histories, and correspond to leaves of the tree, where no players have further moves. Each history is associated with a label of a player whose turn it is to move, and a set of actions available to the player. For the most part, we make the simple assumption that a player knows the full history. Definition 3.1 (Extensive-Form Representation). The extensive-form representation of a sequential-move game has Copyright 2014 Parkes & Seuken. Draft: Do not distribute without permission. 45

4 3 Game Theory II: Sequential-Move and Repeated Games N = {1,...,n} agents, indexed by i. A set H of sequences (finite or infinite) of actions that satisfy the following 3 properties: The empty sequence " is a member of H If sequence h 2 H then any sequence h 0 formed by an initial subsequence of h is also a member of H. (Each member of H is a history, each component of a history is an action taken by an agent). Ahistory(a k ) k=0,...,t 1 2 H is terminal if it is infinite, or there is no a T such that (a k ) k=0,...,t 2 H. The set of terminal histories is denoted Z. For each agent i, theutilityu i (h) 2 R for each terminal history h 2 Z. For every non-terminal history h 2 H \ Z: the agent i = P (h) 2 N whose turn it is to move, and the set of actions A i (h) Ãi available to the agent, where Ãi includes the set of all actions available to the agent in some history. We adopt shorthand extensive-form game to refer to the extensive-form representation of a sequential-move game. Example 3.1 (continued). In the bargaining game in Figure 3.1, the set of histories is H = {", (me), (even), (you), (me,y),(me, n), (even, y), (even, n), (you, y), (you, n)}. Of these, Z = {(me, y), (me, n), (even, y), (even, n), (you, y), (you, n)} are the terminal histories, and associated with a utility for each player. For example, u 1 ((me, y)) = 3 and u 2 ((me, y)) = 1. The history " is the empty history and represents the first move in the game. We have P (") =1and P (me) =P (even) =P (you) =2. Each non-terminal history is associated with the set of actions available to the player whose turn it is to move: A 1 (") ={me, even, you} and A 2 ((me)) = A 2 ((even)) = A 2 ((you)) = {y, n}. The representation is more succinct than the normal-form representation. The extensive form records the utility once for all strategy profiles with strategy me for player 1 and strategies {nnn, nny, nyn, nyy} for player 2. Player 2 declines the offer for all these strategies. The extensive form can be extended to allow for simultaneous moves at every step, and this is necessary to model repeated games (see Example 3.2 and Section 3.4). A strategy for player i is denoted s i 2 S i,wheres i is the set of all possible strategies. Although this is the same notation as we adopted for mixed strategies in simultaneous-move games in Chapter 2, it will be sufficient for our purposes to only consider non randomized strategies, s i, for sequential-move games. A strategy s i defines an action s i (h) 2 A i (h) for all non-terminal histories h for which i = P (h). In this way, a strategy defines an action at every node where it is a player s turn, and is a complete description of play. This complete description is important for performing a strategic analysis of sequential move games. Let u i (s) 2 R denote the utility to agent i given a strategy profile s =(s 1,...,s n ). This is the utility to agent i at the terminal history h 2 Z reached under strategy profile s. Definition 3.2 (Nash equilibrium, extensive-form game). A strategy profile s is a Nash equilibrium 46 Copyright 2014 Parkes & Seuken. Draft: Do not distribute without permission.

5 3.3 Subgame-perfect Equilibrium Figure 3.3: Backward induction to find the subgame-perfect equilibrium in the bargaining game. in an extensive-form game, if for all agents i, u i (s i,s i) u i (s i,s i), for all strategies s i 2 S i (3.1) Example 3.1 (continued). The Nash equilibrium (you,nny) in the bargaining game is represented as strategy profile: s 1 (") =you 8 < n s 2 (h) = n : y,ifh =(me),ifh =(even),ifh =(you) If all histories are finite, then a game is a finite extensive-form game. The extensive form also allows for infinite histories, which are infinite sequences of actions. In this way, the extensive form can model games that continue forever, such as the infinitely repeated games we ll see later in the chapter, and associate a utility profile with every infinite history. 3.3 Subgame-perfect Equilibrium Non-credible threats in a strategy are actions that would not be rational for a player if the play reached the history associated with the proposed action. The following example illustrates the use of backward induction as a way to select for a Nash equilibrium that does not include such threats. Backward induction is a natural approach to identifying an equilibrium in a finite extensive-form game. Working from the bottom of the tree to the root, it assigns an optimal action to the player whose turn it is at a node, given the decisions assigned to nodes below this node. Copyright 2014 Parkes & Seuken. Draft: Do not distribute without permission. 47

6 3 Game Theory II: Sequential-Move and Repeated Games Figure 3.4: Backward induction to find the subgame-perfect equilibrium in a two-round repeated Prisoner s Dilemma. Example 3.1 (continued). Consider Figure 3.3. Backward induction assigns move y to each of the nodes associated with histories (me), (even) and (you), respectively. Given this, backward induction assigns move me to the decision of player 1 at the root of the tree. The equilibrium obtained by backward induction is (me,yyy). For another example, consider a simple two round, repeated Prisoner s Dilemma. Example 3.2. The Prisoner s Dilemma is played twice by the same two players (see Figure 3.4). For convenience, we modify the payoffs from the game (Example 2.1) by adding 5 everywhere, in order to make them non-negative. This is a sequential-move game where each player chooses a move simultaneously in each round. Backward induction first solves for a Nash equilibrium in the second round, for each of the possible play in the first round, and assigns the moves (D, D) to each history. Given this, backward induction then assigns moves (D, D) as the play in the first round. In each of these examples, backward induction identifies a Nash equilibrium of the sequentialmove game that does not include non-credible threats. To formalize the particular kind of equilibrium that we are interested in we first define the subgame of an extensive-form game: Definition 3.3 (Subgame). The subgame at history h of an extensive-form game is the extensive-form game forward from the node that corresponds to history h. 48 Copyright 2014 Parkes & Seuken. Draft: Do not distribute without permission.

7 3.3 Subgame-perfect Equilibrium For example, a subgame of the bargaining game is the game rooted at the node defined at history (even). Only player 2 has a move remaining in this subgame. Definition 3.4 (Subgame-perfect equilibrium). A strategy profile s = (s 1,...,s n) is a subgame- perfect equilibrium of an extensive-form game if the strategy profile is a Nash equilibrium in the subgame at every non-terminal history h 2 H \ Z. A subgame-perfect equilibrium is a refinement of a Nash equilibrium, in the sense that every subgame-perfect equilibrium is also a Nash equilibrium but not every Nash equilibrium is a subgame-perfect equilibrium. The first direction follows immediately from the definition of a subgame-perfect equilibrium, since the complete game is one of the subgames. For an example of the other direction, we can again consider the bargaining game. Example 3.1 (continued). The Nash equilibrium (you,nny) of the bargaining game is not a subgame-perfect equilibrium. The action n is not a best-response for player 2 in the subgame that follows history (even), and nor is action n a best-response in the subgame that follows history (me). It is easy to verify that the Nash equilibrium (me,yyy) is subgame perfect. In fact, strategy profile (me,yyy) is the only subgame-perfect equilibrium in the bargaining game. The intuition for this comes from recognizing that there was a strictly preferred action at every choice made when applying backward induction. We provide a formal argument in regard to uniqueness in the next section The Single-deviation Principle To verify that a strategy profile is a subgame-perfect equilibrium we must verify that the strategy profile forms a Nash equilibrium in every subgame. This task is considerably simplified through the single-deviation principle. The principle is stated here for finite extensive-form games in which one player moves at each step, but also holds in more general settings (see Section 3.4.1). Theorem 3.1 (Single-deviation Principle). A strategy profile is a subgame-perfect equilibrium in a finite extensive-form game if, and only if, there is no subgame at non-terminal history h where the agent whose turn it is to move has a profitable deviation by changing its action at that history h only. Proof. The only if follows directly from the requirement of a subgame-perfect equilibrium. For the if, suppose for contradiction that there is no useful single deviation from strategy profile s, for any agent in any subgame, but s is not a subgame-perfect equilibrium. This means that for some agent, say agent i, there is a subgame from which i has a useful sequence of deviations. Consider a minimal such sequence, such that every deviation is required for the sequence of deviations to be useful. Let h 0 and h 00 denote the histories corresponding to the last two such useful deviations on such a minimal sequence. See Figure 3.5. Player i can improve its payoff (from subgame h 0 ) by deviating to action a 0 i 6= s i (h0 ) at h 0, but only if this Copyright 2014 Parkes & Seuken. Draft: Do not distribute without permission. 49

8 3 Game Theory II: Sequential-Move and Repeated Games Figure 3.5: Illustrating the proof of the single-deviation principle. is followed by an additional deviation at the subgame associated with history h 00. Let action a 00 i 6= s i (h00 ) denote this deviation. Let i, i 0 and 00 i denote the payoff when following s from h 0, when deviating once to a 0 i, and when deviating again to a00 i at h 00, respectively. We have i 00 > i. But, by assumption we have i = i 0 and 0 i = 00 i, and a contradiction. To verify that a strategy profile s is a subgame-perfect equilibrium it is sufficient to consider every subgame in turn, and check that the agent i with the move at the start of the subgame can do no better by doing something else at the start of the subgame and subsequently following its strategy, s i. The single-deviation principle is very useful. One immediate application establishes the existence of subgame-perfect (and thus Nash) equilibria in extensive-form games. Theorem 3.2. Each finite extensive-form game with a finite number of actions has a subgameperfect equilibrium. Proof. The proof is constructive, by backward induction from the terminal histories. A finite game only has histories of a finite length. Since there is only a finite number of actions, and one player makes a move at each node, there can only be a finite number of leaves (and thus histories). Given this, we can work backwards from the leaves to the root of the tree, assigning to each node an action that is a best response given the actions assigned to subsequent actions (recall we assume extensive-form games in which one player moves at each step). By construction, this strategy profile satisfies the single-deviation principle, and is a subgame-perfect equilibrium by Theorem 3.1. If during backward induction, at every subgame there is an action that is strictly preferred to every other action at this subgame, then the game has a unique subgame-perfect equilibrium. 50 Copyright 2014 Parkes & Seuken. Draft: Do not distribute without permission.

9 3.3 Subgame-perfect Equilibrium This holds in the barganing game, and in a more general sense that we return to in Section 3.4 in the finitely repeated Prisoner s Dilemma. Backward induction provides a simple method for computing a subgame-perfect equilibrium in sequential-move games. The implied algorithm requires a number of steps that is linear in the number of nodes in the tree, and thus linear in the extensive-form representation of the game. Use caution when applying the single-deviation principle: (1) the single-deviation principle only identifies a subgame-perfect equilibrium, and there could be additional Nash equilibria, and (2) it is important that the payoff from a single deviation is considered at the subgame rooted at the history from which the deviation is considered. It is not enough, for example, to check in a two player game that u 1 (s 1,s 2 ) u 1 (s 0 1,s 2) for all possible single deviations from some strategy s 1 to s 0 1. This does not even establish that (s 1,s 2 ) is a Nash equilibrium. Rather, we must look to see whether s 0 1 is better in the subgame at which the single deviation occurs A Critique of Subgame-perfect Equilibria Subgame-perfect equilibria remove non-credible threats. Still, various critiques can be made in regard to the predictions offered about behavior in sequential-move games. One such criticism relates to what to do when actions are observed that should not be played in a subgame-perfect equilibrium, which we illustrate via the centipede game: Example 3.2 (Centipede game). The game in Figure 3.6, called the centipede game, models a setting where two players alternately have the opportunity to take action stop (S) or continue (C). Stop ends the game. The player to act prefers S to C if the other player will immediately choose S. But if the other player will choose C if given the opportunity, then choosing C and then S after C from the other player is better still. By backward induction, the unique subgame-perfect equilibrium is to play S in every step. Not only does this seem unlikely to occur in practice, it is also problematic in the following sense. Suppose player 1 plays C and now player 2 decides how to act. By backward induction, player 2 should choose S, reasoning that player 1 will choose S in the next step if player 2 plays C now. But by the same analysis, player 1 should have played S in the first step. But player 1 did not. The observed action of player 1 contradicts the assumption about how player 1 will act in the future. What should player 2 do? This difficulty, where observed play is inconsistent with the model for how players should be behaving, is not easy to resolve and beyond the scope of our discussion. One thing that can help is to move to a setting where players are uncertain about the payoffs of each other, so that there is some possibility that another player could have payoffs for which observed actions are utility-maximizing. We will study games of incomplete information in Chapter 6, in the context of auction design. Copyright 2014 Parkes & Seuken. Draft: Do not distribute without permission. 51

10 3 Game Theory II: Sequential-Move and Repeated Games Figure 3.6: A six period version of the centipede Game More General Models The extensive-form representation can be generalized in a number of ways, including: (a) allowing for randomization, which can be modeled either through mixed strategies (a player randomizes over pure strategies at the beginning of the game) or behavioral strategies (a player randomizes independently at each history). (b) allowing for imperfect information, where one or more of the earlier actions are unobservable to one or more agent who move in a later step. (c) introducing Nature as a special agent, who takes actions that influence payoffs by determining the subgame in which play continues, using imperfect information about these actions to model agents with incomplete information about the payoffs of other agents. (d) allowing for simultaneous moves within a single turn, by making a history h denote a sequence of tuples of actions, and allowing P (h) to denote a set of agents. The chapter notes provide details on these generalizations of the extensive-form. Simultaneous moves can also be modeled through imperfect information, where players play sequentially but without observing the actions of earlier players. We illustrate these generalizations in the following example: Example 3.3. Let s think about how to model the game of poker. Nature can be used to model shuffling the deck of cards and dealing. Nature takes a sequence of actions at the start of the game, each action representing the deal of cards to each subsequent player; it concludes with an action that represents the order of cards remaining in the deck. By adopting a mixed strategy for Nature, this model captures the randomness inherent in shuffling a deck of cards. The model captures the uncertainty that a player has about the cards of other players by assuming imperfect information, so that each player only observes the action of Nature that define his own cards. The same modeling approach can be adopted for auctions, where a bidder knows her own value but is uncertain about the value of others. 52 Copyright 2014 Parkes & Seuken. Draft: Do not distribute without permission.

11 3.4 Repeated Games 3.4 Repeated Games In a repeated game, a simultaneous-move game is played over and again by the same players. The game that is repeated in each period is called the stage game, and it may be repeated an infinite number of times. Repeated games provide useful models for many practical settings; e.g., firms that compete with each other in setting prices over an extended period of time, politicians voting on bills, or friends deciding each day where to have lunch. In studying repeated games, we re interested in how the ability to adapt or condition future actions on current actions changes the set of possible equilibria. We adopt notation Ãi to denote the set of actions available to player i in each period, and ũ i (a) to denote the payoff to player i in the stage game given action profile a. This keeps the notation for the utility in the stage game distinct from the notation for the utility in the repeated game. We consider both finitely and infinitely repeated games, and use the repeated Prisoner s Dilemma as a running example (see Example 3.2 for a two-period example.) Definition 3.5 (Repeated Game). In a repeated game G T, the same simultaneous-move game G =(N, A, ũ) (the stage game) is played by the same players for T periods, with every agent having perfect information about the history of actions in all previous periods. In an infinitely repeated game G 1 the stage game G is repeated forever. Let k index a period, with k 2{0, 1,...,T 1} in a finitely repeated game. Let a k denote the action profile selected by players in period k. To represent a finitely repeated game in extensive form we allow for simultaneous moves in each period, and associate histories with sequences of action profiles rather than sequences of actions. The utility function on a complete history, h =(a 0,a 1,...,a T 1 ), is defined as the sum of the utility to player i across all T periods: TX 1 u i (h) = ũ i (a k ). (3.2) k=0 If we were to simply sum the payoffs over all periods in an infinitely repeated game the sum can be unbounded. To address this, we model players who discount the future. Let, with 0 < <1, denote the discount factor. Suppose that a player s utility is a, b and c in periods 0, 1 and 2 respectively. The total discounted utility is a + b + 2 c. The larger the discount factor, the more patient an agent becomes. Example 3.4. If the sequence of play in a three-round, repeated Prisoner s Dilemma is (C, C), (C, C) and then (D, C), and =0.9, then player 1 s total discounted utility is 3+ (0.9)3 + (0.9) 2 5=9.75. An extensive-form representation can be obtained by defining the utility of player i for an infinite sequence of play h =(a 0,a 1,...), as 1X u i (h) = kũ i (a k ), (3.3) k=0 Copyright 2014 Parkes & Seuken. Draft: Do not distribute without permission. 53

12 3 Game Theory II: Sequential-Move and Repeated Games discount factor per-period payoff Figure 3.7: The discounted sum payoff for different per-period payoffs and discount factors where history h is an infinite sequence of action profiles. To calculate the total discounted payoff, we use the formula for the geometric series, 1X k=0 k v = v 1, (3.4) which holds for any value v 2 R, and any discount factor 0 < the total payoffs are finite. <1. From this, we see that Example 3.5. Let s consider two different discount factors, 2{0.5, 0.9}, and two different per-period payoffs, either 1 or 4; see Figure 3.7. A patient agent, with = 0.9 and per-period payoff of 1, has a larger discounted utility than an impatient agent with = 0.5 and per-period payoff of 4. Thus, the discount factor can have a large effect on payoffs. The discounting of future payoff can be motivated as follows: There is always some probability 1 that the current period is the last, so that the next period will be reached with probability, the one after that with probability 2,etc. Future payoffs are worth less because they represent a quantity such as money that falls in value each period because of inflation. There is experimental evidence that people behave as if they have a discount factor. For the rest of the book, we assume discounting when considering infinitely repeated games Equilibrium Analysis A strategy in a repeated game defines an action for the player after every possible sequence of observations of play. We will again appeal to a single-deviation anaylsis approach: Theorem 3.3 (Single-deviation Principle for Repeated Games). A strategy profile is a subgameperfect equilibrium in a finitely or infinitely repeated game if and only if, for every subgame, no agent can improve its utility by changing its action in the current period and leaving the rest of its strategy unchanged. For an infinitely repeated game, the single-deviation principle holds when discounting is used in defining agent payoffs (see the chapter notes). An immediate implication of the single-deviation principle is the following observation: 54 Copyright 2014 Parkes & Seuken. Draft: Do not distribute without permission.

13 3.4 Repeated Games Theorem 3.4 (Uniqueness). If the stage game has a unique Nash equilibrium, then the only subgame-perfect equilibrium strategy in a finitely repeated game is to play the stage game Nash equilibrium strategy after each possible history. Proof. The proof is constructive, by backward induction from the final period T 1. The only possible equilibrium in period T 1 is a Nash equilibrium of the stage game, which is unique. The play in period T 2 does not affect the play in period T 1, thus by the single-deviation principle the play must be a Nash equilibrium of the stage game, which is unique. Otherwise, at least one player would have a useful single-action deviation. This argument continues all the way to the first period, and completes the proof. Example 3.2. Since (D, D) is the unique Nash equilibrium in the Prisoner s Dilemma stage game, then by Theorem 3.4, the only subgame-perfect equilibrium strategy in a finitely repeated Prisoner s Dilemma is to play D after each possible history. There are two important limitations of this uniqueness theorem. First, it only applies to finitely repeated games. Second, it does not preclude additional, non subgame-perfect, Nash equilibria. We see examples of a Nash equilibrium and a subgame-perfect equilibrium that support play (C, C) in an infinitely repeated Prisoner s Dilemma in Sections and 3.6 respectively. In applying the single-deviation principle to the analysis of repeated games, it is useful to group histories (or subgames) into sets, such that every history in a set has the same single-deviation analysis. By this, we mean that a single deviation to any other action has exactly the same effect on an agent s utility from every history in such a set. Define two histories h and h 0 to have an equivalent single-deviation analysis with respect to strategy profile s =(s 1,...,s n ), if (a) for a game that is a finitely repeated game, the histories h and h 0 have the same length, and (b) the strategies in s when restricted to subgames h and h 0 (i.e., ignoring any variations leading up to these subgames and just looking forward) are identical. In this subgame equivalence approach to analysis, having grouped histories according to properties (a) and (b) it is sufficient to prove that there is no useful single-deviation for a representative history in each group. We return to this approach in Section Any two histories that have these properties have the same single-deviation analysis. The sequence of play following a deviation in the current period will be identical because of properties (a) and (b). Any differences in past play leading to these two histories are irrelevant because the payoffs going forward are identical, since they depend only on the payoffs in the repeated stage game Open-loop Strategies We first consider the following simple class of non-adaptive strategies: Definition 3.6 (Open-Loop Strategy). An open-loop strategy s i for player i in a repeated game satisfies s i (h) =s i (h 0 ), for all histories h, h 0 that are the same length. Copyright 2014 Parkes & Seuken. Draft: Do not distribute without permission. 55

14 3 Game Theory II: Sequential-Move and Repeated Games In particular, a player s action can depend on the number of periods since the start of the game but not on the actions in previous periods. Example 3.3. Open-loop strategies in the repeated Prisoner s Dilemma include: cooperate in every period alternate between cooperate and efect cycle through cooperate, cooperate, and defect every three periods. An open-loop strategy may randomize over actions, as long as this randomization does not depend on past play. These strategies are all open loop in the sense of control theory: the choice of action in the current period does not depend on feedback from past periods. Open-loop strategies are interesting because of the following theorem. Theorem 3.5. An open-loop strategy profile in which a Nash equilibrium of the stage game is played in each period of a finite or infinitely repeated game is a subgame-perfect equilibrium of the repeated game. Proof. Let s denote an open-loop strategy profile in which a Nash equilibrium of the stage game is played in each period. Consider any history h and any player i. By the single-deviation principle, it is sufficient to establish that there is no useful deviation by player i from profile s in this period. For this, just note that its action best-responds to the actions of the others in the current period (because s plays a Nash equilibrium of the stage game in every period), and that its choice of action does not affect its action or the actions of other players in future periods because s is open-loop. Example 3.4. In the game of Chicken (Example 2.9), a cyclic pattern of play with action profiles (Y,S), (Y,S), (S, Y ), (Y,S), (Y,S), (S, Y ), (Y,S),... is a subgame-perfect equilibrium. The theorem applies to playing both pure-strategy and mixed-strategy Nash equilibria of the stage game as an open-loop strategy. Neither player has a useful deviation to any other strategy, including adaptive strategies that are not open loop. 3.5 Automaton Strategies Open-loop strategies are simple to describe, but preclude the kinds of interesting adaptive behavior that we might expect in repeated games. A general challenge with strategies in extensive-form games is that their representation size grows exponentially in the number of periods. How many unique strategies are there in a three-period, repeated Prisoner s dilemma? There are =2 21, (3.5) representing one choice in period 0, one choice in period 1 for each of the four possible action profiles in period 0, and one choice in period 2 for each of the 4 4 = 16 combinations of action profiles in periods 0 and 1. We can check that if the Prisoner s dilemma were repeated 10 times, that there would be unique strategies! 56 Copyright 2014 Parkes & Seuken. Draft: Do not distribute without permission.

15 3.5 Automaton Strategies Figure 3.8: Four example automaton strategies for the repeated Prisoner s Dilemma, expressed from the perspective of agent 1: (a) tit-for-tat, (b) always-defect, (c) grim trigger and (d) punisher. The arrow coming from the left points to the start state. Each state is annotated with the action taken in that state. Transitions between states depend on the actions selected by players in the current period. The notation (,a 2 ) on an edge indicates this transition occurs whenever agent 2 plays action a 2, and (, ) indicates that this transition always occurs. In this section, we adopt a finite state automaton to provide a succinct representation of a strategy. Such an automaton strategy can be used to represent strategies in both finite and infinitely repeated games. An automaton strategy describes a procedure for determining the action an agent will play after each possible history. An automaton strategy uses an internal state to keep track of the history of play, with state transitions based on observed actions. Associated with each internal state is the action the agent will play when in this state. To keep things simple we only consider automaton strategies without randomization. Any strategy in a finitely repeated game can be represented as an automaton strategy; e.g., a simple represetation uses a separate state for each possible history of play. But the main interest in automaton strategies is when they provide a succinct representation. Example 3.4 (continued). Figure 3.8 shows four examples of automaton strategies for the infinitely repeated Prisoner s Dilemma. The example strategies are (a) tit-for-tat (TfT), (b) always-defect, (c) grim trigger and (d) punisher. All four automaton strategies are illustrated from the perspective of agent 1. The TfT strategy is to initially cooperate, and then play the same action in period t +1 as the other player adopted in period t: If you cooperate, I ll cooperate. But if you defect, then I ll defect. The TfT automaton has two states, each associated with a node in the directed Copyright 2014 Parkes & Seuken. Draft: Do not distribute without permission. 57

16 3 Game Theory II: Sequential-Move and Repeated Games graph. In one state, which is also the start state (indicated graphically by the incoming edge from the left), the agent cooperates, hence the state is annotated with the C action. In the other state the agent defects. Transitions between states, which occur on the basis of observed actions in the current period, complete the description of the strategy. Directed edges represent transitions, and are labeled with an action profile (a 1,a 2 ) to specify that they occur when these actions have been observed. In TfT, when the other player s action is D the automaton changes to the D state. This is indicated by the directed edge between the two states with label (,D), which means that it occurs for any action of player 1 (hence the ) and action D of player 2. When the other player s action is C the automaton remains in state C, indicated by the self-loop with label (,C). The three other automaton strategies in Figure 3.8 behave as follows: Always defect: defect all the time. Grim trigger: start by cooperating but defect forever as soon as the other player has defected. Punisher: start by cooperating, and defect for three periods before returning to cooperate if the other player defects in the same period that punisher cooperates. The automata define behavior for impossible strategy profiles. For example, the TfT automaton specifies a transition after action profile (,C), and thus after action profile (D, C). But this aciton profile cannot occur given the way the strategy is defined because player 1 will only play C in the C state. It is important to define all transitions in order to analyze play off the equilibrium path (following a deviation by one or more players), and complete the analysis required to verify that strategies form a subgame-perfect equilibrium. We return to this in Example 3.4. Formally, we define an automaton strategy as follows: Definition 3.7 (Automaton Strategy). An automaton strategy m i for player i in a repeated game is defined by (Q i,q 0 i, succ i,f i ) where: Q i is the set of machine states q 0 i 2 Q i is the start state succ i : Q i à 7! Q i is the next state mapping, where à = Ã1... Ãn is the set of action profiles and Ãi the action set for player i f i : Q)i 7! Ãi is the action mapping Example 3.4 (continued). The TfT automaton strategy for the repeated Prisoner s Dilemma is defined by, Q 1 = {1, 2}, q 0 1 =1, f 1(1) = C, f 1 (2) = D, andtransitions, 8 1, if a =(C, C) >< 1, if a =(D, C) succ 1 (1,a)= 2, if a =(C, D) >: 2, if a =(D, D) 8 1, if a =(C, C) >< 1, if a =(D, C) succ 1 (2,a)= 2, if a =(C, D) >: 2, if a =(D, D) (3.6) 58 Copyright 2014 Parkes & Seuken. Draft: Do not distribute without permission.

17 3.5 Automaton Strategies An automaton with ` states can only express strategies that distinguish between at most ` kinds of histories. For example, an automaton with two states can store one bit of information about the history. Consider TfT, for example, where one state records that the other player just cooperated and the other state that the other player just defected Equilibrium Analysis Suppose we would like to establish that automaton strategies m =(m 1,...,m n ) form a Nash equilibrium. For this, we must show, for every player i, and fixing the automata (m 1,...,m i 1,m i+1,...,m n ) of other players, that there is no better strategy than the strategy defined by automaton strategy m i. In general this can be a hard task, because it requires reasoning about the effect of different automata on the actions that will be selected by other agents. But we can provide an example of this analysis for a simple automaton strategy such as grim trigger (Figure 3.8 c). Example 3.4 (continued). Let s establish that the grim trigger in Figure 3.8 (c) is a Nash equilibrium of the infinitely repeated Prisoner s Dilemma, for all sufficiently large. Fix automaton m 2 to grim trigger. Consider any strategy s 1 that differs from that defined by automaton m 1.Ifs 1 changes the equilibrium path (and thus player 1 s payoff), there must be some first period k where player 1 plays D rather than C. This triggers player 2 to play D for all future periods whatever player 1 does, and thus after deviating to D in period k, itis best for player 1 to play D in all future periods. This is a complete characterization of the only possible useful deviation by player 1 from m 1. The discounted payoff to player 1 under (m 1,m 2 ) is 3/(1 ) by the formula for the geometric series. Deviating to strategy s 1 yields a payoff of 3 for the first k 1 periods, then 5 in period k, and then 1 for all subsequent periods. The situation facing player 1 is illustrated in Figure 3.9. The first k 1 periods are the same under m 1 and s 1,andsofor(m 1,m 2 ) to be a Nash equilibrium, we need a discount factor <1 such that, apple 3 1, (3.7) for the total discounted payoff forward from period k. By simple algebra, this holds for any 0.5, and grim trigger is a Nash equilibrium for 0.5. Is grim trigger a subgame-perfect equilibrium? To establish that automaton strategies m form a subgame-perfect equilibrium, it is necessary to show that no player i has a useful single deviation from the automaton strategy m i in any subgame. By following the subgame-equivalence approach, we can group subgames for which the strategies under automata m are identical. In particular, because the effect of history on an agent s strategy is completely determined by the machine state corresponding to the history, it is sufficient to verify that no agent has a useful single deviation from any state profile q 2 Q 1... Q n, considering all state profiles q that can be reached through some sequence of actions (and not just those sequences of actions that occur in equilibrium.) Based on this analysis, we establish that grim trigger is not subgame perfect: Copyright 2014 Parkes & Seuken. Draft: Do not distribute without permission. 59

18 3 Game Theory II: Sequential-Move and Repeated Games Figure 3.9: Payoffs to player 1 considering a deviation from grim trigger in some period k in the repeated Prisoner s Dilemma. Example 3.4 (continued). Grim trigger is not a subgame-perfect equilibrium in the infinitely repeated Prisoner s Dilemma, for any discount factor 2 (0, 1). Let (m 1,m 2 ) denote a pair of grim trigger automata. Proceeding by the subgame-equivalence approach to singledeviation analysis, there are four distinct state profiles for the purpose of subgame analysis: (q 1 = C, q 2 = C), (q 1 = D, q 2 = D), (q 1 = D, q 2 = C) and (q 1 = C, q 2 = D), and all can be reached by some sequence of (not necessarily on-equilibrium-path) play. Let s consider state profile (q 1 = C, q 2 = D), and period k of the repeated game. Player 2 is in state D, and so player 1 has defected. Because player 1 is still in state C then player 1 must have just defected in the last period. Following the automaton strategies, the subsequent play from period k is (C, D), (D, D) and then (D, D) in all future periods. But if player 1 deviates and plays D in period k, before following m 1,theplayis(D, D) in period k and all future periods. Note that player 1 adopts D in periods k +1 onwards, because player 2 plays D in period k, andsod is prescribed by m 1. This deviation is useful unless, 1 1 apple 0+. (3.8) 1 1 But there is no discount factor 2 (0, 1) that satisfies this inequality. This shows that player 1 prefers to defect in the immediate next period, having just defected, and establishes that grim trigger is not a subgame-perfect equilibrium. Exercise 3.3 establishes that there is a discount factor for which there is no useful single deviation in any of the other three state profiles, isolating the problem to this particular aspect of the strategy. This analysis highlights the importance of defining transitions in an automaton strategy for all action profiles, even those that will never be played given the design of the automaton. In particular, to verify that a strategy profile forms a subgame-perfect equilibrium, we need to consider single deviations from all histories. For example, the automaton for player 1 must specify how to act in a period after player 1 has deviated first by playing action D, evenif player 1 would not defect (unless player 2 had already defected) when following the automaton. The analysis approach of considering single deviations from machine state profiles is suitable for games that are repeated an infinite times. As we observed in Section when discussing 60 Copyright 2014 Parkes & Seuken. Draft: Do not distribute without permission.

19 3.6 Folk Theorems the subgame-equivalence approach, in finitely repeated games the length of history also matters for the purpose of single-deviation analysis. 3.6 Folk Theorems In this section, we develop general descriptions of adaptive strategies that are subgame-perfect equilibria and are capable of sustaining cooperative outcomes in games such as the Prisoner s Dilemma. This style of result is referred to as a folk theorem because it was believed to be true in the folk law of game theory before being formally proved. An action profile a of the stage game is enforceable if there exists a (possibly mixed) Nash equilibrium strategy a of the stage game, for which ũ i (a) > ũ i (a )=e i, (3.9) for all agents i, wheree i is the expected utility to agent i given strategy a. We do not require that action profile a is a Nash equilibrium of the stage game. The strict inequality is important in obtaining the folk theorem. Example 3.4 (continued). In the Prisoner s Dilemma, the unique Nash equilibrium of the stage game is a =(D, D), withpayoff(1, 1) to each player. Given that the payoffs for the other action profiles are {(3, 3), (0, 5), (5, 0)}, the only enforceable action profile is a =(C, C). Theorem 3.6 (Folk Theorem). Given a stage game G with an enforceable action profile a, there exists a subgame-perfect equilibrium of the infinitely repeated game G 1, for all sufficiently large, where action profile a is played in equilibrium in every period. Proof. Let a denote the (perhaps mixed) Nash equilibrium in the stage game that enforces action profile a, such that payoff v i =ũ i (a) > ũ i (a )=e i, for every agent i. Consider the following candidate strategy for agent i in G 1 : (Cooperate): Play a i if everyone played according to action profile a in all previous periods. (Punish): Otherwise, play strategy a i in every subsequent round. Consider the profile where all agents play this strategy. To establish that this profile is a subgame-perfect equilibrium we follow the subgameequivalence approach. It is sufficient to consider two different kinds of subgames, since the candidate strategy is the same in all subgames in each of the two cases. (Case 1:) No player has deviated from action profile a in any previous period. By the single-deviation principle it is sufficient to check that, max ũ i (a 0 i,a i )+ a 0 i e i 1 apple v i 1, (3.10) for all agents i. The left-hand side is the immediate payoff to agent i from its best possible deviation, followed by the total discounted payoff it will receive in future periods given that it Copyright 2014 Parkes & Seuken. Draft: Do not distribute without permission. 61

20 3 Game Theory II: Sequential-Move and Repeated Games Figure 3.10: A revised grim trigger strategy. e deviates. We adopt i e 1 instead of i 1 because the infinite payoff stream is pushed back, and thus discounted, by one period. The right-hand side is the total discounted payoff to agent i from following the prescribed strategy. Equivalently, we require: (1 ) max ũ i (a 0 i,a i )+ e i apple v i (3.11) a 0 i Since e i <v i, and the maximum utility is bounded in the stage game, this is satisfied for some <1, if is chosen close enough to 1. (Case 2:) One or more players have deviated from action profile a in the past. In this case, the candidate strategy prescribes the Nash equilibrium a of the stage game in each subsequent period. This is an open-loop strategy, and the Nash equilibrium of the stage game, and thus a subgame-perfect equilibrium (by Theorem 3.5). Based on this, we can slightly change the grim trigger in Figure 3.8 to obtain the revised grim trigger strategy of Figure We show that this revised strategy is a subgame-perfect equilibrium in the infinitely repeated Prisoner s Dilemma. Example 3.4 (continued). A revised grim trigger strategy (Figure 3.10) has the following two phases: (Cooperate): Play C if the play has been (C, C) in all previous periods. (Punish): Otherwise, play D in every period. According to Theorem 3.6, this strategy is a subgame-perfect equilibrium of the infinitely repeated Prisoner s Dilemma for a suitably large <1, because (D, D) enforces action profile (C, C). The difference between this strategy and the grim trigger strategy in Figure 3.8 (c) is that a player now switches to defect if he defected himself. See Exercise 3.3 for further development. Example 3.4 confirms that it is possible to achieve outcomes in equilibrium that are not possible in the stage game. In the case of the Prisoner s Dilemma, we can sustain cooperation through the threat of punishment if either player defects. By adopting as a threat a Nash 62 Copyright 2014 Parkes & Seuken. Draft: Do not distribute without permission.