Game Theory and MANETs: A Brief Tutorial

Transcription

1 Game Theory and MANETs: A Brief Tutorial Luiz A. DaSilva and Allen B. MacKenzie Slides available at GameTheoryTutorial.pdf 1

2 Agenda Fundamentals of Game Theory Decision Making and Utility Theory Normal Form Games Repeated Games Potential Games Game Theory and Wireless Networks: What the Future Holds Reference Allen MacKenzie and Luiz DaSilva, Game Theory for Wireless Engineers, Morgan & Claypool Publishers,

3 Reference Acknowledgements The Office of Naval Research, for their support of our research into game theoretic modeling of ad hoc networks The National Science Foundation, for their support of MacKenzie s research into game theoretic modeling of cooperation in wireless networks James Neel, Jeffrey Reed, Robert Gilles, Vivek Srivastava, Rekha Menon, Ramakant Komali, James Hicks and many others who have contributed significantly to this work 3

4 Agenda Fundamentals of Game Theory Decision Making and Utility Theory Normal Form Games Repeated Games Potential Games Game Theory and Wireless Networks: What the Future Holds What is Game Theory? A bag of analytical tools designed to help us understand the phenomena that we observe when decision-makers interact (Osborne and Rubinstein) The study of mathematical models of conflict and cooperation between intelligent rational decision-makers (Myerson) 4

5 Relevance to Ad Hoc Networks Ad hoc networks typically are Self-organizing Optimized in a decentralized fashion Resource-constrained (power, energy, bandwidth, channels) Game theory can be used to study the distributed decisions made by decision-makers (network nodes) to achieve some goal (maximize performance, minimize resource utilization) Some Potential Pitfalls Confusing an optimization problem and a game. Confusing cooperative and noncooperative game theory. Failing to carefully define the game under consideration. 5

6 Philosophical Pitfalls There are two philosophies for applying game theory: A modeling or direct application of the theory. An engineering or system design application of the theory. These philosophies are mutually exclusive! Examples Power control Nodes set power level to maximize their SINR Routing Source node selects a path to minimize delay Trust and reputation management Nodes decide to what extent to cooperate with others in performing network functions (service discovery, forwarding) Important: In each of these examples, one node s decision affects other nodes in the network 6

7 Agenda Fundamentals of Game Theory Decision Making and Utility Theory Normal Form Games Repeated Games Potential Games Game Theory and Wireless Networks: What the Future Holds Intelligent Decision Making Recall one of our definitions of game theory: The study of mathematical models of conflict and cooperation between intelligent rational decisionmakers (Myerson) What is an intelligent rational decision maker? And how can we model his or her behavior mathematically? The construction of mathematical models of intelligent, rational decision making is called decision theory. Game theory is multiagent decision theory! 7

8 Preference Relations Let X be any set, called the set of outcomes or alternatives. Let f i % be a binary relation on X. x, y " X f i is said to be complete if for all either % x f i y or y f i x. % % f i is said to be transitive if x f i y and y f i z imply % that x f z. % % i % f i % The binary relation is a preference relation if it is complete and transitive. Preference Relations A preference relation expresses an individual player s desirability of one outcome over another. f i % (Weak) Preference Relationship * i a a is preferred at least as much as a * by player i a f % fi Strict Preference Relationship * a f i a * iff a i a but not f % a * f % i a ~ i Indifference Relationship * a ~ i a * iff a i a and f % a * f % i a 8

9 Is This Reasonable? Is it reasonable to require that preferences be complete? Requires that we compare any two objects in X, even if they are unrelated. Is it reasonable to require that preferences be transitive? Requires that we make very fine distinctions. Examples of Preferences Application Layer: Users prefer highquality video over low quality video. Network Layer: Nodes prefer robust, reliable paths over transient, unreliable paths. Link Layer: Users prefer short medium access delays. Physical Layer: Users prefer high SINR and low BER. 9

10 Utility Representation We would like to represent preferences using real numbers. f % u i : X " # A preference relation i is said to be represented by a utility function when x f i y " u i (x) # u i (y). % When can we construct utility representations? If X is finite, or even countable, then we can always construct a utility representation for any preference relation. If X is uncountably infinite, then we may not be able to do so. Example: Lexicographic Preferences Let X = [0,1] x [0,1] (x 1,x 2 ) f i (y 1,y 2 ) if x 1 > y 1 or (x 1 =y 1 and x 2 y 2 ) % How important is this fact? 10

11 Preferences Over Lotteries In many cases, we must specify preferences over lotteries rather than certain outcomes. Which is better? (A) A WLAN connection with probability 0.7 and no connection with probability 0.3. (B) A cellular connection with probability 1. Representing Uncertainty Let Z be the set of outcomes. Let X be the set of choice objects, which are probability distributions over Z. Z = {WLAN, Cellular, none} X = {(p WLAN, p Cellular, 1-p WLAN -p Cellular )} Does there exist an expected utility representation for a preference relation on X? 11

12 Expected Utility Representations f % i A binary relation over X is said to have an expected utility representation if there exists a function u i : Z " # such that p f i q " E p [u(z)] # E q [u(z)] % E p means the expected value with respect to the probability distribution p. The von Neumann-Morgenstern Axioms are key to the existence of expected utility representations. Von Neumann-Morgenstern Axioms Axiom 1. The binary relation preference relation. f % i on X is a Axiom 2 (Independence). For all p,q,r " X and a![0,1], p f i q if and only if % ap + (1" a)r f i % aq + (1" a)r Is this sensible? Allais Paradox 12

13 Von Neumann-Morgenstern Axioms Axiom 3 (Archimedean). For all p,q,r " X such that p f q f r, there exist a,b " (0,1) such that ap + (1" a)r f q f bp + (1" b)r. Is this sensible? Z = {WiFi,DialUp,UntimelyDeath} Existence of Expected Utility Representations If Z is a finite set and X is the set of probability distributions on Z, then a binary relation f i on X % satisfies axioms 1, 2, and 3 if and only if there is an expected utility representation of f i. % For any Z, if f i is a binary relation defined on the % set X of simple probability distributions on Z, then f i satisfies axioms 1, 2, and 3 if and only if % there is an expected utility representation of f i. % 13

14 Agenda Fundamentals of Game Theory Decision Making and Utility Theory Normal Form Games Repeated Games Potential Games Game Theory and Wireless Networks: What the Future Holds Critical Components of a Game A (well-defined) set of 2 or more players A set of actions for each player. A set of preference relationships for each player for each possible action tuple. 14

15 Normal Form Games G = { } N, A, ui N A i Set of players Set of actions available to player i A Action space A = A1! A2! L! An {u i } Set of individual objective functions Normal Form Game Example Resource sharing in a network There is a cost to sharing, but if everyone refuses to share all will suffer What is the expected outcome of this game? Note: this is a 3-player version of the Prisoner s Dilemma 15

16 Dominated Strategies Sometimes it is possible to predict an outcome to the game based on decisions a rational player would make (or the elimination of strategies a rational player would not make) A pure strategy s i is strictly dominated for player i if there exists s i*! S i such that ( s *, s ) > u ( s, s ) # s S u i i! i i i! i! i "! i Further, s i is strictly dominated with respect to if there exists s i*! S i such that ( s *, s ) > u ( s, s ) # s A u i i! i i i! i! i "! i A! i " S! i Iterative deletion of dominated strategies Player 2 L R Player 1 L M R 1,1 2,0 0,3 0.5, 1.5 1,0.5 0,2 16

17 Can we predict the outcome of a paper/rock/scissors game? Player 2 0,0 1,-1-1,1 Player 1-1,1 1,-1 0,0-1,1 1,-1 0,0 Mixed Strategies A player can randomize over her strategy set Denote by σ i a mixed strategy available to player i And σ i (s i ) is the probability that the mixed strategy assigns to s i The expected utility to player I under a joint mixed strategy is u ( $ ) = ( $ ( s N i! " s# S j= 1 j j )) u ( s) i 17

18 Nash Equilibrium A point from which no user can benefit by unilaterally deviating An action tuple a is a Nash equilibrium if, for every player i in N and every action b i in A i, ui ( a ) " ui ( bi, a! i ) More generally expressed in terms of mixed strategies Existence of the Nash Equilibrium Every finite game in strategic form has a Nash equilibrium in pure or mixed strategies The existence of the Nash equilibrium can also be determined for some classes of games with infinite strategy spaces Proof of existence usually relies on fixed point theorems 18

19 Predictive Power of the NE A consistent prediction of the outcome of the game If all players predict the NE, it is reasonable to assume that they will play it Once reached, there is no reason to believe any player will deviate, and the system will remain in equilibrium until conditions change But not without its issues If players start from an action profile that is not an NE, are we sure they eventually reach the NE? (Convergence) What if there are multiple NEs? Is one more likely than the others? (Refinements to the concept of NE) Vulnerable to deviations by a coalition of players Pricing of network services Game theory can be used to design pricing structures that maximize network profitability or social welfare (e.g., sum of users utilities) Model Player strategy: level of service requested of the network, traffic profile Utility: difference between how much a user values a given level of QoS and the price she pays for it (customer surplus) Models in the literature for both static and dynamic pricing 19

20 Flow control model Model a finite set of users sharing a network of queues Player strategy: rate at which the player offers traffic to the network at each available service class Constrained by a fixed maximum rate and maximum number of outstanding packets in the network Performance objective: select an admissible flow control strategy that maximizes average throughput subject to an upper bound on average delay Agenda Fundamentals of Game Theory Decision Making and Utility Theory Normal Form Games Repeated Games Potential Games Game Theory and Wireless Networks: What the Future Holds 20

21 Extensive form The game is represented as a tree Each vertex represents a decision point for one of the players Edges from a vertex represent possible actions available to the player At the leaves, we specify payoffs to each player by following that path from the root Extensive form can account for different information sets Describe how much a player knows when asked to select an action Extensive and normal forms Every game in strategic form can also be represented in extensive form And vice-versa Extensive form does not necessarily imply sequential actions But particularly convenient to represent games that involve sequential actions 21

22 Node cooperation revisited information set Another example 1) What are the equilibria in this game? 2) All these equilibria equally likely? 22

23 A subgame Take a vertex x in an extensive form game Let F(x) represent the set of vertices and edges that follow x, including x A subgame is a subset of the original game such that 1. It is rooted at vertex x, which is the only vertex of that information set; 2. The game contains all vertices in F(x); 3. If a vertex in a particular information set is contained in the subgame, then all vertices in that information set are also contained. Subgame perfection A proper subgame of a game Γ is a subgame whose root is not the root of Γ A subgame perfect equilibrium of game Γ is a Nash equilibrium of G that is also a Nash equilibrium of every proper subgame of Γ 23

24 Repeated Games Players interact repeatedly within a potentially infinite time horizon. Used to model ideas of reputation and punishment in games and in ad hoc networks. Our brief introduction: Setup and Strategies The Equilibria An Example Repeated Games: The Setting A strategic form game, known as the stage game, is played repeatedly. In each stage, all players know the past actions taken by all other players. Players strive to maximize their expected payoff over multiple rounds of the game, using a discounted sum of payoffs. The discount rate, 0 " # <1, expresses how much players value the present over the future. $ % u i = (1"#) (#) k g i (a k ) k= 0 24

25 Repeated Games: The Strategies A history is a record of all actions played by all players in the past. h k = (a 0,a 1,a 2,...,a k ) A player s strategy is a mapping from histories to actions. a i k = f i (h k"1 ) Repeated Games: Equilibria The definition of Nash Equilibrium still applies to repeated games. A Nash Equilibrium strategy profile is one such that, for each player, her chosen strategy maximizes her expected payoff, given the chosen strategies of the other players. Often for repeated games, the NE is refined to the subgame perfect equilbrium. This refinement rules out equilibria which contain empty threats. 25

26 Repeated Game Example: Node Cooperation Consider the repeated P2P file sharing game. With the grim trigger strategy, a player plays Share until an opponent plays Not Share, after which she plays Not Share. All players playing the grim trigger strategy is a Nash equilibrium. But there are infinitely many Nash equilibria Folk Theorems A Folk Theorem considers a subclass of games and identifies a set of payoffs that are feasible under some equilibrium strategy profile Many subclasses of games = many Folk Theorems 26

27 Feasible payoff vector The convex hull of a set U is the smallest convex set that contains U The stage game payoff vector v=(v 1, v 2,, v N ) is feasible if it is an element of the convex hull of pure strategy payoffs for the game Min-max payoff The min-max (or reservation) payoff establishes the best payoff that each player can guarantee for herself, regardless of others actions The min-max payoff for player i is v i = min $ " A ) max$ #" ( A ) gi ( $ i, $! i #! i i i! ( i ) 27

28 Feasible individually rational payoffs In any Nash equilibrium of the repeated game, player i s payoff is at least her reservation payoff The set of feasible strictly individually rational payoffs is { v! V v > v " i N} i i! Folk Theorem In a repeated game, any combination of payoffs such that each player gets at least her min-max payoff is sustainable, provided that each player believes the game will be repeated with high probability Thm: For every feasible strictly individually rational payoff vector v, there exists! <1 such that for all! "(!,1) there is a Nash equilibrium of the game with payoffs v. 28

29 Power control in CDMA Model utility as an increasing function of SINR and a decreasing function of power Single-stage game yields a unique, but inefficient, Nash equilibrium Repeated game yields power assignments that result in fair/efficient network operation Under the threat of punishment of users that deviate from the strategy (other users threaten to increase their transmit power to the Pareto-inefficient levels dictated by the NE of the single-stage game) MAC in Aloha In each slot, users independently choose to transmit If more than one user transmits, collision results Payoffs are subject to a per-slot discount factor δ There exists a value of the cost of failed transmission for which the aggregate throughput achieved in the game equals the maximum throughput of a slotted Aloha system where transmit decisions are made in a centralized manner 29

30 Agenda Fundamentals of Game Theory Decision Making and Utility Theory Normal Form Games Repeated Games Potential Games Game Theory and Wireless Networks: What the Future Holds Potential Games Why Nash Equilibrium? Multiple equilibria may exist - infinitely many in case of repeated games No guarantee of convergence Possibly inefficient In potential games very simple adaptation strategies lead to Nash Equilibrium play. Under certain conditions, these equilibria may also be efficient! 30

31 Best Reply Dynamic Consider a normal form game played repeatedly. At each stage, exactly one player is offered the opportunity to change her action from the previous stage. With the best reply strategy, a player will always switch to an action which is a best reply to the current actions of other players. Better Reply Dynamic With the better reply strategy, a player will always switch to an action which gives a higher payoff than her current action, given the current actions of other players. With the random better reply strategy, a player will choose an action at random from those which yield a higher payoff than her current action, given the current actions of other players. 31

32 Potential Games: Definition An exact potential function, is a function V : A " R such that for any player, i, and any action tuples, V (a i,a "i ) "V (b i,a "i ) = u i (a i,a "i ) " u i (b i,a "i ) A game with an exact potential function is called an exact potential game. Exact Potential Game Example Exact Potential Function

33 Potential Games: Definition An ordinal potential function, is a function V : A " R such that for any player, i, and any action tuples, V (a i,a "i ) "V (b i,a "i ) > 0 # u i (a i,a "i ) " u i (b i,a "i ) > 0 A game with an ordinal potential function is called an ordinal potential game. Potential Games: Properties Potential function maximizers are guaranteed to be Nash Equilibria. If a point maximizes an exact or ordinal potential function, then an unilateral deviation cannot possibly increase a players utility. Furthermore, under simple conditions, all potential games have at least one pure strategy NE. All finite potential games have at least one pure strategy NE. If the strategy space S is compact and the potential function is continuous, then the game must have at least one pure strategy NE. 33

34 Exact Potential Game Example Exact Potential Function Potential Games: Properties Finite Improvement Path Property An improvement path is a sequence of strategy profiles {s 1,s 2,s 3, } such that only one player, i n, changes strategy from s n-1 to s n and For finite potential games, all improvement paths have are of finite length. Paper-Rock-Scissors is not a potential game! u i n (s n"1 ) < u i n (s n ) 0,0-1,1 1,-1 1,-1 0,0-1,1-1,1 1,-1 0,0 34

35 Potential Games: Best Response Convergence For potential games with pure strategy NE, the best response dynamic will converge to a NE. For finite games, this can be proven using the finite improvement path property. For infinite games (with compact action spaces) this can be proven using theorems from nonlinear programming. Potential Games: Better Response Convergence For finite potential games, the better response dynamic will converge to a NE. Again, this can be proven using the finite improvement path property. For infinite potential games with NE, the random better response dynamic will converge to a NE. This is a recent result which we have not yet published, but hope to publish soon. 35

36 Potential Games: Identification Unfortunately, given an arbitrary game, it can be difficult to determine whether or not a game is a potential game. This problem is especially difficult for ordinal potential games. We will provide some hints for exact potential games. Potential Games: Identification A coordination game is a game in which all users have the same utility function. That is u i (s) = C(s) Obviously, a coordination game is an exact potential game. A dummy game is a game in which each player s payoff is a function of only the actions of other players. That is, for each i, u i (s) = D i (s "i ) Dummy games are also exact potential games. Every exact potential game can be written as the sum of a coordination game and a dummy game. 36

37 Sum Example ,0.5,0.5-2,0.5, ,-2,-2-2,-2, = ,-2,0.5-2,-4.5,-2-4.5,-2,-2-4.5,-4.5,-4.5 Coordination Game Dummy Game Potential Game Potential Games: Interpretation In some cases, the potential function may also serve a social welfare function. In such cases, the NE which are potential function maximizers are efficient and can result from simple adaptation processes. 37

38 Potential Game Application: Interference Avoidance In a CDMA system, users are differentiated by their choice of spreading codes. A user s spreading code can be viewed as a strategy. The payoff to the strategy will be related to its orthogonality with (or, rather, lack of correlation with) the spreading codes of other users. Interference Avoidance Suppose that each player chooses a spreading code, s i, from S i = {s i " # M s i =1} Let S be an M x N matrix whose ith column is s i. Also part of the model is additive M- dimensional white Gaussian noise, assumed to have an M x M covariance matrix R z. 38

39 Interference Avoidance One possible utility function is the SINR for a correlation receiver: 1 u i (S) = s T i R i s i Here R i = S "i S T "i + R z, which is the autocorrelation matrix of the interference plus noise. It turns out that the negated generalized total squared correlation function is an ordinal potential function: V (S) = " SS T + R z F 2 Agenda Fundamentals of Game Theory Decision Making and Utility Theory Normal Form Games Repeated Games Potential Games Game Theory and Wireless Networks: What the Future Holds 39

40 The Role of Information in Distributed Decisions Nodes in ad hoc networks must make distributed decisions But in practice, they have limited information regarding other network participants, channel and network conditions The cost of anarchy and the cost of ignorance How much control information must be disseminated in the network to enable nodes to make sound decisions? Game-theoretic models Imperfect information a node does not know the type of others in the network (e.g., different utility functions) Imperfect monitoring a node assesses others actions probabilistically based on a private or public signal Cognitive Radios and Learning Radios react to changes in the channel (spectrum utilization), end-to-end performance assessment, and learn from outcomes of their past decisions How will these radios learn and how fast will they arrive at effective solutions? Machine learning may hold part of the answer Game theory can help, especially in assessing convergence properties 40

41 Emergent Behavior Behavior exhibited by a group that cannot be ascribed to any particular member of the group No central control, yet generally yields benefits to the group as a whole (birds flocking, termite mounds, cities that self-organize) Simple decisions lead to global efficiency Local information can lead to global wisdom Game theoretic models can be used to Determine whether simple decisions by network nodes lead to optimal (or near-optimal) outcomes Assess how much a node must observe its neighbors to make effective decisions Mechanism Design Area of game theory that studies how to engineer incentive mechanisms that lead independent, selfinterested participants to outcomes that are globally desirable Selfish decisions often lead to inefficient equilibria What incentives are appropriate to steer nodes towards desirable equilibria? Game theory often use to analyze wireless communications problems Mechanism design can provide a formal framework to the design of protocols 41

42 Modeling of Mobility Most current work analyzes the network under static (or at least stable) conditions Can be viewed as a snapshot of current network conditions Game theory can be used to analyze the impact of mobility Perturbations to the equilibrium Learning and convergence under constant change Summary Game theory is applied to model conflict and cooperation among independent, rational decision makers Natural application to the modeling of ad hoc networks Nash equilibrium is a consistent predictor of the outcome of a game Refinements, such as subgame perfect equilibrium, apply to certain classes of games, such as repeated games Convergence is an important issue Potential games possess useful convergence properties for better and best response strategies Cognitive networks, emergent behavior, the cost of ignorance are potential areas of investigation using GT 42