CS 387: GAME AI BOARD GAMES 5/24/2016 Instructor: Santiago Ontañón santi@cs.drexel.edu Class website: https://www.cs.drexel.edu/~santi/teaching/2016/cs387/intro.html
Reminders Check BBVista site for the course regularly Also: https://www.cs.drexel.edu/~santi/teaching/2016/cs387/intro.html Thursday, project 4 submission deadline
Outline Board Games Game Tree Search Portfolio Search Monte Carlo Search UCT
Outline Board Games Game Tree Search Portfolio Search Monte Carlo Search UCT
Game AI Architecture AI Strategy Decision Making World Interface (perception) Movement
So far, we have seen: Perception Movement (Steering behaviors): FPS, Car driving Pathfinding FPS, RTS, RPG, etc. Decision Making FPS, RPG, RTS, etc. Tactics and Strategy FPS, RTS PCG Many genres.
Board Games Main characteristic: turn-based The AI has a lot of time to decide the next move
Board Games Not just chess
Board Games Not just chess
Board Games Not just chess
Board Games Not just chess
Board Games Not just chess
Board Games Not just chess
Board Games From an AI point of view: Turn-based Discrete actions Complete information (mostly) Those features make these games amenable to game tree search!
Outline Board Games Game Tree Search Portfolio Search Monte Carlo Search UCT
Game Tree Current Situation Player 1 action U(s) U(s) U(s)
Game Tree Current Situation Player 1 action Pick the action that leads to the state with maximum expected utility U(s) U(s) U(s)
Game Tree Game trees capture the effects of successive action executions: Current Situation Player 1 action Player 2 action U(s) U(s) U(s) U(s) U(s) U(s)
Game Tree Game trees capture the effects of successive action executions: Current Situation Player 1 action Pick the action that leads to the state with maximum expected utility after taking into account what the other players might do Player 2 action U(s) U(s) U(s) U(s) U(s) U(s)
Game Tree Game trees capture the effects of successive action executions: In this example, we look ahead only one Current Situation player 1 action and one player 2 action. But we could grow the tree arbitrarily deep Player 1 action Player 2 action U(s) U(s) U(s) U(s) U(s) U(s)
Minimax Principle Positive utility is good for player 1, and negative for player 2 Player 1 chooses actions that maximize U, player 2 chooses actions that minimize U Current Situation Player 1 action Player 2 action U(s) = -1 U(s) = 0 U(s) = -1 U(s) = 0 U(s) = 0 U(s) = 0
Minimax Principle Positive utility is good for player 1, and negative for player 2 Player 1 chooses actions that maximize U, player 2 chooses actions that minimize U Current Situation Only looking at the utility values, which move should player 1 choose? Player 1 action Player 2 action U(s) = -1 U(s) = 0 U(s) = -1 U(s) = 0 U(s) = 0 U(s) = 0
Minimax Principle Positive utility is good for player 1, and negative for player 2 Player 1 chooses actions that maximize U, player 2 chooses actions that minimize U Current Situation Player 1 action Player 2 action (min) U(s) = -1 U(s) = 0 U(s) = -1 U(s) = 0 U(s) = 0 U(s) = 0
Minimax Principle Positive utility is good for player 1, and negative for player 2 Player 1 chooses actions that maximize U, player 2 chooses actions that minimize U Current Situation Player 1 action Player 2 action (min) U(s) = -1 U(s) = -1 U(s) = 0 U(s) = -1 U(s) = 0 U(s) = -1 U(s) = 0 U(s) = 0 U(s) = 0
Minimax Principle Positive utility is good for player 1, and negative for player 2 Player 1 chooses actions that maximize U, player 2 chooses actions that minimize U Current Situation Player 1 action (max) Player 2 action (min) U(s) = -1 U(s) = -1 U(s) = 0 U(s) = -1 U(s) = 0 U(s) = -1 U(s) = 0 U(s) = 0 U(s) = 0
Minimax Algorithm Minimax(state, player, MAX_DEPTH) IF MAX_DEPTH == 0 RETURN (U(state),-) BestAction = null BestScore = null FOR Action in actions(player, state) (Score,Action2) = Minimax(result(action, state), nextplayer(player), MAX_DEPTH-1) IF BestScore == null (player == 1 && Score>BestScore) (player == 2 && Score<BestScore) BestScore = Score BestAction = Action ENDFOR RETURN (BestScore, BestAction)
Minimax Algorithm Minimax(state, player, MAX_DEPTH) IF MAX_DEPTH == 0 RETURN (U(state),-) BestAction = null BestScore = null FOR Action in actions(player, state) (Score,Action2) = Minimax(result(action, state), nextplayer(player), MAX_DEPTH-1) IF BestScore == null (player == 1 && Score>BestScore) (player == 2 && Score<BestScore) BestScore = Score BestAction = Action ENDFOR RETURN (BestScore, BestAction)
Minimax Algorithm Needs: Utility function U Way to determine which actions can a player execute in a given state MAX_DEPTH controls how deep is the search tree going to be: Size of the tree is exponential in MAX_DEPTH Branching factor is the number of moves that can be executed per state The higher MAX_DEPTH, the better the AI will play There are ways to increase speed: alpha-beta pruning
Minimax Algorithm Needs: Utility function U Way to determine which actions can a player execute in a given state MAX_DEPTH controls how deep is the search tree going to be: Size of the tree is exponential in MAX_DEPTH Branching factor is the number of moves that can be executed per state - Given: - Branching factor: B - Maximum tree depth: D The higher MAX_DEPTH, the better the AI will play - What is the time complexity? - What is the memory complexity? There are ways to increase speed: alpha-beta pruning
Successes of Minimax Deep Blue defeated Kasparov in Chess (1997) Checkers was completely solved by Jonathan Shaeffert (2007): If no players make mistakes, the game is a draw (like tick-tack-toe) Go: Using a variant of minimax, based on Monte Carlo Tree Search In 2011 The program Zen19S reached 4 dan (professional humans are rated between 1 to 9 dan) In 2016 AlphaGO defeated Lee SeDol (one of the best players in the world)
Interesting Uses of Minimax bastet (Bastard Tetris): http://blahg.res0l.net/2009/01/bastet-bastard-tetris/
Iterative Deepening As described before, minimax receives a MAX_DEPTH and it is impossible to predict how much time will it take to execute In a game, minimax will receive a certain amount of time (e.g. 20 seconds) that it can use to decide the next move Solution: iterative deepening
Iterative Deepening Idea: Open the tree at depth 1 If there is still time, open it at depth 2 If there is still time, open it at depth 3 Etc.
Iterative Deepening Idea: Open the tree at depth 1 If there is still time, open it at depth 2 If there is still time, open it at depth 3 Etc. If we end up searching up to depth, say 5, how much time is wasted?
Iterative Deepening Idea: Open the tree at depth 1 If there is still time, open it at depth 2 If there is still time, open it at depth 3 Etc. If we end up searching up to depth, say 5, how much time is wasted? Given the branching factor d, each subsequent iteration is d times larger in average than the previous. For typical values of d (larger than 10), the extra cost of iterative deepening is negligible
Alpha-Beta Pruning Not all the nodes in the search tree are relevant for deciding the next move 5 2 4 1 3 4 2 6 1
Alpha-Beta Pruning Not all the nodes in the search tree are relevant for deciding the next move 2 2 1 1 5 2 4 1 3 4 2 6 1
Alpha-Beta Pruning Not all the nodes in the search tree are relevant for deciding the next move 2 2 1 1 What would happen is this value was higher? What would happen if this value was lower? 5 2 4 1 3 4 2 6 1
Alpha-Beta Pruning Not all the nodes in the search tree are relevant for deciding the next move 2 2 1 1 What would happen is this value was higher? What would happen if this value was lower? NOTHING! 5 2 4 1 3 4 2 6 1
Alpha-Beta Pruning Not all the nodes in the search tree are relevant for deciding the next move 2 2 1 1 These two nodes are irrelevant! They do not have to be explored! This is because the first node has a 1, which is lower than the lowest found in any other branch so far 5 2 4 1 3 4 2 6 1
Minimax Algorithm α = - infinity β = infinity alphabeta(state, MAX_DEPTH, α, β, player) if MAX_DEPTH = 0 or state is a terminal state return U(state) if player= 1 for action in actions(player, state) α := max(α, alphabeta(result(action,state), MAX_DEPTH-1, α, β, 2)) if β α break return α else for action in actions(player, state) β := min(β, alphabeta(result(action,state), MAX_DEPTH-1, α, β, 1)) if β α break return β
Minimax Algorithm α = - infinity β = infinity alphabeta(state, MAX_DEPTH, α, β, player) if MAX_DEPTH = 0 or state is a terminal state return U(state) if player= 1 for action in actions(player, state) α := max(α, alphabeta(result(action,state), MAX_DEPTH-1, α, β, 2)) if β α break return α else for action in actions(player, state) β := min(β, alphabeta(result(action,state), MAX_DEPTH-1, α, β, 1)) if β α break return β
Alpha-Beta Pruning
Alpha-Beta Pruning Does pruning occur independently of the order in which nodes are visited? 2 2 1 1 5 2 4 1 3 4 2 6 1
Alpha-Beta Pruning Notice that pruning depends on the order in which the children are explored 2 2 1 1 If we expand the 1 first, then 2 and 6 do not have to be explored 5 2 4 1 3 4 2 6 1
Alpha-Beta Pruning How to decide a good order for children expansion?
Alpha-Beta Pruning How to decide a good order for children expansion? Idea: Iterative deepening Explore first the children that was selected as the best move in the previous iteration of iterative deepening With this modification, iterative deepening is actually faster in practice than just opening the tree at a given depth! Other domain specific heuristics exist for well known games such as Chess.
Outline Board Games Game Tree Search Portfolio Search Monte Carlo Search UCT
What is an Action? Consider a complex board game like Settlers or Scrabble, what is the set of actions a player can perform in her turn?
What is an Action? Consider a complex board game like Settlers or Scrabble, what is the set of actions a player can perform in her turn? Way too many actions to consider in minimax!!
Portfolio Search Consider the game of Monopoly The set of possible actions is too large (just imagine all possible deals we can offer any player!) However, we can do the following: We can devise 3 or 4 strategies to play the game: A. Never do any deal nor build any house, just roll dies and buy streets. B. Never do any deal, but build one house in the most expensive street we can. C. Never do any deal, but build as many houses as we can, in the cheapest street we can. D. Do not build houses, but offer a deal to get the cheapest full set we could get by trading a single card with one player (offering her some predefined amount of money, a factor of the price of what we are getting)
Portfolio Search Consider the game of Monopoly Certainly, these different strategies would do better in The set of possible actions different is too situations. large (just imagine all The key idea portfolio search is to consider these possible deals we can offer any player!) strategies as the actions. However, we can do the following: We can devise 3 or 4 strategies to play the game: A. Never do any deal nor build any house, just roll dies and buy streets. B. Never do any deal, but build one house in the most expensive street we can. C. Never do any deal, but build as many houses as we can, in the cheapest street we can. D. Do not build houses, but offer a deal to get the cheapest full set we could get by trading a single card with one player (offering her some predefined amount of money, a factor of the price of what we are getting)
Minimax Portfolio Search At each level of the tree, use each of the predefined strategies to generate the next action, and only consider those actions. Action proposed By strategy B Action proposed By strategy A Action proposed By strategy C
Minimax Portfolio Search At each level of the tree, use each of the predefined strategies to generate the next action, and only consider those actions. Action proposed By strategy B Action proposed By strategy A Action proposed By strategy C This simple idea can make minimax search feasible in games with a set of actions that is too large to consider the whole tree.
Simple Portfolio Search Forget about game trees, just do this: given a set of strategies S For each s1 in S: For each s2 in S: Simulate a game for D game cycles where player 1 uses s1, and player 2 uses s2 Compute the average reward obtained by each strategy s1, and select the one that achieved the highest average.
Portfolio Search Imagine this situation: Branching factor B Search up to a depth D We have a set of S strategies (S << B) What is the time / memory complexity of: Minimax? Minimax portfolio search? Simple portfolio search?
Portfolio Search Imagine this situation: Branching factor B Search up to a depth D We have a set of S strategies (S << B) What is the time / memory complexity of: Minimax? B^D, D Minimax portfolio search? S^D, D Simple portfolio search? D * S^2, 2
Portfolio Search Imagine this situation: In terms of play strength: Branching Minimax factor > B minimax portfolio search > simple portfolio search Search up to a depth In D terms of computational needs: We have Minimax a set of > S minimax strategies portfolio search > simple portfolio search What is the time / memory complexity of: Minimax? B^D, D Minimax portfolio search? S^D, D Simple portfolio search? D * S^2, 2 Thus, if you can use minimax, that s the simplest thing to do. But if you cannot (due to CPU constraints), portfolio search is a good option to consider.