Texas hold em Poker AI implementation: Ander Guerrero Digipen Institute of technology Europe-Bilbao Virgen del Puerto 34, Edificio A 48508 Zierbena, Bizkaia ander.guerrero@digipen.edu This article describes how to implement a Texas hold em AI that behave as a real human; in order to create a game where the player enjoys and is able to win if is good enough. Poker is an interesting topic in the context of AI. There is incomplete information, one or more competing agents, a set of well-defined rules and a clear objective. The talk is divided in a couple of points that will explain how each part of the algorithm is implemented, the variations that exist to compute that part of the algorithm and why is implemented like that. The full article holds the knowledge get during the process of research and tries to transmit it to the reader, in order to encourage the readers to do each own algorithm with the resources explain during the paper. What is Texas hold em Texas hold em is a variation of five cards hand Poker, in this game you have two cards being dealt face down to each player and then five community cards that will be dealt face-up in this order: a series of three ( the flop ), then an addition single card ( the turn ) and another addition card ( the river ). Although it looks like you have three bet stages, truly you have four, because the first stage is done just with the private cards. Word Fold Check Call Bet Raise Description Give up betting and the entire round. When the current bet is cero you can do nothing (no bet) and still keep playing. Put the amount of the current bet. Put a bet when no others player have done so. Raise the amount of minimum bet. Hand The composition of the hand cards of each player with the community cards (max 5 of 7). Table 1 Description of some words that are commonly used in Poker.
Currently available software The Poker AI market is very big; there are a lot of products related to the Poker software. That kind of software is usually done to improve the quality of each player. The Poker market that covers improving each player s skills can be divided in some groups as statistical software that provides insight of the Poker players, offline training software that will give advice during play and various calculators. The rest of the market is covered by online casinos and a couple of games done to just entertain. The software part that covers the improvement of the quality of the players is not well seen in the casinos because they are trying to prevent players using software to automate play. Since the market is a little bit questionable, this paper is focused on the Poker AI for entertainment that can be used in some Poker games. The goal The goal was to construct a good and solid algorithm with the aim of implement a simple AI for a Poker game; since it is a probability game there is not a magic formula that holds the perfect solution always. The interesting parts of the Poker game are the betting parts, so Texas hold em was the best solution. Now, the goal becomes to create an algorithm that recreates the perfect ambient where the player seems to be playing against a real person. The AI has four main phases: - Calculate the hand value. - Calculate the hand strength. - Calculate the pot odds and Rate of return. - Choose the Fold/Call/Rise decision A.K.A. FCR decision. The algorithm The algorithm is a current combination of knowledge base of some players and a Monte-Carlo simulation. Calculate the hand value. The first part of this algorithm is very simple and it is almost used on every Poker game to compute the hand value. The hand value is a cost given to the AI hand taking into account the current possible combination. Each hand has each own value. The first step is to sort the hand in descending order; the second part is to check for all possible combinations; in descending order too.
Hand Example Straight flush Quads Full House Flush Straight Three of a kind Two pair Pair No Pair Table 2- Name and example of each possible combination in descending order. Calculate the Hand Strength The hand Strength is the chances that a hand has to win a game, where 0.0 is a lost and 1.0 is a sure win. To calculate this value there are some different approaches as knowledge-based formulas where an specific domain of Poker knowledge is store in a bot. There is an improved approach that holds with a simple formula if the money to bet is enough in order to avoid the bot going all in without a sure win. Although this is the most simple and straightforward option, you need to be a very good player or, at least, know what to do in almost every moment of the game.
There is a second algorithm that also works to calculate the hand strength that is based in decision-making prediction, as the Poker is a finite strategy game with finite decisions this approach is also valid, but not the best, because the possible decision although they are finite the possibilities are a lot. The approach that this algorithm is going to describe is Monte-Carlo simulation. Monte-Carlo simulation is based on the repetition of an operation that has some random parameters with the purpose of getting an approximation of the best outcome in almost every scenario possible. The strength of the simulation is related to the number of loops that the machine is able to do, since Poker is not a real time game, we are able to simulate the game a lot of times and get a very big precision. Create new deck float score = 0 Remove your hand and any known community cards Repeat X times Shuffle the new deck Deal remaining community cards and opponents hands Evaluate all hands If you have the best hand then score += 1/ (number of people with the same hand value) end loop Hand Strength = score/number of loops (1000 in this case) Code example 1 - Pseudo code the shows how to calculate hand strength. Calculate the pot odds and Rate of return The pot odds are the current relation between the entire round pot and the current bet. For example: if there is an amount of coins in the pot of 40 and to call we need an amount of 20, the pot odds will be 20/ (20 + 40). The rate of return is the relation between the hand strength and the pot odds; this last value is going to be used to decide which option is going to pick the AI fold rise or call. The algorithm until this point is the same for every AI behaviour because the values that we get are going to be the same for everyone. The next step is where the human factor is introduced. PotOdds = bet / (bet + pot) RateOfReturn = HandStrength / PotOdds Code example 2 - How to calculate the pot odds and the rate of return.
The Fold/Call/Rise decision A.K.A. FCR decision The FCR decision is the part of the algorithm in charge of deciding the action of the AI, here is also where the algorithm becomes knowledge base. Although the most of the algorithms uses one formula to treat almost every stage of the game, this algorithm will cover each stage separately, bluffs, bets or calls are different in each stage. Our goal in each stage, if the AI has good cards, is to try to convince the player not to fold and to continue playing until the end. The first stage ( Pre-Fold ) should be just a check of the sum of the cards and the rest is done with the hand strength. Changing this part of the algorithm will change how the AI behaves, this part should be a bunch of ifs and random checks that will determine whether the AI is great or not. This is the most important part, by tweaking those values we can emulate perfectly how a real human will behave creating a great experience. Conclusion The main goal of the paper was to cover a concrete algorithm in order to be applied to Texas hold em that simulates a real human behaviour. The market that covers Poker software is normally software done, just to train players or automate play. This market is trying to create the most complex AI to beat and therefore the AI is not being implemented to be enjoyed by casual players. This algorithm shows how to implement a non-very hard to beat AI that can be enjoyed by people like me that knows how to play, but is not a great player. The research done in Poker is very big, but since the game has a lot of possibilities and results there is not an algorithm that solves the entire game and, of course, the research done just implies Poker games where the amount to bet is limited. The investigation done in no limit Poker is very little, and therefore could be improved a lot; this algorithm also could be improving until reach an exact formula that holds almost every possible situation. The aim of the paper was to encourage people to create more playable algorithms that can be played by casual players and therefore, be used in games that can be sold as casual games for any platforms; and at the same time, challenging, not just bots that knows how to play better than you and are done to train people that knows how to play really well.
References Web Source: E. Schuijtvlot (2011). Application of AI in Poker VU University Amsterdam; Faculty of Science. Available online at: https://www.few.vu.nl/en/images/werkstukschuijtvlot_tcm39-225501.pdf West, M. (2007). Programming Poker AI Cowboy Programming. Available online at: http://cowboyprogramming.com/2007/01/04/programming-poker-ai/ Wikipedia (2014). Computer poker players. Available online at: http://en.wikipedia.org/wiki/computer_poker_players Wikipedia (2014). Texas hold em. Available online at: http://en.wikipedia.org/wiki/texas_hold%27em