TEMPORAL DIFFERENCE LEARNING IN CHINESE CHESS

Size: px
Start display at page:

Download "TEMPORAL DIFFERENCE LEARNING IN CHINESE CHESS"

Transcription

1 TEMPORAL DIFFERENCE LEARNING IN CHINESE CHESS Thong B. Trinh, Anwer S. Bashi, Nikhil Deshpande Department of Electrical Engineering University of New Orleans New Orleans, LA Tel: (504) Fax: (504) Abstract Reinforcement learning, in general, has not been totally successful at solving complex realworld problems which can be described by nonlinear functions. However, temporal difference learning is a type of reinforcement learning algorithm that has been researched and applied to various prediction problems with promising results. This paper discusses the application of temporal-difference learning in the training of a neural network to play a scaled-down version of the board game Chinese Chess. Preliminary results show that this technique is favorable for producing desired results. In test cases where minimal factors of the game are presented, the network responds favorably. However, when introducing more complexity, the network does not function as well, but generally produces reasonable results. These results indicate that temporal difference learning has the potential to solve real-world problems of equal or greater complexity. Continuing research in the application of neural networks to complex strategic games will most likely lead to more responsive and accurate systems in the future. Introduction Neural networks have been widely implemented for solving various problems, mostly classification, pattern recognition, and function approximations. However, they have not been researched as extensively in the science of games. This shortcoming may be due to two reasons. Games may be considered trivial subjects not worthy of serious research, and neural networks have not yet evolved into a precise science. Thus, complex problems such as those presented by strategic games cannot be readily fed into a generic neural network and be expected to output reasonable results. There are many areas in neural networks that have yet to be discovered and explored in order to sufficiently handle or even solve complex problems. Artificial intelligence (AI) today is just that -- artificial, and only capable of solving a fraction of what genuine intelligence (humans) can solve. Therein lies the justification for designing neural networks to play games and solve puzzles. Though they may seem to not yield any obvious applications in other fields, through this research new learning methods may be developed that can be applied to similar real-life problems and new insights may be gained.

2 Chinese Chess Not much research has been done or at least published in the field of strategic learning in games. The only games that have been successfully implemented via neural networks are Checkers and Backgammon [1]. In fact, the neural network engine used is able to learn the game from zero-knowledge merely by self-play and learning from observed consequences. Chess and other strategic board games have been deemed too complex to be fully implemented via neural networks, and so progress in that field has been hindered. Some have even totally discounted neural networks as ever being able to learn the game well enough to challenge human master-level players. However, the engine that has been proven to be successful in a board game, namely Backgammon, is a learning algorithm called Temporal Difference. This method seems versatile enough that it may hold promise for other strategic games in general, and this is the learning method that is used in this project to train a neural network to play the Asian variant of the popular European game of Chess - - Chinese Chess. Game description Chinese Chess is played in many of the Asian countries and it is very similar to Chess, having many similar pieces with similar moves. A brief description of the games is necessary for understanding the scope of this project. The actual game contains other pieces that this report does not address and the rules governing the games are far more detailed than this project will cover. Developing a neural network to play the full game of Chinese Chess would obviously require many more years of research. Thus, only a scaled-down version of the game is implemented in order to make the game more manageable. Only half of the actual playing field is used with a few selected playing pieces. The game consists of the battleground or playing field represented by square boxes as in Chess (see Figure 1). Figure 1. Chinese Chess Board and Pieces

3 However, the pieces are positioned on the intersections, not within the boxes. The object of this two-player game is to capture the opponent s Chief (or King) by gradually penetrating the opponent's defenses by eliminating the player's pieces. The two opposing sides initially have the same playing pieces but with different colors, usually red and blue. The pieces relevant to this project are the Chief, who is confined to one-block movements in a central square-area called the Palace. His resources include soldiers, cannons, horses and chariots. Soldiers are restricted to one-block forward progressions. Chariots have the freedom to slide in vertical or horizontal directions with no limit on distance. Cannons move the same way as chariots, but can only eliminate an opponent s piece by jumping over another piece as in checkers. Horses move in L-shaped patterns as knights do in Chess. Game Design To implement this game, the developer must consider which parameters of the game are necessary for inputs to the network to solve the problem. First, the board must be represented. Positions of each piece on the board must also be represented. Finally, the system must consider possible moves of each player s pieces and evaluate and decide on a best move. Neural networks can be designed to perform most of these operations. The board layout and positions of playing pieces are easily and distinctly represented by numbers, as is done in some chess board representations [2]. For example, all positions on the board are represented by a matrix of the same dimension as the playing field. Unique numbers (weighing the value of a certain piece) are assigned to each distinct playing piece, and opposing sides are differentiated by a leading signed or unsigned value [3]. The number +5 may represent the Chief on the red or blue side whereas a -5 would designate the Chief on the other side. In order to consider possible moves of each playing piece on the game board, an extensive search must be applied to find possible moves. As neural networks have not been specifically designed to perform searches, this part of the project requires computer software to perform searches that would generate possible moves for each piece. The details of how a computer program performs searches beyond the scope of this paper, but the concept is basically a tree that branches out indicating the possible sequential positions of each piece that is movable [3]. Thus, the moves in this game are updated sequential board layouts represented by a matrix and serve as inputs to the network. The neural network s purpose in this project is to evaluate the possible moves and determine the best move, which would contribute to ultimately winning a game. Learning Algorithm The inputs to the network would be the positions of the playing pieces. Pieces are weighted by value and are assigned a number, as shown in Table 1. Table 1. Weighted Value of Playing Pieces Player Piece Red Side Blue Side Soldier +1-1 Cannon +2-2

4 Horse +3-3 Chariot +4-4 Chief +5-5 The output of the network would be a single value to indicate the evaluation of each possible move. For each possible move, the network is input with an update of the matrix board representation indicating the new position of a piece. With this information, the network attempts to evaluate possible moves (generated by a computer search program) and suggest the best one. Learning is achieved via a method similar to backpropagation - temporal difference. Unsupervised learning algorithms allow a neural network to discover the underlying structure of a complex problem or function that would otherwise be indecipherable [4]. Temporal difference learning is an unsupervised reinforcement learning method for increasing the accuracy of predictions by recursively learning a prediction function via feedback [5]. Weights are updated from the difference of temporally successive predictions rather than the difference between the predicted and actual outputs, as is the case for most supervised learning algorithms [6]. Random events that appear to have no conceivable pattern can be described approximately by some function. An unsupervised neural network may learn to discover this complex random pattern or function. In this paper, temporal difference attempts to assign credit to each move by taking the difference between temporally successive predictions of the network. The usual backpropagation supervised approach involves feeding back the error calculated from difference of the current actual network output and the desired target. Temporal difference incorporates a slight modification in attempting to decrease errors by taking the difference between the two most recent successive predictions of the network and feeding that back to adjust the weights accordingly to better match the desired output. The error can be thought of as a sum of changes in the predictions. Thus, weights are updated according to the following equation: new weight = (current weight) + (sum of changes in predictions) n w = final w + initial w t t = 1 where W is the weight matrix updated by the sum of the weight changes for each of n observations at time t. Weight change is traditionally computed by ( ) P w = α z Pt w t where a is the learning rate and the gradient is the vector of partial derivatives of the predictions P t at time t with respect to the weight vectors w. The term (z - P t ) represents the error or difference between the predicted outcome and the actual outcome. However, in temporal difference, this equation must be modified so that the error is redefined to be the difference between temporally successive i th predictions

5 and the weight-change equation becomes n ( ) ( ) z Pt Pi+ 1 Pi i= t ( ) w = α t t + 1 t w k = 1 P P P k in which the change in weights will depend on the successively determined predictions and the sum of changes in past values of the previous predictions [4]. The transfer function of the neural network is chosen to be a hyperbolic tangent sigmoid, and the output is a single value taking on a value ranging from -1 to +1. Negative values indicate that the blue side has won, is winning or is making good moves. Positive values indicate a similar situation for the red side. Implementation and Testing The above learning algorithm is programmed into a neural network. Inputs to the network are possible moves (represented by changes in the board game s matrix) and outputs indicate what the network predicts to be the "value" of each move - good or bad (i.e., if it would contribute to the ultimate goal of reaching +1 or -1 for a game victory). Several cases (move sequences of different games) are presented to train the neural network from zero-knowledge. To begin training at a rudimentary level, only a few pieces are played in these game sequences. Initial random weights are adjusted by the algorithm such that the pattern of predicted outcomes of each move match the final outcome of the game (ultimate network output). After the network has been trained and the weights have been adjusted, test cases are fed into the network. When the network is fed the trained set of input data, the network predicted output correctly matches the actual outcome of the game. When new game sequences are input, the neural network is able to correctly predict the outcome of the game most of the times. These preliminary results indicate that the network is able to learn the game and capable of evaluating each move s contribution to the final outcome of the game. Following are examples of tests performed on the neural network. In the end-game case involving only the horse and chief on a 4x4 playing field, the neuralnetwork-controlled horse could check the opponent s chief by either of two moves. For the two different scenarios, the neural network indicated that they are equally good moves since they ultimately lead to the same result. Good moves are indicated by output values increasing towards positive unity for the neural-networkcontrolled side. Good moves for the opponent would be indicated by increasingly negative output values. The network is also tested for accuracy. When the neural-networkcontrolled horse placed the opponent s chief in check, but did not follow through by eliminating the Chief on the next move, the neural network detected this as a weak move and the output value correspondingly decreased. This confirmed that the

6 network was correctly responding to the position of the players pieces. To further test and confirm proper behavior of the network, the neural network was forced to evaluate bad moves. The neural-network-controlled pieces were presented to the network as moving into positions that would favor the opponent. The network again yielded correct results, since the output values became increasingly negative (favoring the opponent) as the bad moves were being made. Finally, the network is asked to evaluate a complete sequence of moves of a scaled-down version of the game. Since this involved a larger playing field and more pieces than the test cases, the network did not function consistently, although results do indicate that it generally functions correctly. In some cases where the opposing side made a good move, the network deemed it as a good move for the other side. However, in most other cases, the network correctly evaluated the moves as favoring one side or the other by the shifting output values. Of course, good moves are subjective, so they can understandably be considered subjective to an artificial intelligence that is the neural network. However, this network could be trained with more game sequences, perhaps thousands more, to improve its accuracy in evaluating each move as they pertain to the end-game result. And the advantage of temporal difference learning is that it is capable of learning by self-play since errors of predictions are fed back and weights are adjusted accordingly to match the actual output. Conclusion A neural network modeled on an effective learning algorithm such as the temporal difference method has been shown to be capable of playing complex games. The temporal difference algorithm used to train the network in this project shows promise for further development in this area. In the most basic setup, the network responds almost flawlessly. With more complexity introduced, the network sometimes may incorrectly evaluate the value of a particular move, but with additional training, the network outputs will most definitely improve. The results of this paper support the use of the temporal difference method as a potentially robust method for neural networks to solve complex problems. This learning algorithm is a promising starting point for similar developments in solving problems in other fields involving similar levels of complexity, and the method may also be refined and/or built upon with further research. References [1] S. Thrun. Learning to Play the Game of Chess. Advances in Neural Information Processing Systems, 7: , [2] M. Schmidt. Temporal Difference Learning and Chess. Technical Report, Aarhus University, Computer Science Department, June 20, [3] M. Schmidt. Neural Networks and Chess. Thesis, Aarhus University, Computer Science Department, July 19, 1993.

7 [4] C.L. Isbell. Explorations of the Practical Issues of Learning Prediction Control Tasks Using Temporal Difference Learning Methods. Master Thesis, Massachusetts Institute of Technology, December [5] G. Tesauro. Temporal Difference Learning and TD-Gammon. Communications of the ACM, 38(3): [6] R.S. Sutton. On Step-Size and Bias in Temporal-Difference Learning. Proceedings of the Eighth Yale Workshop on Adaptive and Learning Systems, pp91-96, 1994.

Reinforcement Learning in Games Autonomous Learning Systems Seminar

Reinforcement Learning in Games Autonomous Learning Systems Seminar Reinforcement Learning in Games Autonomous Learning Systems Seminar Matthias Zöllner Intelligent Autonomous Systems TU-Darmstadt zoellner@rbg.informatik.tu-darmstadt.de Betreuer: Gerhard Neumann Abstract

More information

TD-Gammon, a Self-Teaching Backgammon Program, Achieves Master-Level Play

TD-Gammon, a Self-Teaching Backgammon Program, Achieves Master-Level Play NOTE Communicated by Richard Sutton TD-Gammon, a Self-Teaching Backgammon Program, Achieves Master-Level Play Gerald Tesauro IBM Thomas 1. Watson Research Center, I? 0. Box 704, Yorktozon Heights, NY 10598

More information

Training a Back-Propagation Network with Temporal Difference Learning and a database for the board game Pente

Training a Back-Propagation Network with Temporal Difference Learning and a database for the board game Pente Training a Back-Propagation Network with Temporal Difference Learning and a database for the board game Pente Valentijn Muijrers 3275183 Valentijn.Muijrers@phil.uu.nl Supervisor: Gerard Vreeswijk 7,5 ECTS

More information

CPS331 Lecture: Search in Games last revised 2/16/10

CPS331 Lecture: Search in Games last revised 2/16/10 CPS331 Lecture: Search in Games last revised 2/16/10 Objectives: 1. To introduce mini-max search 2. To introduce the use of static evaluation functions 3. To introduce alpha-beta pruning Materials: 1.

More information

CS221 Project Final Report Gomoku Game Agent

CS221 Project Final Report Gomoku Game Agent CS221 Project Final Report Gomoku Game Agent Qiao Tan qtan@stanford.edu Xiaoti Hu xiaotihu@stanford.edu 1 Introduction Gomoku, also know as five-in-a-row, is a strategy board game which is traditionally

More information

Temporal-Difference Learning in Self-Play Training

Temporal-Difference Learning in Self-Play Training Temporal-Difference Learning in Self-Play Training Clifford Kotnik Jugal Kalita University of Colorado at Colorado Springs, Colorado Springs, Colorado 80918 CLKOTNIK@ATT.NET KALITA@EAS.UCCS.EDU Abstract

More information

Game Design Verification using Reinforcement Learning

Game Design Verification using Reinforcement Learning Game Design Verification using Reinforcement Learning Eirini Ntoutsi Dimitris Kalles AHEAD Relationship Mediators S.A., 65 Othonos-Amalias St, 262 21 Patras, Greece and Department of Computer Engineering

More information

ECE 517: Reinforcement Learning in Artificial Intelligence

ECE 517: Reinforcement Learning in Artificial Intelligence ECE 517: Reinforcement Learning in Artificial Intelligence Lecture 17: Case Studies and Gradient Policy October 29, 2015 Dr. Itamar Arel College of Engineering Department of Electrical Engineering and

More information

Five-In-Row with Local Evaluation and Beam Search

Five-In-Row with Local Evaluation and Beam Search Five-In-Row with Local Evaluation and Beam Search Jiun-Hung Chen and Adrienne X. Wang jhchen@cs axwang@cs Abstract This report provides a brief overview of the game of five-in-row, also known as Go-Moku,

More information

Teaching a Neural Network to Play Konane

Teaching a Neural Network to Play Konane Teaching a Neural Network to Play Konane Darby Thompson Spring 5 Abstract A common approach to game playing in Artificial Intelligence involves the use of the Minimax algorithm and a static evaluation

More information

Google DeepMind s AlphaGo vs. world Go champion Lee Sedol

Google DeepMind s AlphaGo vs. world Go champion Lee Sedol Google DeepMind s AlphaGo vs. world Go champion Lee Sedol Review of Nature paper: Mastering the game of Go with Deep Neural Networks & Tree Search Tapani Raiko Thanks to Antti Tarvainen for some slides

More information

An Artificially Intelligent Ludo Player

An Artificially Intelligent Ludo Player An Artificially Intelligent Ludo Player Andres Calderon Jaramillo and Deepak Aravindakshan Colorado State University {andrescj, deepakar}@cs.colostate.edu Abstract This project replicates results reported

More information

TD-Leaf(λ) Giraffe: Using Deep Reinforcement Learning to Play Chess. Stefan Lüttgen

TD-Leaf(λ) Giraffe: Using Deep Reinforcement Learning to Play Chess. Stefan Lüttgen TD-Leaf(λ) Giraffe: Using Deep Reinforcement Learning to Play Chess Stefan Lüttgen Motivation Learn to play chess Computer approach different than human one Humans search more selective: Kasparov (3-5

More information

Foundations of AI. 6. Adversarial Search. Search Strategies for Games, Games with Chance, State of the Art. Wolfram Burgard & Bernhard Nebel

Foundations of AI. 6. Adversarial Search. Search Strategies for Games, Games with Chance, State of the Art. Wolfram Burgard & Bernhard Nebel Foundations of AI 6. Adversarial Search Search Strategies for Games, Games with Chance, State of the Art Wolfram Burgard & Bernhard Nebel Contents Game Theory Board Games Minimax Search Alpha-Beta Search

More information

Learning Unit Values in Wargus Using Temporal Differences

Learning Unit Values in Wargus Using Temporal Differences Learning Unit Values in Wargus Using Temporal Differences P.J.M. Kerbusch 16th June 2005 Abstract In order to use a learning method in a computer game to improve the perfomance of computer controlled entities,

More information

IMPROVING TOWER DEFENSE GAME AI (DIFFERENTIAL EVOLUTION VS EVOLUTIONARY PROGRAMMING) CHEAH KEEI YUAN

IMPROVING TOWER DEFENSE GAME AI (DIFFERENTIAL EVOLUTION VS EVOLUTIONARY PROGRAMMING) CHEAH KEEI YUAN IMPROVING TOWER DEFENSE GAME AI (DIFFERENTIAL EVOLUTION VS EVOLUTIONARY PROGRAMMING) CHEAH KEEI YUAN FACULTY OF COMPUTING AND INFORMATICS UNIVERSITY MALAYSIA SABAH 2014 ABSTRACT The use of Artificial Intelligence

More information

Feature Learning Using State Differences

Feature Learning Using State Differences Feature Learning Using State Differences Mesut Kirci and Jonathan Schaeffer and Nathan Sturtevant Department of Computing Science University of Alberta Edmonton, Alberta, Canada {kirci,nathanst,jonathan}@cs.ualberta.ca

More information

Artificial Neural Networks. Artificial Intelligence Santa Clara, 2016

Artificial Neural Networks. Artificial Intelligence Santa Clara, 2016 Artificial Neural Networks Artificial Intelligence Santa Clara, 2016 Simulate the functioning of the brain Can simulate actual neurons: Computational neuroscience Can introduce simplified neurons: Neural

More information

Artificial Intelligence Search III

Artificial Intelligence Search III Artificial Intelligence Search III Lecture 5 Content: Search III Quick Review on Lecture 4 Why Study Games? Game Playing as Search Special Characteristics of Game Playing Search Ingredients of 2-Person

More information

Foundations of Artificial Intelligence

Foundations of Artificial Intelligence Foundations of Artificial Intelligence 6. Board Games Search Strategies for Games, Games with Chance, State of the Art Joschka Boedecker and Wolfram Burgard and Frank Hutter and Bernhard Nebel Albert-Ludwigs-Universität

More information

Creating a Poker Playing Program Using Evolutionary Computation

Creating a Poker Playing Program Using Evolutionary Computation Creating a Poker Playing Program Using Evolutionary Computation Simon Olsen and Rob LeGrand, Ph.D. Abstract Artificial intelligence is a rapidly expanding technology. We are surrounded by technology that

More information

Introduction to Spring 2009 Artificial Intelligence Final Exam

Introduction to Spring 2009 Artificial Intelligence Final Exam CS 188 Introduction to Spring 2009 Artificial Intelligence Final Exam INSTRUCTIONS You have 3 hours. The exam is closed book, closed notes except a two-page crib sheet, double-sided. Please use non-programmable

More information

Bootstrapping from Game Tree Search

Bootstrapping from Game Tree Search Joel Veness David Silver Will Uther Alan Blair University of New South Wales NICTA University of Alberta December 9, 2009 Presentation Overview Introduction Overview Game Tree Search Evaluation Functions

More information

Foundations of Artificial Intelligence

Foundations of Artificial Intelligence Foundations of Artificial Intelligence 6. Board Games Search Strategies for Games, Games with Chance, State of the Art Joschka Boedecker and Wolfram Burgard and Bernhard Nebel Albert-Ludwigs-Universität

More information

Foundations of AI. 6. Board Games. Search Strategies for Games, Games with Chance, State of the Art

Foundations of AI. 6. Board Games. Search Strategies for Games, Games with Chance, State of the Art Foundations of AI 6. Board Games Search Strategies for Games, Games with Chance, State of the Art Wolfram Burgard, Andreas Karwath, Bernhard Nebel, and Martin Riedmiller SA-1 Contents Board Games Minimax

More information

Training a Neural Network for Checkers

Training a Neural Network for Checkers Training a Neural Network for Checkers Daniel Boonzaaier Supervisor: Adiel Ismail June 2017 Thesis presented in fulfilment of the requirements for the degree of Bachelor of Science in Honours at the University

More information

Adversarial Search and Game Playing

Adversarial Search and Game Playing Games Adversarial Search and Game Playing Russell and Norvig, 3 rd edition, Ch. 5 Games: multi-agent environment q What do other agents do and how do they affect our success? q Cooperative vs. competitive

More information

A Quoridor-playing Agent

A Quoridor-playing Agent A Quoridor-playing Agent P.J.C. Mertens June 21, 2006 Abstract This paper deals with the construction of a Quoridor-playing software agent. Because Quoridor is a rather new game, research about the game

More information

Augmenting Self-Learning In Chess Through Expert Imitation

Augmenting Self-Learning In Chess Through Expert Imitation Augmenting Self-Learning In Chess Through Expert Imitation Michael Xie Department of Computer Science Stanford University Stanford, CA 94305 xie@cs.stanford.edu Gene Lewis Department of Computer Science

More information

Board Representations for Neural Go Players Learning by Temporal Difference

Board Representations for Neural Go Players Learning by Temporal Difference Board Representations for Neural Go Players Learning by Temporal Difference Helmut A. Mayer Department of Computer Sciences Scientic Computing Unit University of Salzburg, AUSTRIA helmut@cosy.sbg.ac.at

More information

Experiments on Alternatives to Minimax

Experiments on Alternatives to Minimax Experiments on Alternatives to Minimax Dana Nau University of Maryland Paul Purdom Indiana University April 23, 1993 Chun-Hung Tzeng Ball State University Abstract In the field of Artificial Intelligence,

More information

Monte Carlo Tree Search

Monte Carlo Tree Search Monte Carlo Tree Search 1 By the end, you will know Why we use Monte Carlo Search Trees The pros and cons of MCTS How it is applied to Super Mario Brothers and Alpha Go 2 Outline I. Pre-MCTS Algorithms

More information

Training Neural Networks for Checkers

Training Neural Networks for Checkers Training Neural Networks for Checkers Daniel Boonzaaier Supervisor: Adiel Ismail 2017 Thesis presented in fulfilment of the requirements for the degree of Bachelor of Science in Honours at the University

More information

Decision Making in Multiplayer Environments Application in Backgammon Variants

Decision Making in Multiplayer Environments Application in Backgammon Variants Decision Making in Multiplayer Environments Application in Backgammon Variants PhD Thesis by Nikolaos Papahristou AI researcher Department of Applied Informatics Thessaloniki, Greece Contributions Expert

More information

CPS 570: Artificial Intelligence Two-player, zero-sum, perfect-information Games

CPS 570: Artificial Intelligence Two-player, zero-sum, perfect-information Games CPS 57: Artificial Intelligence Two-player, zero-sum, perfect-information Games Instructor: Vincent Conitzer Game playing Rich tradition of creating game-playing programs in AI Many similarities to search

More information

On Verifying Game Designs and Playing Strategies using Reinforcement Learning

On Verifying Game Designs and Playing Strategies using Reinforcement Learning On Verifying Game Designs and Playing Strategies using Reinforcement Learning Dimitrios Kalles Computer Technology Institute Kolokotroni 3 Patras, Greece +30-61 221834 kalles@cti.gr Panagiotis Kanellopoulos

More information

CS 380: ARTIFICIAL INTELLIGENCE MONTE CARLO SEARCH. Santiago Ontañón

CS 380: ARTIFICIAL INTELLIGENCE MONTE CARLO SEARCH. Santiago Ontañón CS 380: ARTIFICIAL INTELLIGENCE MONTE CARLO SEARCH Santiago Ontañón so367@drexel.edu Recall: Adversarial Search Idea: When there is only one agent in the world, we can solve problems using DFS, BFS, ID,

More information

CS440/ECE448 Lecture 11: Stochastic Games, Stochastic Search, and Learned Evaluation Functions

CS440/ECE448 Lecture 11: Stochastic Games, Stochastic Search, and Learned Evaluation Functions CS440/ECE448 Lecture 11: Stochastic Games, Stochastic Search, and Learned Evaluation Functions Slides by Svetlana Lazebnik, 9/2016 Modified by Mark Hasegawa Johnson, 9/2017 Types of game environments Perfect

More information

April 25, Competing and cooperating with AI. Pantelis P. Analytis. Human behavior in Chess. Competing with AI. Cooperative machines?

April 25, Competing and cooperating with AI. Pantelis P. Analytis. Human behavior in Chess. Competing with AI. Cooperative machines? April 25, 2018 1 / 47 1 2 3 4 5 6 2 / 47 The case of chess 3 / 47 chess The first stage was the orientation phase, in which the subject assessed the situation determined a very general idea of what to

More information

Contents. Foundations of Artificial Intelligence. Problems. Why Board Games?

Contents. Foundations of Artificial Intelligence. Problems. Why Board Games? Contents Foundations of Artificial Intelligence 6. Board Games Search Strategies for Games, Games with Chance, State of the Art Wolfram Burgard, Bernhard Nebel, and Martin Riedmiller Albert-Ludwigs-Universität

More information

Presentation Overview. Bootstrapping from Game Tree Search. Game Tree Search. Heuristic Evaluation Function

Presentation Overview. Bootstrapping from Game Tree Search. Game Tree Search. Heuristic Evaluation Function Presentation Bootstrapping from Joel Veness David Silver Will Uther Alan Blair University of New South Wales NICTA University of Alberta A new algorithm will be presented for learning heuristic evaluation

More information

The Basic Kak Neural Network with Complex Inputs

The Basic Kak Neural Network with Complex Inputs The Basic Kak Neural Network with Complex Inputs Pritam Rajagopal The Kak family of neural networks [3-6,2] is able to learn patterns quickly, and this speed of learning can be a decisive advantage over

More information

Game-playing: DeepBlue and AlphaGo

Game-playing: DeepBlue and AlphaGo Game-playing: DeepBlue and AlphaGo Brief history of gameplaying frontiers 1990s: Othello world champions refuse to play computers 1994: Chinook defeats Checkers world champion 1997: DeepBlue defeats world

More information

Prediction of Missing PMU Measurement using Artificial Neural Network

Prediction of Missing PMU Measurement using Artificial Neural Network Prediction of Missing PMU Measurement using Artificial Neural Network Gaurav Khare, SN Singh, Abheejeet Mohapatra Department of Electrical Engineering Indian Institute of Technology Kanpur Kanpur-208016,

More information

CHAPTER 6 BACK PROPAGATED ARTIFICIAL NEURAL NETWORK TRAINED ARHF

CHAPTER 6 BACK PROPAGATED ARTIFICIAL NEURAL NETWORK TRAINED ARHF 95 CHAPTER 6 BACK PROPAGATED ARTIFICIAL NEURAL NETWORK TRAINED ARHF 6.1 INTRODUCTION An artificial neural network (ANN) is an information processing model that is inspired by biological nervous systems

More information

CS 4700: Foundations of Artificial Intelligence

CS 4700: Foundations of Artificial Intelligence CS 4700: Foundations of Artificial Intelligence Bart Selman Reinforcement Learning R&N Chapter 21 Note: in the next two parts of RL, some of the figure/section numbers refer to an earlier edition of R&N

More information

Foundations of AI. 5. Board Games. Search Strategies for Games, Games with Chance, State of the Art. Wolfram Burgard and Luc De Raedt SA-1

Foundations of AI. 5. Board Games. Search Strategies for Games, Games with Chance, State of the Art. Wolfram Burgard and Luc De Raedt SA-1 Foundations of AI 5. Board Games Search Strategies for Games, Games with Chance, State of the Art Wolfram Burgard and Luc De Raedt SA-1 Contents Board Games Minimax Search Alpha-Beta Search Games with

More information

Contents. MA 327/ECO 327 Introduction to Game Theory Fall 2017 Notes. 1 Wednesday, August Friday, August Monday, August 28 6

Contents. MA 327/ECO 327 Introduction to Game Theory Fall 2017 Notes. 1 Wednesday, August Friday, August Monday, August 28 6 MA 327/ECO 327 Introduction to Game Theory Fall 2017 Notes Contents 1 Wednesday, August 23 4 2 Friday, August 25 5 3 Monday, August 28 6 4 Wednesday, August 30 8 5 Friday, September 1 9 6 Wednesday, September

More information

Artificial Intelligence

Artificial Intelligence Artificial Intelligence CS482, CS682, MW 1 2:15, SEM 201, MS 227 Prerequisites: 302, 365 Instructor: Sushil Louis, sushil@cse.unr.edu, http://www.cse.unr.edu/~sushil Games and game trees Multi-agent systems

More information

An intelligent Othello player combining machine learning and game specific heuristics

An intelligent Othello player combining machine learning and game specific heuristics Louisiana State University LSU Digital Commons LSU Master's Theses Graduate School 2011 An intelligent Othello player combining machine learning and game specific heuristics Kevin Anthony Cherry Louisiana

More information

ADVERSARIAL SEARCH. Today. Reading. Goals. AIMA Chapter Read , Skim 5.7

ADVERSARIAL SEARCH. Today. Reading. Goals. AIMA Chapter Read , Skim 5.7 ADVERSARIAL SEARCH Today Reading AIMA Chapter Read 5.1-5.5, Skim 5.7 Goals Introduce adversarial games Minimax as an optimal strategy Alpha-beta pruning 1 Adversarial Games People like games! Games are

More information

Adversary Search. Ref: Chapter 5

Adversary Search. Ref: Chapter 5 Adversary Search Ref: Chapter 5 1 Games & A.I. Easy to measure success Easy to represent states Small number of operators Comparison against humans is possible. Many games can be modeled very easily, although

More information

Humanization of Computational Learning in Strategy Games

Humanization of Computational Learning in Strategy Games 1 Humanization of Computational Learning in Strategy Games By Benjamin S. Greenberg S.B., C.S. M.I.T., 2015 Submitted to the Department of Electrical Engineering and Computer Science in Partial Fulfillment

More information

CS 1571 Introduction to AI Lecture 12. Adversarial search. CS 1571 Intro to AI. Announcements

CS 1571 Introduction to AI Lecture 12. Adversarial search. CS 1571 Intro to AI. Announcements CS 171 Introduction to AI Lecture 1 Adversarial search Milos Hauskrecht milos@cs.pitt.edu 39 Sennott Square Announcements Homework assignment is out Programming and experiments Simulated annealing + Genetic

More information

Temporal Difference Learning for the Game Tic-Tac-Toe 3D: Applying Structure to Neural Networks

Temporal Difference Learning for the Game Tic-Tac-Toe 3D: Applying Structure to Neural Networks 2015 IEEE Symposium Series on Computational Intelligence Temporal Difference Learning for the Game Tic-Tac-Toe 3D: Applying Structure to Neural Networks Michiel van de Steeg Institute of Artificial Intelligence

More information

Opponent Modelling In World Of Warcraft

Opponent Modelling In World Of Warcraft Opponent Modelling In World Of Warcraft A.J.J. Valkenberg 19th June 2007 Abstract In tactical commercial games, knowledge of an opponent s location is advantageous when designing a tactic. This paper proposes

More information

CS 229 Final Project: Using Reinforcement Learning to Play Othello

CS 229 Final Project: Using Reinforcement Learning to Play Othello CS 229 Final Project: Using Reinforcement Learning to Play Othello Kevin Fry Frank Zheng Xianming Li ID: kfry ID: fzheng ID: xmli 16 December 2016 Abstract We built an AI that learned to play Othello.

More information

Creating a Dominion AI Using Genetic Algorithms

Creating a Dominion AI Using Genetic Algorithms Creating a Dominion AI Using Genetic Algorithms Abstract Mok Ming Foong Dominion is a deck-building card game. It allows for complex strategies, has an aspect of randomness in card drawing, and no obvious

More information

CS 2710 Foundations of AI. Lecture 9. Adversarial search. CS 2710 Foundations of AI. Game search

CS 2710 Foundations of AI. Lecture 9. Adversarial search. CS 2710 Foundations of AI. Game search CS 2710 Foundations of AI Lecture 9 Adversarial search Milos Hauskrecht milos@cs.pitt.edu 5329 Sennott Square CS 2710 Foundations of AI Game search Game-playing programs developed by AI researchers since

More information

Monte Carlo tree search techniques in the game of Kriegspiel

Monte Carlo tree search techniques in the game of Kriegspiel Monte Carlo tree search techniques in the game of Kriegspiel Paolo Ciancarini and Gian Piero Favini University of Bologna, Italy 22 IJCAI, Pasadena, July 2009 Agenda Kriegspiel as a partial information

More information

A Reinforcement Learning Algorithm Applied to Simplified Two-Player Texas Hold em Poker

A Reinforcement Learning Algorithm Applied to Simplified Two-Player Texas Hold em Poker A Reinforcement Learning Algorithm Applied to Simplified Two-Player Texas Hold em Poker Fredrik A. Dahl Norwegian Defence Research Establishment (FFI) P.O. Box 25, NO-2027 Kjeller, Norway Fredrik-A.Dahl@ffi.no

More information

Automated Suicide: An Antichess Engine

Automated Suicide: An Antichess Engine Automated Suicide: An Antichess Engine Jim Andress and Prasanna Ramakrishnan 1 Introduction Antichess (also known as Suicide Chess or Loser s Chess) is a popular variant of chess where the objective of

More information

Machine Learning Othello Project

Machine Learning Othello Project Machine Learning Othello Project Tom Barry The assignment. We have been provided with a genetic programming framework written in Java and an intelligent Othello player( EDGAR ) as well a random player.

More information

CS 331: Artificial Intelligence Adversarial Search II. Outline

CS 331: Artificial Intelligence Adversarial Search II. Outline CS 331: Artificial Intelligence Adversarial Search II 1 Outline 1. Evaluation Functions 2. State-of-the-art game playing programs 3. 2 player zero-sum finite stochastic games of perfect information 2 1

More information

Ar#ficial)Intelligence!!

Ar#ficial)Intelligence!! Introduc*on! Ar#ficial)Intelligence!! Roman Barták Department of Theoretical Computer Science and Mathematical Logic So far we assumed a single-agent environment, but what if there are more agents and

More information

MITOCW Project: Backgammon tutor MIT Multicore Programming Primer, IAP 2007

MITOCW Project: Backgammon tutor MIT Multicore Programming Primer, IAP 2007 MITOCW Project: Backgammon tutor MIT 6.189 Multicore Programming Primer, IAP 2007 The following content is provided under a Creative Commons license. Your support will help MIT OpenCourseWare continue

More information

Artificial Intelligence. Minimax and alpha-beta pruning

Artificial Intelligence. Minimax and alpha-beta pruning Artificial Intelligence Minimax and alpha-beta pruning In which we examine the problems that arise when we try to plan ahead to get the best result in a world that includes a hostile agent (other agent

More information

IMPROVEMENTS TO A QUEUE AND DELAY ESTIMATION ALGORITHM UTILIZED IN VIDEO IMAGING VEHICLE DETECTION SYSTEMS

IMPROVEMENTS TO A QUEUE AND DELAY ESTIMATION ALGORITHM UTILIZED IN VIDEO IMAGING VEHICLE DETECTION SYSTEMS IMPROVEMENTS TO A QUEUE AND DELAY ESTIMATION ALGORITHM UTILIZED IN VIDEO IMAGING VEHICLE DETECTION SYSTEMS A Thesis Proposal By Marshall T. Cheek Submitted to the Office of Graduate Studies Texas A&M University

More information

MyPawns OppPawns MyKings OppKings MyThreatened OppThreatened MyWins OppWins Draws

MyPawns OppPawns MyKings OppKings MyThreatened OppThreatened MyWins OppWins Draws The Role of Opponent Skill Level in Automated Game Learning Ying Ge and Michael Hash Advisor: Dr. Mark Burge Armstrong Atlantic State University Savannah, Geogia USA 31419-1997 geying@drake.armstrong.edu

More information

B1 Problem Statement Unit Pricing

B1 Problem Statement Unit Pricing B1 Problem Statement Unit Pricing Determine the best buy (the lowest per unit cost) between two items. The inputs will be the weight in ounces and the cost in dollars. Display whether the first or the

More information

CMSC 671 Project Report- Google AI Challenge: Planet Wars

CMSC 671 Project Report- Google AI Challenge: Planet Wars 1. Introduction Purpose The purpose of the project is to apply relevant AI techniques learned during the course with a view to develop an intelligent game playing bot for the game of Planet Wars. Planet

More information

Hybrid of Evolution and Reinforcement Learning for Othello Players

Hybrid of Evolution and Reinforcement Learning for Othello Players Hybrid of Evolution and Reinforcement Learning for Othello Players Kyung-Joong Kim, Heejin Choi and Sung-Bae Cho Dept. of Computer Science, Yonsei University 134 Shinchon-dong, Sudaemoon-ku, Seoul 12-749,

More information

CSCI 4150 Introduction to Artificial Intelligence, Fall 2004 Assignment 7 (135 points), out Monday November 22, due Thursday December 9

CSCI 4150 Introduction to Artificial Intelligence, Fall 2004 Assignment 7 (135 points), out Monday November 22, due Thursday December 9 CSCI 4150 Introduction to Artificial Intelligence, Fall 2004 Assignment 7 (135 points), out Monday November 22, due Thursday December 9 Learning to play blackjack In this assignment, you will implement

More information

a b c d e f g h i j k l m n

a b c d e f g h i j k l m n Shoebox, page 1 In his book Chess Variants & Games, A. V. Murali suggests playing chess on the exterior surface of a cube. This playing surface has intriguing properties: We can think of it as three interlocked

More information

Game Tree Search. Generalizing Search Problems. Two-person Zero-Sum Games. Generalizing Search Problems. CSC384: Intro to Artificial Intelligence

Game Tree Search. Generalizing Search Problems. Two-person Zero-Sum Games. Generalizing Search Problems. CSC384: Intro to Artificial Intelligence CSC384: Intro to Artificial Intelligence Game Tree Search Chapter 6.1, 6.2, 6.3, 6.6 cover some of the material we cover here. Section 6.6 has an interesting overview of State-of-the-Art game playing programs.

More information

Othello/Reversi using Game Theory techniques Parth Parekh Urjit Singh Bhatia Kushal Sukthankar

Othello/Reversi using Game Theory techniques Parth Parekh Urjit Singh Bhatia Kushal Sukthankar Othello/Reversi using Game Theory techniques Parth Parekh Urjit Singh Bhatia Kushal Sukthankar Othello Rules Two Players (Black and White) 8x8 board Black plays first Every move should Flip over at least

More information

Co-Evolving Checkers Playing Programs using only Win, Lose, or Draw

Co-Evolving Checkers Playing Programs using only Win, Lose, or Draw Co-Evolving Checkers Playing Programs using only Win, Lose, or Draw Kumar Chellapilla a and David B Fogel b* a University of California at San Diego, Dept Elect Comp Eng, La Jolla, CA, 92093 b Natural

More information

Outline. Introduction to AI. Artificial Intelligence. What is an AI? What is an AI? Agents Environments

Outline. Introduction to AI. Artificial Intelligence. What is an AI? What is an AI? Agents Environments Outline Introduction to AI ECE457 Applied Artificial Intelligence Fall 2007 Lecture #1 What is an AI? Russell & Norvig, chapter 1 Agents s Russell & Norvig, chapter 2 ECE457 Applied Artificial Intelligence

More information

Reinforcement Learning of Local Shape in the Game of Go

Reinforcement Learning of Local Shape in the Game of Go Reinforcement Learning of Local Shape in the Game of Go David Silver, Richard Sutton, and Martin Müller Department of Computing Science University of Alberta Edmonton, Canada T6G 2E8 {silver, sutton, mmueller}@cs.ualberta.ca

More information

Adjustable Group Behavior of Agents in Action-based Games

Adjustable Group Behavior of Agents in Action-based Games Adjustable Group Behavior of Agents in Action-d Games Westphal, Keith and Mclaughlan, Brian Kwestp2@uafortsmith.edu, brian.mclaughlan@uafs.edu Department of Computer and Information Sciences University

More information

Universiteit Leiden Opleiding Informatica

Universiteit Leiden Opleiding Informatica Universiteit Leiden Opleiding Informatica Predicting the Outcome of the Game Othello Name: Simone Cammel Date: August 31, 2015 1st supervisor: 2nd supervisor: Walter Kosters Jeannette de Graaf BACHELOR

More information

FreeCiv Learner: A Machine Learning Project Utilizing Genetic Algorithms

FreeCiv Learner: A Machine Learning Project Utilizing Genetic Algorithms FreeCiv Learner: A Machine Learning Project Utilizing Genetic Algorithms Felix Arnold, Bryan Horvat, Albert Sacks Department of Computer Science Georgia Institute of Technology Atlanta, GA 30318 farnold3@gatech.edu

More information

Game Playing for a Variant of Mancala Board Game (Pallanguzhi)

Game Playing for a Variant of Mancala Board Game (Pallanguzhi) Game Playing for a Variant of Mancala Board Game (Pallanguzhi) Varsha Sankar (SUNet ID: svarsha) 1. INTRODUCTION Game playing is a very interesting area in the field of Artificial Intelligence presently.

More information

Game Playing. Philipp Koehn. 29 September 2015

Game Playing. Philipp Koehn. 29 September 2015 Game Playing Philipp Koehn 29 September 2015 Outline 1 Games Perfect play minimax decisions α β pruning Resource limits and approximate evaluation Games of chance Games of imperfect information 2 games

More information

Games and Adversarial Search

Games and Adversarial Search 1 Games and Adversarial Search BBM 405 Fundamentals of Artificial Intelligence Pinar Duygulu Hacettepe University Slides are mostly adapted from AIMA, MIT Open Courseware and Svetlana Lazebnik (UIUC) Spring

More information

Documentation and Discussion

Documentation and Discussion 1 of 9 11/7/2007 1:21 AM ASSIGNMENT 2 SUBJECT CODE: CS 6300 SUBJECT: ARTIFICIAL INTELLIGENCE LEENA KORA EMAIL:leenak@cs.utah.edu Unid: u0527667 TEEKO GAME IMPLEMENTATION Documentation and Discussion 1.

More information

Learning to play Dominoes

Learning to play Dominoes Learning to play Dominoes Ivan de Jesus P. Pinto 1, Mateus R. Pereira 1, Luciano Reis Coutinho 1 1 Departamento de Informática Universidade Federal do Maranhão São Luís,MA Brazil navi1921@gmail.com, mateus.rp.slz@gmail.com,

More information

AN IMPROVED NEURAL NETWORK-BASED DECODER SCHEME FOR SYSTEMATIC CONVOLUTIONAL CODE. A Thesis by. Andrew J. Zerngast

AN IMPROVED NEURAL NETWORK-BASED DECODER SCHEME FOR SYSTEMATIC CONVOLUTIONAL CODE. A Thesis by. Andrew J. Zerngast AN IMPROVED NEURAL NETWORK-BASED DECODER SCHEME FOR SYSTEMATIC CONVOLUTIONAL CODE A Thesis by Andrew J. Zerngast Bachelor of Science, Wichita State University, 2008 Submitted to the Department of Electrical

More information

Player Profiling in Texas Holdem

Player Profiling in Texas Holdem Player Profiling in Texas Holdem Karl S. Brandt CMPS 24, Spring 24 kbrandt@cs.ucsc.edu 1 Introduction Poker is a challenging game to play by computer. Unlike many games that have traditionally caught the

More information

Artificial Intelligence 1: game playing

Artificial Intelligence 1: game playing Artificial Intelligence 1: game playing Lecturer: Tom Lenaerts Institut de Recherches Interdisciplinaires et de Développements en Intelligence Artificielle (IRIDIA) Université Libre de Bruxelles Outline

More information

Contents. List of Figures

Contents. List of Figures 1 Contents 1 Introduction....................................... 3 1.1 Rules of the game............................... 3 1.2 Complexity of the game............................ 4 1.3 History of self-learning

More information

CS 188: Artificial Intelligence Spring Game Playing in Practice

CS 188: Artificial Intelligence Spring Game Playing in Practice CS 188: Artificial Intelligence Spring 2006 Lecture 23: Games 4/18/2006 Dan Klein UC Berkeley Game Playing in Practice Checkers: Chinook ended 40-year-reign of human world champion Marion Tinsley in 1994.

More information

Multiple Agents. Why can t we all just get along? (Rodney King)

Multiple Agents. Why can t we all just get along? (Rodney King) Multiple Agents Why can t we all just get along? (Rodney King) Nash Equilibriums........................................ 25 Multiple Nash Equilibriums................................. 26 Prisoners Dilemma.......................................

More information

TUD Poker Challenge Reinforcement Learning with Imperfect Information

TUD Poker Challenge Reinforcement Learning with Imperfect Information TUD Poker Challenge 2008 Reinforcement Learning with Imperfect Information Outline Reinforcement Learning Perfect Information Imperfect Information Lagging Anchor Algorithm Matrix Form Extensive Form Poker

More information

Game Tree Search. CSC384: Introduction to Artificial Intelligence. Generalizing Search Problem. General Games. What makes something a game?

Game Tree Search. CSC384: Introduction to Artificial Intelligence. Generalizing Search Problem. General Games. What makes something a game? CSC384: Introduction to Artificial Intelligence Generalizing Search Problem Game Tree Search Chapter 5.1, 5.2, 5.3, 5.6 cover some of the material we cover here. Section 5.6 has an interesting overview

More information

Extending the STRADA Framework to Design an AI for ORTS

Extending the STRADA Framework to Design an AI for ORTS Extending the STRADA Framework to Design an AI for ORTS Laurent Navarro and Vincent Corruble Laboratoire d Informatique de Paris 6 Université Pierre et Marie Curie (Paris 6) CNRS 4, Place Jussieu 75252

More information

A Machine-Learning Approach to Computer Go

A Machine-Learning Approach to Computer Go A Machine-Learning Approach to Computer Go Jeffrey Bagdis Advisor: Prof. Andrew Appel May 8, 2007 1 Introduction Go is an ancient board game dating back over 3000 years. Although the rules of the game

More information

DeepStack: Expert-Level AI in Heads-Up No-Limit Poker. Surya Prakash Chembrolu

DeepStack: Expert-Level AI in Heads-Up No-Limit Poker. Surya Prakash Chembrolu DeepStack: Expert-Level AI in Heads-Up No-Limit Poker Surya Prakash Chembrolu AI and Games AlphaGo Go Watson Jeopardy! DeepBlue -Chess Chinook -Checkers TD-Gammon -Backgammon Perfect Information Games

More information

Learning to Play like an Othello Master CS 229 Project Report. Shir Aharon, Amanda Chang, Kent Koyanagi

Learning to Play like an Othello Master CS 229 Project Report. Shir Aharon, Amanda Chang, Kent Koyanagi Learning to Play like an Othello Master CS 229 Project Report December 13, 213 1 Abstract This project aims to train a machine to strategically play the game of Othello using machine learning. Prior to

More information

Algorithms for Data Structures: Search for Games. Phillip Smith 27/11/13

Algorithms for Data Structures: Search for Games. Phillip Smith 27/11/13 Algorithms for Data Structures: Search for Games Phillip Smith 27/11/13 Search for Games Following this lecture you should be able to: Understand the search process in games How an AI decides on the best

More information