Intelligent Non-Player Character with Deep Learning. Intelligent Non-Player Character with Deep Learning 1

Transcription

1 Intelligent Non-Player Character with Deep Learning Meng Zhixiang, Zhang Haoze Supervised by Prof. Michael Lyu CUHK CSE FYP Term 1 Intelligent Non-Player Character with Deep Learning 1

2 Intelligent Non-Player Character with Deep Learning 2

3 Background We all know the results Intelligent Non-Player Character with Deep Learning 3

4 Agenda o Background o Motivation & Objective o Methodologies o Design & Implementation o Results & Discussion o Conclusion Intelligent Non-Player Character with Deep Learning 4

5 Agenda o Background o Development of AI in Go, Chess and Chinese Chess o Difference among Go, Chess and Chinese Chess o Motivation & Objective o Methodologies o Design & Implementation o Results & Discussion o Conclusion Intelligent Non-Player Character with Deep Learning 5

6 Development of AI in Go No Good Results Zen beat Takemiya Masaki at five stones handicap Mar 2012 AlphaGo beat Lee Sedol Mar 2016 Minimax Searching Pruning Monte Carlo Deep Learning Intelligent Non-Player Character with Deep Learning 6

7 Difference between Go and Chess/Chinese Chess Intelligent Non-Player Character with Deep Learning 7

8 Development of AI in Chess Deep Blue beat Garry Kasparov May 1997 Stockfish won TCEC 2013, 2014, 2015 Giraffe plays at the level of an FIDE International Master on a PC Sep 2015 Minimax Searching Evaluation Function Hand-Coded Knowledge Minimax Searching Evaluation Function Hand-Coded Knowledge Deep Reinforcement Learning TCEC: Top Chess Engine Championship FIDE: World Chess Federation Intelligent Non-Player Character with Deep Learning 8

9 Difference between Chess and Chinese Chess Intelligent Non-Player Character with Deep Learning 9

10 Development of AI in Chinese Chess Tiansuo Inspur System beat five Grandmaster players Aug 2006 Chess Nade beat three Master players Nov 2009 Now??? Minimax Searching Alpha-Beta Pruning Hand-Coded Knowledge Minimax Searching Alpha-Beta Pruning Hand-Coded Knowledge Deep Learning??? Intelligent Non-Player Character with Deep Learning 10

11 Motivation Intelligent Non-Player Character with Deep Learning 11

12 Objective Server Human Player User Interface Game AI Intelligent Non-Player Character with Deep Learning 12

13 Agenda o Background o Motivation & Objective o Methodologies o Supervised Learning o Convolutional Neural Network o Design & Implementation o Results & Discussion o Conclusion Intelligent Non-Player Character with Deep Learning 13

14 Supervised Learning o Supervised Learning o the right answer is given o Regression Problem & Classification Problem o Unsupervised Learning o no right answer is given o Clustering Problem Intelligent Non-Player Character with Deep Learning 14

15 Neural Network o Non-linear Hypotheses o Neurons and Brain o Backpropagation Intelligent Non-Player Character with Deep Learning 15

16 Convolutional Neural Network o Feed-forward o Organization of Animal Visual Cortex o Image Recognition Local Receptive Fields Shared Weights and Biases Intelligent Non-Player Character with Deep Learning 16

17 Agenda o Background o Motivation & Objective o Methodologies o Design & Implementation o Project Workflow o Results & Discussion o Conclusion Intelligent Non-Player Character with Deep Learning 17

18 Project Workflow Accuracy Testing Model Design Model Building Model Training Model Testing Real Performance Testing Intelligent Non-Player Character with Deep Learning 18

19 Design Overview Game AI Policy Network Evaluation Network Predict probabilities of next moves Evaluate winning rate Intelligent Non-Player Character with Deep Learning 19

20 Game AI Structure Piece Selector Message Receiver Format Converter Feature Extractor Decision Maker Message Sender Move Selector Intelligent Non-Player Character with Deep Learning 20

21 Feature Channels Feature Channel 1 Feature Channel 2 Feature Channel 3 Feature Channel 4 Feature Channel 5 Feature Channel 6 Feature Channel 7 Feature Channel 8 Feature Channel 9 (only for Move Selector) Pieces belonging to different sides Pieces of Advisor type Pieces of Bishop type Pieces of Cannon type Pieces of King type Pieces of Knight type Pieces of Pawn type Pieces of Rock type Valid moves for the selected piece Intelligent Non-Player Character with Deep Learning 21

22 Feature Channels Chessboard Status 1 st Feature Channel 4 th Feature Channel 9 th Feature Channel Intelligent Non-Player Character with Deep Learning 22

23 Piece Selector & Move Selector Intelligent Non-Player Character with Deep Learning 23

24 Piece Selector & Move Selector Extracted Features First Hidden Convolutional Layer Second Hidden Convolutional Layer Third Hidden Layer (Softmax Layer) Probability Distribution Rectified Linear Unit (ReLU) Intelligent Non-Player Character with Deep Learning 24

25 Selection Strategy o Strategy 1: o Select the piece with highest possibility given by Piece Selector o Select the destination of that piece with highest possibility given by Move Selector o Strategy 2: o Calculate the probability of moving a piece * the probability of a destination of that piece o Select the combination with highest probability Intelligent Non-Player Character with Deep Learning 25

26 Project Workflow Accuracy Testing Model Design Model Building Model Training Model Testing Real Performance Testing Intelligent Non-Player Character with Deep Learning 26

27 TensorFlow o an open source software library o for numerical computation o using data flow graphs o flexibility and portability Intelligent Non-Player Character with Deep Learning 27

28 Project Workflow Accuracy Testing Model Design Model Building Model Training Model Testing Real Performance Testing Intelligent Non-Player Character with Deep Learning 28

29 Training Dataset Collected Game Records Features and Targets Training Dataset for Different NN models Intelligent Non-Player Character with Deep Learning 29

30 FEN Format rnbakab1r/ /1c1111nc1/p1p1p1p1p/ / / P1P1P1P1P/1C11C1111/ /RNBAKABNR, r Intelligent Non-Player Character with Deep Learning 30

31 Format Conversion 炮二平五 马二进三 车一进一 车一平六 车六进七 车九进一 炮八进五 炮五进四 车九平六 前车进一 车六平四 车四进六 炮八平五 炮８平５ 马８进７ 车９平８ 车８进６ 马２进１ 炮２进７ 马７退８ 士６进５ 将５平６ 士５退４ 炮５平６ 将６平５ Intelligent Non-Player Character with Deep Learning 31

32 Training Strategy o Piece Selector and Move Selector are trained separately o Shuffle the training dataset containing over 1,600,000 moves o Train the models batch by batch o Test the accuracy along the process o An untrained testing dataset containing over 80,000 moves Intelligent Non-Player Character with Deep Learning 32

33 Project Workflow Accuracy Testing Model Design Model Building Model Training Model Testing Real Performance Testing Intelligent Non-Player Character with Deep Learning 33

34 Results Piece Selector Accuracy accuracy = # of correct predictions / total # of test cases prediction: the choice with the highest probability Intelligent Non-Player Character with Deep Learning 34

35 Results Move Selector Accuracy Intelligent Non-Player Character with Deep Learning 35

36 Results Move Selector Accuracy Advisor 89.8% Bishop 91.2% Cannon 54.1% King 79.8% Knight 70.1% Pawn 90.4% Rock 53.6% Move Selector Accuracy Intelligent Non-Player Character with Deep Learning 36

37 Results Intelligent Non-Player Character with Deep Learning 37

38 Results Intelligent Non-Player Character with Deep Learning 38

39 Results Intelligent Non-Player Character with Deep Learning 39

40 Results Selection Strategy 1 Selection Strategy 2 Intelligent Non-Player Character with Deep Learning 40

41 Discussion o Possible Reasons: o CNN not deep enough o Training dataset not large enough o Records in training dataset may not be the optimal choices o For one chessboard status, there may be different move choices in training dataset o It s hard to judge which choice is better in current phase Intelligent Non-Player Character with Deep Learning 41

42 Conclusion o Achieved overall high accuracy o Performed badly in some cases o Need further improvement o Reinforcement Learning o Not limited by training dataset o Evaluation Network o To judge which move is better Intelligent Non-Player Character with Deep Learning 42

43 Q&A Intelligent Non-Player Character with Deep Learning 43