Integrating Learning in a Multi-Scale Agent

Transcription

1 Integrating Learning in a Multi-Scale Agent Ben Weber Dissertation Defense May 18, 2012

2 Introduction AI has a long history of using games to advance the state of the field [Shannon 1950]

3 Real-Time Strategy Games Building human-level AI for RTS games remains an open research challenge StarCraft II, Blizzard Entertainment

4 Task Environment Properties Chess StarCraft Taxi Driving Fully vs. partially observable Deterministic vs. stochastic Episodic vs. sequential Fully Partially Partially Deterministic Deterministic* Stochastic Sequential Sequential Sequential Static vs. dynamic Static Dynamic Dynamic Discrete vs. continuous Discrete Continuous Continuous Single vs. multiagent Multi Multi Multi [Russell & Norvig 2009]

5 Motivation RTS games present complex environments and complex tasks Professional players demonstrate a broad range of reasoning capabilities Human behavior can be observed, emulated, and evaluated [Langley 2011, Mateas 2002]

6 Hypothesis Reproducing expert-level StarCraft gameplay involves integrating heterogeneous reasoning capabilities

7 Research Questions What competencies are necessary for expert StarCraft gameplay? Which competencies can be learned from demonstrations? How can these competencies be integrated in a real-time agent?

8 Overview StarCraft Multi-Scale AI Learning from Demonstration Integrating Learning Evaluation

9 StarCraft Expert gameplay 300+ APM Evolving meta-game Exhibited capabilities Estimation Anticipation Adaptation [Flash, Pro-gamer]

10 StarCraft Gameplay Expand Tech Tree Attack Opponent Manage Economy Produce Units

11 Gameplay Scales in StarCraft Individual Worker harassment Squad Global Aggressive mine placement Support siege line

12 State Space The following number of states are possible, considering only unit type and location: (Type * X * Y) Units States on a 256x256 tile map: (100*256*256) 1700 > 10 11,500

13 Decision Complexity The set of possible actions that can be executed at a particular moment: O(2 W (A * P) + 2 T (D + S) + B(R + C)) W number of workers A number of the type of worker assignments P average number of workspaces T number of troops D number of movement directions [Aha et al. 2005]

14 Decision Complexity The set of possible actions that can be executed at a particular moment: Assumption O(W * A * P + T * D * S + B(R + C)) Unit actions can be selected independently Resulting complexity: Assuming 50 worker units on a 256x256 tile map results in more than 1,000,000 possible actions

15 StarCraft Complex gameplay Real-world properties Highly-competitive Sources of expert gameplay

16 Research Question #1 What competencies are necessary for expert StarCraft gameplay?

17 Multi-Scale AI Multiple scales Actions are performed across multiple levels of coordination Interrelated tasks Performance in each tasks impacts other tasks Real-time Actions are performed in real time

18 Reactive Planning Provides useful mechanisms for building multi-scale agents Advantages Efficient behavior selection Interleaved plan expansion and execution Disadvantages Lacks deliberative capabilities [Loyall 1997, Mateas 2002]

19 Agent Design Implemented in the ABL reactive planning language Architecture Extension of McCoy & Mateas integrated agent framework Partitions gameplay into distinct competencies Uses a blackboard for coordination [McCoy & Mateas 2008]

20 EISBot Managers Strategy Manager Income Manager Production Manager Tactics Manager Recon Manager Gather Resources Construct Buildings Attack Opponent Scout Opponent

21 Multi-Scale Idioms Design patterns for authoring multi-scale AI Idioms Message passing Daemon behaviors Managers Unit subtasks Behavior locking

22 Idioms in EISBot Initial_tree Tactics Manager Strategy Manager Income Manager Form Squad Attack Enemy Pump Probes Timing Attack WME Probe Stop WME Squad Monitor Legend Subgoal Daemon behavior Squad Attack Squad Retreat Dragoon Dance Message passing

23 Multi-Scale AI StarCraft gameplay is multi-scale Reactive planning provides mechanisms for multi-scale reasoning Idioms are applied in EISBot to support StarCraft gameplay

24 Research Question #2 Which competencies can be learned from demonstrations?

25 Learning from Demonstration Objective Emulate capabilities exhibited by expert players by harnessing gameplay demonstrations Methods Classification and regression model training Case-based goal formulation Parameter selection for model optimization

26 Strategy Prediction Tasks Identify opponent build orders Predict when buildings will be constructed 400 Spawning Pool Timing Game Time (minutes) [Hsieh & Sun 2008]

27 Approach Feature encoding Each player s actions are encoded in a single vector Vectors are labeled using a build-order rule set Features describe the game cycle when a unit or building type is first produced by a player t, time when x is first produced by P f(x) = { 0, x was not (yet) produced by P

28 Recall Precision Strategy Prediction Results 1 NNge Boosting Rule Set State Lattice Game Time (minutes)

29 Strategy Learning Task Learn build-orders from demonstration Trace Algorithm Converts replays to a trace representation Formulates goals based on most similar situation q = argmin c ϵ L distance(s, c) g = s + (q - q) [Ontañón et al. 2010]

30 Trace Retrieval: Example Consider a planning window of size 2 S =< 3, 0, 1, 1 > T 1 =< 2, 0, 0.5, 1 > T 2 =< 3, 0, 0.7, 1 > T 3 =< 4, 1, 0.9, 1 > T 4 =< 4, 1, 1.1, 2 >

31 Trace Retrieval: Step 1 The system retrieves the most similar case, q S =< 3, 0, 1, 1 > T 1 =< 2, 0, 0.5, 1 > T 2 =< 3, 0, 0.7, 1 > T 3 =< 4, 1, 0.9, 1 > T 4 =< 4, 1, 1.1, 2 >

32 Trace Retrieval : Step 2 q is retrieved S =< 3, 0, 1, 1 > T 1 =< 2, 0, 0.5, 1 > T 2 =< 3, 0, 0.7, 1 > T 3 =< 4, 1, 0.9, 1 > T 4 =< 4, 1, 1.1, 2 >

33 Trace Retrieval : Step 3 The difference is computed: T 4 T 2 = <1,1,0.4,1> S =< 3, 0, 1, 1 > T 1 =< 2, 0, 0.5, 1 > T 2 =< 3, 0, 0.7, 1 > T 3 =< 4, 1, 0.9, 1 > T 4 =< 4, 1, 1.1, 2 >

34 Trace Retrieval : Step 4 g is computed: S =< 3, 0, 1, 1 > T 1 =< 2, 0, 0.5, 1 > T 2 =< 3, 0, 0.7, 1 > T 3 =< 4, 1, 0.9, 1 > T 4 =< 4, 1, 1.1, 2 > g = s + (T 4 T 2 ) = <4, 1, 1.4, 2>

35 Prediction Error (RMSE) Strategy Learning Results 14 Opponent modeling with a window size of Null IB1 Trace MultiTrace Actions performed by player

36 State Estimation Task Estimate enemy positions given prior observations Particle Model Apply movement model Remove visible particles Reweight particles [Thrun 2002, Bererton 2004]

37 Parameter Selection Free parameters Trajectory weights Decay rates State estimation is represented as an optimization problem Input: parameter weights Output: particle model error Replays are used to implement a particle model error function

38 Threat Prediction Error State Estimation Results Game Time (Minutes) Null Model Perfect Tracker Default Model Optimized Model

39 Learning from Demonstration Anticipation Classification and regression models Adaptation Case-based goal formulation Estimation Model optimization

40 Research Question #3 How can these competencies be integrated in a real-time agent?

41 Agent Architecture

42 Integration Approaches Augmenting working memory External Components External plan generation External goal formulation Working Memory

43 Augmenting Working Memory Supplementing working memory with additional beliefs

44 External Plan Generation Generating plans outside the scope of ABL

45 External Goal Formulation Formulating goals outside the scope of ABL

46 Goal-Driven Autonomy A framework for building self introspective agents GDA agents monitor plan execution, detect discrepancies, and explain failures Implementations Hand-authored rules Case-based reasoning [Molineaux et al. 2010, Muñoz-Avila et al. 2010]

47 GDA Subtasks Expectation generation Discrepancy detection Explanation generation Goal formulation

48 Implementation

49 Integrating Learning ABL agents can be interfaced with external learning components Applying the GDA model enabled tighter coordination across capabilities EISBot incorporates ABL behaviors, a particle model, and a GDA implementation

50 Evaluation Claim Reproducing expert-level StarCraft gameplay involves integrating heterogeneous reasoning capabilities Experiments Ablation studies User study

51 GDA Ablation Study Agent configurations Base Formulator Predictor GDA Free parameters Planning window size Look-ahead window size Discrepancy period Discrepancies Explanations Goals Discrepancy Detector Explanation Generator Goal Formulator Goal Manager

52 GDA Results Overall results from the GDA experiments Agent Win Ratio Base 0.73 Formulator 0.77 Predictor 0.81 GDA 0.92

53 User Study Experiment setup Matches hosted on ICCup 3 trials Testing script 1. Launch StarCraft 2. Connect to server 3. Host match 4. Announce experiment [Dennis Fong, Pro-gamer]

54 ICCup Score Performance on Tau Cross Base Formulator Predictor GDA Number of Games Played

55 ICCup Results Agent Longinus Python Tau Cross Overall Base Formulator Predictor GDA

56 EISBot Ranking Rankings achieved by the complete GDA agent Trial Longinus Python Tau Cross Average Percentile Ranking 32 nd 8 th 66 th 48 th

57 Evaluation Ablation Studies Optimized particle model Complete GDA model Integrating additional capabilities into EISBot improved performance EISBot performed at the level of a competitive amateur StarCraft player

58 Conclusion Objective Identify and realize capabilities necessary for expert-level StarCraft gameplay in an agent Approach Decompose gameplay Learn capabilities from demonstrations Integrate learned gameplay models Evaluate versus humans and agents

59 Contributions Idioms for authoring multi-scale agents Methods for learning from demonstration Integration approaches for ABL agents

60 Integrating Learning in a Multi-Scale Agent Ben G. Weber Ph.D. Candidate Expressive Intelligence Studio bweber@soe.ucsc.edu Funding NSF Grant IIS

61 References Aha, Molineaux, & Ponsen Learning to Win: Case-Based Plan Selection in a Real-Time Strategy Game, Proceedings of ICCBR. Bererton State Estimation for Game AI using Particle Filters, Proceedings of AAI Workshop on Challenges in Game AI. Hsieh & Sun Building a Player Strategy Model by Analyzing Replays of Real-Time Strategy Games, Proceedings of IJCNN. Langley Artificial Intelligence and Cognitive Systems, AISB Quarterly. Loyall Believable Agents: Building Interactive Personalities, Ph.D. thesis, CMU. Mateas Believable Agents: Building Interactive Personalities, Ph.D. thesis, CMU.

62 References McCoy & Mateas An Integrated Agent for Playing Real-Time Strategy Games, Proceedings of AAAI. Molineaux, Klenk, Aha Goal-Driven Autonomy in a Navy Strategy Simulation, Proceedings of AAAI. Muñoz-Avila, Aha, Jaidee, Klenk, Molineaux Applying Goal Driven Autonomy to a Team Shooter Game, Proceedings of FLAIRS. Ontañón, Mishra, Sugandh, Ram On-line Case-Based Planning, Computational Intelligence. Russell & Norvig Artificial Intelligence: A Modern Approach. Shannon Programming a Computer for Playing Chess, Philosophical magazine. Thrun Particle Filters in Robotics, Proceedings of UAI.