Large-Scale Platform for MOBA Game AI

Transcription

1 Large-Scale Platform for MOBA Game AI Bin Wu & Qiang Fu 28 th March 2018

2 Outline Introduction Learning algorithms Computing platform Demonstration

3 Game AI Development Early exploration Transition Rapid development Explosive growth 1950s-1960s 1970s-1980s 1990s-2000s 2010s Checkers beat state champion Chess4.5 beat human players Deep Blue (IBM) beat Garry Kasparov Alpha Go (DeepMind) defeat Lee Sedol, and Jie Ke

4 Applications of Game AI Gaming Research Core applications in gaming industry Ideal testbed for general AI research Pre-game procedures e.g., game designing Player experience e.g., teammates, enemies Others e.g., E-sports Massive data from human players Low experimental costs General ability for inception and decision From virtual world to real world

5 Game AI Research Topic Game AI has become a research hot topic after the success of AlphaGo Many AI giants have joined game AI research Moving from Go->RTS, MOBA, etc. DOTA 2 1v1 beat top human players. 5v5 to be activated in 2018 Released Starcraft AI platform, preliminary results in simple scenarios Released Starcraft II AI platform,not able to defeat built-in AI

6 MOBA Game 5 v.s. 5 game Obtain gold/exp gain advantages on equipment win fights destroy enemy s base Enemy s Turrets Goal: Destroy Enemy s Base Attack/Skills Control Movement Control Neutral creeps: source of money/power/level/ Equipment Purchase

7 MOBA Game Micro Combat Movement Use of skills

8 MOBA Game Macro Strategies Back up Laning Ganking Stealing base

9 MOBA AI - Key Challenge Learning Algorithm Computing Platform

10 Learning Algorithms

11 Learning Algorithms - Challenges Complexity 10^20000 Multi-agent 5v5 coordination Sparse and delayed rewards 20,000+ frames per game Imperfect Info Partially observable

12 Learning Algorithms - Challenges Complexity >> Go End to end solutions (SL/RL) do not work well Not able to finish basic movement/attack Similar observations made DeepMind Go MOBA State space 3^360 10^170 (361 pos, 3 states each) 10^20000 (10 heroes,2000+pos * 10+states) Left, right, Skill 1,2,3+pos/target recover return etc Action space 250^150 10^360 (250 pos available, 150 decisions per game in average) 20^ ^20000 (20 actions,20,000 frames per game)

13 Learning Algorithms - Challenges Multi-agent Macro strategy level Four defending, while one steals the base Micro combat level Tanks protecting assassins

14 Learning Algorithms - Challenges Sparse and delayed rewards Go <360 steps MOBA >20,000 steps

15 Learning Algorithms - Challenges Imperfect Information Maps are partially observable Guess enemy s positions/strategy Actively explore to gain vision

16 Model Architecture Divide and Conquer Strategy Transfer Split for simplification Solution space ~10^20000->~10^2000 Combat

17 Model - Transfer Where to send heroes? Compared to Go game Put heroes as stones Put maps as boards Predict good position Hotspots Prediction Transfer

18 Model - Strategy Key resources in MOBA Modeling macro objectives Describe hotspots transition series before destroying the key resource

19 Model - Strategy 宏观 Session 切分示例 Describe hotspots transition series before destroying the key resource Start Dragon Mid 1st turret Slain Dark dragon Dragon Bottom 1 st turret Mid 2 nd turret Mid 3 rd turret Base Stealing blue creep Killing bottom lane creeps Attacking bottom 1 st turret

20 Model - Transfer Network with Macro Strategy Key resources Hotspots

21 Model - Combat Multi-task on buttons Action space Directions Skill releasing position

22 Learning Framework Imitation + Reinforcement Learning

23 Computing Platform

24 MOBA Game AI Platform Computing Platform Computational power large-scale CPU/GPU virtualization Learning platform Efficient and easy-to-use platform Service Feature extraction Game environment deployment Model training Reinforcement learning Task Managem ent Tencent cloud function Machine learning Resource allocation Elastic computation Kubernetes resource allocation Deploym ent Online service Idle resource pool Online service Offline service Docker + mixed online/offline technique Docker + GPU virtualization for shared resource Computat ional units Millions of CPUs Thousands of GPUs

25 Computational Power Computational Costs GPU CPU MOBA AI thousands millions CPU/GPU Demands Challenge Solution The more is the better Improve resource utilization efficiency without additional costs CPU/GPU virtualization for shared resources

26 CPU Virtualization Elastic and dynamic resource pool millions of CPU cores 70% - Idle resource pool New resources not yet delivered Old resources not yet cleared Returned resources 30% - Idle slots in online service Online service resource usage # of CPU cores # CPU avg % Percentage millions 20% 20%->65% using docker isolation Elastic & Dynamic Resource Pool

27 GPU Virtualization Goal: improve GPU usage efficiency Resource usage # of GPU % of low load machines GPU avg usage thousands 65% 28% Optimization idea

28 GPU Virtualization [12] Time-slice share Parallel share

29 Learning Platform Core Technique Version Update Frequency Feature extraction Hours Model training One day RL training One day

30 Learning Platform - Feature Extraction Platform Game replays Game Raw Data Features Training samples Models Evaluation pre Feature extraction shuffle Training Evaluation gamecore Demand 1 Feature extraction from up to hundreds of thousands of replays Challenge: demands up to 210 thousand CPU cores per day Solution CPU virtualization docker elastic & dynamic resource pool Demand 2 Multiple tasks, each with millions of entries Challenge: Parallel task scheduling Solution: Tencent Serverless Cloud Function

31 Learning Platform - Serverless Cloud Function Advantage of Cloud Function Function As A Service Millions of CPU cores available Free of charge in idle slots 30% of costs on average SDK SDK COS CMQ Application layer API Access layer Function Call Function Config Function Coordination Control layer Function Function Function Execution layer

32 Learning Platform - Model Training Platform 1.Requirement Billions of samples per task Fast model training 2.Solution Multi-GPU, multi-machine Machine learning platform 3.Challenges IO Efficient data inputs Efficient computation Communication Efficient parameters exchange Training Platform Big Data Result

33 Model Training Platform - IO Data IO Multiprocessing Lock free queue Efficient computation Data pre-caching OP speed up by multi-threading

34 Model Training Platform - Communication Parameters exchange NCCL2 [11] Efficient communication between GPUs RDMA Efficient communication across nodes

35 Model Training Platform - Performance Acceleration 70 Multi-GPU Multi-Machine Speed-up Optimization results (acceleration ratio) GPU 8GPUs 16GPUs 32GPUs 64GPUs Before After Upper bound IO Computation Communication

36 Learning Platform - Reinforcement Learning Platform Demands Hierarchical RL Various scenarios Large-scale parallel self-play Millions of games Automatic task management Unified framework Model analysis Evaluation

37 RL Platform - Hierarchical RL Hierarchical RL Scenario specific Solution General Hierarchical RL Features Macro task selection Micro task selection Effectively handles long-term planning and delayed rewards Value network for guiding sub-task policy learning 打野 清兵 团战 Jungle Laning Combat

38 RL Platform Parallel Training Large-scale parallel self-play Solution Docker image for gamecore version management Parallel training framework

39 RL Platform Automatic task management Unified framework for model analysis and evaluation Task submission Task start/stop Results visualization Reward curve 雷达图 Prediction distribution Self play results

40 RL Platform Performance Ten million scenarios per day 20s per scenario with 16 GPUs Millions of full games 10min+ per game with 128 GPUs

41 Demonstration

42 Visualization

43 Demo Quadra-kill Under Turret Micro combat Fight against mid-high level testers Killing while avoiding harm from turret

44 Demo Pentakill Micro combat Fight against mid-high level testers

45 Demo Transfer & Strategy Opening

46 Demo Transfer & Strategy First Dragon appears at 2:00

47 Demo Transfer & Strategy Besiege and Destroy the Base

48 Demo RL Before reinforcement After reinforcement

49 Summary

50 Tencent Game AI Research Pursue general AI via game AI research MOBA AI Algorithm Imitation + Reinforcement Learning Computing platform Feature extraction platform Millions of CPUs Model training platform Thousands of GPUs Reinforcement learning platform Hierarchical RL

51 Tencent Game AI Research Future work Algorithm Tactic-level search and planning Multi-agent RL Computational power Search/planning platform MCTS Reinforcement learning platform Multi-agent RL

52 About Tencent AI Lab Our journey Tencent is identified by China Ministry of Science and Technology to build national open innovation platform for AI medical imaging Today Our team consists of 70 world-class AI scientists and 300 research engineers Tencent establishes its corporate-level AI Lab Tencent announces leading AI researcher Dr Tong ZHANG as the Director of Tencent AI Lab Jueyi Fine Art wins the UEC World Cup Tencent establishes its Seattle AI Lab and announces leading Speech Recognition expert Dr Dong Yu as Deputy Director

53 About Tencent AI Lab Game AI Environment for AGI Diverse game ecosystem Massive user base Social AI New ways to communicate WeChat: ~1 billion MAU QQ: 850 million MAU Content AI Perceiving the world and generating content China s leading news, video, music and literature platforms Medical AI Impact and advance industry Building a national open innovation platform for AI medical imaging

54 Thank you

55 References [1] Silver, David, et al. "Mastering the game of Go with deep neural networks and tree search." Nature (2016): [2] Artificial Intelligence Startup Landscape Trends and Insights - Q NOVEMBER 20, 2016 VENTURE SCANNER. [3] Tian, Yuandong, et al. "ELF: An Extensive, Lightweight and Flexible Research Platform for Real-time Strategy Games." arxiv preprint arxiv: (2017). [4] O Vinyals et al. StarCraft II: A New Challenge for Reinforcement Learning. Aug. 9, 2017 [5] We've created an AI which beats the world's top professionals at 1v1 matches of Dota 2. [6] Ontanó n, Santiago, Gabriel Synnaeve, Alberto Uriarte, Florian Richoux, David Churchill, and Mike Preuss. "RTS AI: Problems and Techniques." (2015): [7] Miles, Chris, and Sushil J. Louis. "Co-evolving real-time strategy game playing influence map trees with genetic algorithms." Proceedings of the International Congress on Evolutionary Computation, Portland, Oregon. IEEE Press, [8] Jang, Su-Hyung, and Sung-Bae Cho. "Evolving neural NPCs with layered influence map in the real-time simulation game Conqueror." Computational Intelligence and Games, CIG'08. IEEE Symposium on. IEEE, [9] Weber, Ben George, Michael Mateas, and Arnav Jhala. "Building Human-Level AI for Real-Time Strategy Games." AAAI Fall Symposium: Advances in Cognitive Systems. Vol [10] Xingjian, S. H. I., et al. "Convolutional LSTM network: A machine learning approach for precipitation nowcasting." Advances in neural information processing systems [11] Nathan Luehr. NCCL: ACCELERATED COLLECTIVE COMMUNICATIONS FOR GPUS. April 5, GPU Technology Conference [12] CUDA MULTI-PROCESS SERVICE.