The Power of Choice in! Data-Aware Cluster Scheduling

Transcription

1 The Power of Choice in! Data-Aware Cluster Scheduling Shivaram Venkataraman 1, Aurojit Panda 1 Ganesh Ananthanarayanan 2, Michael Franklin 1, Ion Stoica 1 1 UC Berkeley, 2 Microsoft Research amplab

2 Trends: Big Data 60$ Projected&Growth& Increase&over&2010& 50$ 40$ 30$ 20$ 10$ Moore's$Law$ Overall$Data$ Particle$Accel.$ DNA$Sequencers$ 0$ 2010$ 2011$ 2012$ 2013$ 2014$ 2015$ Data grows faster than Moore s Law! [Kathy Yelick LBNL, VLDBJ 2012, Dhruba Borthakur] 2

3 Trends: Big Data Increase&over&2010& 60$ 50$ 40$ 30$ 20$ Projected&Growth& Moore's$Law$ Overall$Data$ Particle$Accel.$ Facebook Hive cluster Last 4 years: data growth 2500x! queries/day 60x! DNA$Sequencers$ Microsoft Scope Cluster 10$ The number of daily jobs 0$ has doubled every six 2010$ 2011$ 2012$ 2013$ 2014$ 2015$ months for the past two years. [Kathy Yelick LBNL, VLDBJ 2012, Dhruba Borthakur] 3

4 Trends: Low Latency 10 min 1 min 10s 2s 2004: MapReduce batch job 2009: Hive 2010: Dremel 2012: In-memory Spark 4

5 Big Data or Low Latency? SQL Query : 2.5 TB on 100 machines? > 15 minutes 1-5 Minutes < 10s 5

6 Sampling 6

7 Applications Approximate Query Processing blinkdb, presto, minitable Machine learning algorithms stochastic gradient, coordinate descent 7

8 Choices N Any K 8

9 Choices N Any K Sampling à Smaller Inputs + Choice 9

10 Example N = 4 K = 2 M M R 10

11 Existing Available (N) = 2 Required (K) = Rack 3 4 Time Available Data Unavailable Data Running Busy 11

12 Choice-Aware Available (N) = 4 Required (K) = Rack 3 4 Time Available Data Running Busy 12

13 Choice-Aware Available (N) = 4 Launched (M) = 3 Required (K) = Rack 3 4 Time Available Data Running Busy 13

14 KMN Scheduler - How much can KMN improve locality - Propagate benefits across stages - Handling stragglers 14

15 Job à DAG KMN Scheduler One-to-One Many-to-One 15

16 One-to-One Stages Locality Disk ~ 100MB/s Network ~ 10 Gbps (~1GB/s) Memory ~ 50GB/s 16

17 KMN Locality 1 2 Any K N ! N $ # " K & % Choices 17

18 Locality, K=100 K Number of blocks chosen N Number of blocks available 1 K/N=1.0 K/N=0.5 K/N=0.1 Prob. of Locality KMN significantly improves locality Utilization 18

19 Many-to-One Stages KMN Scheduler 19

20 Many-to-One Stage M 1 M 2 M 3 R 1 R 2 15 transfers M 4 R 3 M 5 20

21 M 1 M 2 M 4 M 5 Many-To-One Transfers Core M1 R2 M1 R3 M3 R3 M2 R2 M3 R3 M2 R3 M 3 R 1 R 2 R 3 21

22 Bottleneck Link Core Bottleneck Link Link with Max. transfers Cross Rack Data Skew M 1 M 2 M 4 M 5 M 3 Maximum transfers Minimum transfers R 1 R 2 R 3 = 6 2 = 3 22

23 Facebook Trace Cross Rack Data Skew Maximum transfers Minimum transfers CDF <50 tasks tasks >150 tasks Cross Rack Data Skew 23

24 Power of Choice M 3 M 2 M 1 M 6 M 7 M 4 M 5 Load balancing: balls and bins Insight: Run extra tasks (M > K) Cross Rack Data Skew = 3 24

25 Power of Choice M 3 M 2 M 1 M 6 M 7 M 4 M 5 M = 7, K = 5 Cross Rack Data Skew = 2 Technique: Spread out choice of K tasks to reduce skew 25

26 Handling Stragglers M 1 M 2 M 3 M 4 M 6 M 5 M 7 Rack Rack Stragglers vs. Cross-Rack Data Skew Time 26

27 Using KMN // Create Spark RDD file = sc.textfile( tpc- h.data ) // Select a 10% sample using KMN sample = file.blocksample(0.1) // RDD operations sample.map { li => (li.linestatus, li.quantity) }.collect() 27

28 Also in the paper User-defined sampling functions Placing reduce tasks Killing extra tasks 28

29 Evaluation Facebook traces replay Long DAGs (Stochastic Gradient Descent) SQL queries from Conviva Reducer placement Varying Utilization Baseline: Use a pre-selected random sample Setup: 100 m2.4xlarge EC2 machines, 60GB RAM/mc 29

30 Facebook Overall Baseline KMN-M/K=1.05 Job Size > Job Completion Time (s) 30

31 Cross Rack Skew Cross Rack Skew >8 4-to-8 <=4 Baseline KMN-M/K=1.0 KMN-M/K=1.05 KMN-M/K= Shuffle Stage Time (seconds) 31

32 How many extra tasks? M/K=1.0 M/K=1.1 M/K= Cross-Rack Skew M/K=1.0 M/K=1.1 M/K= Cross Rack Skew tasks > 150 tasks 32

33 KMN: How many stages? Aggregate2 Aggregate3 Stochastic Gradient Descent Aggregate1 Gradient 33

34 KMN: How many stages? KMN Stages Time (s) Aggregate3 Gradient Aggregate2 Aggregate1 Gradient 34

35 KMN: How many stages? KMN Stages Time (s) Gradient Aggregate2 Aggregate3 Gradient + Agg Aggregate1 Gradient 35

36 KMN: How many stages? KMN Stages Time (s) Gradient Aggregate1 Aggregate2 Aggregate3 Gradient + Agg1 Gradient + Agg Gradient 36

37 KMN: How many stages? KMN Stages Time (s) Gradient Aggregate2 Aggregate3 Gradient + Agg1 Gradient + Agg Gradient Aggregate1 Gradient + Agg

38 Related Work Power of Choice Power-of-Two choices [TPDS 01] Sparrow [SOSP 13] Improving Cluster Scheduling Quincy [SOSP 09] alsched [SOCC 12] Dolly [NSDI 13] 38

39 KMN Scheduler N Any K Emerging applications: ML algorithms, AQP Improves locality, Balances network transfers 39