MapReduce: Simplified Data Processing on Large Clusters

Transcription

1 MapReduce: Simplified Data Processing on Large Clusters Jeffrey Dean and Sanjay Ghemawat September 1, 2015

2 ments. The user would write code similar to the follow- Programming pseudo-code: Model The map function emits each word plus an associated map(string key, String value): // key: document name // value: document contents for each word w in value: EmitIntermediate(w, "1"); reduce(string key, Iterator values): // key: a word // values: a list of counts int result = 0; for each v in values: result += ParseInt(v); Emit(AsString(result));

3 Execution Model (1) fork User Program (1) fork (1) fork (2) assign map Master (2) assign reduce worker split 0 split 1 split 2 split 3 split 4 (3) read worker (4) local write (5) remote read worker worker (6) write output file 0 output file 1 worker Input files Map phase Intermediate files (on local disks) Reduce phase Output files Figure 1: Execution overview Inverted Index: The map function parses each document, and emits a sequence Presenter: of word, Aditya document Devarakonda ID large clusters of commodity PCs connected together with switched MapReduce: Ethernet Simplified [4]. In our Data environment: Processing on Large Clusters

4 Prior Art Programming models: OpenMP, MPI, UPC, Charm++.

5 Prior Art Programming models: OpenMP, MPI, UPC, Charm++. Resource management: SLURM, HTCondor.

6 Prior Art Programming models: OpenMP, MPI, UPC, Charm++. Resource management: SLURM, HTCondor. Machine setup: Supercomputer, network of computers.

7 Key Players in MapReduce Master

8 Key Players in MapReduce Master Workers (Mappers and Reducers)

9 Key Players in MapReduce Master Workers (Mappers and Reducers) Combiners

10 Key Players in MapReduce Master Workers (Mappers and Reducers) Combiners Partitioners

11 Fault Tolerance Worker failure

12 Fault Tolerance Worker failure Master failure

13 Parallel Computing Issues Locality

14 Parallel Computing Issues Locality Load balancing

15 Parallel Computing Issues Locality Load balancing Communication

16 Parallel Computing Issues Locality Load balancing Communication Invariants (What assumptions does MR make).

17 Performance on 1TB Sort Input (MB/s) Done Input (MB/s) Done Input (MB/s) Done Shuffle (MB/s) Shuffle (MB/s) Shuffle (MB/s) Output (MB/s) Seconds (a) Normal execution Output (MB/s) Seconds (b) No backup tasks Output (MB/s) Seconds (c) 200 tasks killed Figure 3: Data transfer rates over time for different executions of the sort program original text line as the intermediate key/value pair. We used a built-in Identity function as the Reduce operator. the first batch of approximately 1700 reduce tasks (the entire MapReduce was assigned about 1700 machines,

18 Usage at Google Figure 4: MapReduce instances over time Number of jobs 29,423 Average job completion time 634 secs Machine days used 79,186 days Input data read 3,288 TB Intermediate data produced 758 TB Output data written 193 TB Average worker machines per job 157 Average worker deaths per job 1.2 Average map tasks per job 3,351 Average reduce tasks per job 55 Unique map implementations 395 Unique reduce implementations 269 Unique map/reduce combinations 426

19 Conclusion [MapReduce] allows us to express the simple computations we were trying to perform but hides the messy details of parallelization, fault-tolerance, data distribution and load balancing in a library. [Introduction, DG04] My take away: Simplicity at the cost of generality and performance. However, it is a novel, easy-to-use, quick-to-production system for *simple* data parallel programs and applications on *commodity* clusters where fault tolerance is a primary goal.