LearnViz CompSci
Home / Learn / Distributed Systems & Cloud / MapReduce

Intermediate Topic 33 of 51 Distributed Systems & Cloud

MapReduce

Programming model splitting work into map and reduce phases.

How it works

MapReduce is a programming model and framework for processing large datasets in parallel across a cluster. A map function processes input records and emits intermediate key-value pairs; the framework shuffles and groups pairs by key; a reduce function combines the values for each key into results. The runtime handles partitioning input, scheduling tasks near their data, retrying failed tasks and stragglers with speculative execution and writing results to a distributed file system. Combiners reduce shuffle data.

Where it is used

Word counts, log analysis, web indexing, ETL, inverted index construction and large-scale joins were classic MapReduce jobs at Google and in Hadoop.

Common pitfall

Disk-based intermediate data makes iterative algorithms and interactive queries slow, which led to Spark and other engines keeping data in memory. Skewed keys create straggling reducers. Writing complex pipelines as chains of jobs is awkward, so higher-level tools such as Hive and Pig emerged. Design for associative, commutative reductions where possible.

Where it fits in the course

MapReduce is topic 33 of 51 in Distributed Systems & Cloud, rated Intermediate. It follows Distributed File Systems. The next topic is Batch vs Stream Processing.

Practice and explore

How to study this topic

  1. Read the explanation above and restate the definition in your own words.
  2. Open the interactive lesson and answer its mini quiz.
  3. Read the common pitfall, then write a short example that avoids it.
  4. Check your understanding in the Distributed Systems & Cloud quiz, and revisit the topic if you miss a question.

Frequently asked questions

What is MapReduce?
Programming model splitting work into map and reduce phases. It belongs to Distributed Systems & Cloud and is an intermediate-level topic.
Where do people use MapReduce?
Word counts, log analysis, web indexing, ETL, inverted index construction and large-scale joins were classic MapReduce jobs at Google and in Hadoop.
What is the most common mistake with MapReduce?
Disk-based intermediate data makes iterative algorithms and interactive queries slow, which led to Spark and other engines keeping data in memory.
What should I learn before this topic?
Start with Distributed File Systems, the topic just before it in Distributed Systems & Cloud.
How can I practice it?
Answer the mini quiz in the interactive lesson, then take the Distributed Systems & Cloud quiz.
Is MapReduce hard to learn?
It is an Intermediate topic. Be comfortable with the Beginner topics in this area first, then read the common pitfall carefully, since it is where most mistakes happen.
How long does it take to learn MapReduce?
Most learners can read the explanation and finish the interactive lesson in under an hour. Fluency comes from applying the idea, so plan a few short practice sessions with the quiz.
Which topics are related to MapReduce?
Distributed File Systems, Batch vs Stream Processing, Byzantine Fault Tolerance are closely connected. The full list is in the related topics section of this page.
What should I study after MapReduce?
Continue with Batch vs Stream Processing, the next topic in Distributed Systems & Cloud.

Related topics