I'd rather play golf

This past Monday I listened to Shivnath Babu of Duke University present a lecture entitled “MADDER and Self-tuning data analytics on Hadoop with Starfish“. In a nutshell, the Starfish project is about the development of self-tuning and self-managing technology for Hadoop systems, and many of the same challenges that exist in self-managing relational database systems, such as SQL Anywhere, have their parallels in Hadoop systems.

Papers and source code for Starfish can be found here, made available under the Apache license.

My notes from Shivnath’s talk (slides can be found here):

• Big data analytics is often used in the context of companies such as Google, Yahoo!, Facebook, and eBay. However, web-scale data exists in other realms as well: scientists, journalists, economists, biologists, physicists, and systems researchers all have a “big data” problem. What matters is not just raw size, but also workload complexity: counts and aggregates, are one example, text/image processing is another.
• “Madder” comes from “Mad”, which is from Joe Hellerstein and others at UC Berkeley. “Madder” is an acronym for Magnetic, Agile, Deep, Data-lifecycle-aware, Elastic, and Robust:
  • Magnetic: easy to get data into the system;
  • Agile: make changes (data and/or requirements) easy;
  • Deep: support the full spectrum of analytics – write MapReduce programs in Java, Python, or R, or use interfaces like Pig or Jaql;
  • Data-lifecycle-aware: difficult to quantify – consider all phases of data processing over information; not just loading, but processing, archiving, and so on;
  • Elastic: adapt resources and costs to the actual workload;
  • Robust: graceful degradation during unanticipated events or increases in workload.
• What are the tuning problems with Hadoop clusters? They include (1) job-level MapReduce configuration, (2) workflow optimization, (3) workload management, (4) data layout tuning, and (5) cluster sizing.
• A MapReduce job can be represented as a 4-tuple: job j = < program p, data d, resources r, configuration c >. But Hadoop has upwards of 190 configuration parameters, and the space of potential choices includes the number of Map tasks, the number of Reduce tasks, the amount of partitioning between Map and Reduce tasks, memory allocation, and so on. With so many configuration parameters, optimal execution can be a problem.
• Starfish contains three components: a Profiler, a “What-if” engine, and optimizers. The Profiler executes MapReduce jobs to collect job profiles (a concise execution summary) that includes the recording of information of task “phases” (read, map, collect, spill, merge). The “What-if” engine, given a profile of j = < p, d, r, c > estimates the profile for j’ = < p, d', r', c' >. The optimizers enumerate execution plans but analyzing the optimization space efficiently.
• Generating execution profiles can be done either via measurement, or via the What-if engine. The goals of generating profiles via measurement are (1) have zero overhead when off, low overhead when on; (2) require no mods to Hadoop itself; and (3) support unmodified MapReduce programs written in Java/Python/Ruby/C++.
• Profiling approach: dynamic on-demand instrumentation. Event-condition-action rules are specified in Java, which monitor task phases of MapReduce job execution. Currently this uses BTrace (Hadoop internals are in Java). Instrumentation introduces the capturning of raw profiling data at each JVM for both map and reduce steps. For jobs that involve large numbers of map tasks, sampling is used to reduce the amount of overhead. Total overhead percentage introduced is between 5% to 30%, depending on how much of the complete workload is profiled.
• The “What-if” engine takes a job profile, and hypothetical properties, resources, and configuration settings, and creates a virtual job profile. In Starfish today, this does not generalize across different programs, and the analysis is done through simulation. Eventually the goal is to move the “What-if” engine into Hadoop itself.
• “What if” questions that Stafish can answer:
  1. how will job j’s execution time change if the number of reduce tasks is changed from 20 to 40?
  2. how will compression impact Map performance? what if the amount of data is increased by 40%?

  This is all a means to an end; goal of Starfish is to get to the Job Optimizer, which is to find the best configuration settings for a hypothetical job j = < p, d', r' >.

• Experimental results indicate that “What-if” analysis, based on cost-based optimization, is better than routinely applying “rules of thumb” to Hadoop configurations. However, Starfish does not (yet) support dynamic run-time adaptive processing, which could improve overall performance significantly.