source: http://www.gartner.com/technology/research/methodologies/hype-cycle.jsp

EMR 4

EMR is the largest Hadoop distribution by market share

EMR 4 based on Apache Bigtop

Integration with AWS services (EC2, S3, Redshift)

Ephemeral clusters

Intelligent resizing

EMR Sandbox (Zeppelin)

Spark 1.5

Not for everyone (cloud lock-in)

Spark

We write distributed applications

Right level of abstraction to reason about

"Write once, run everywhere"

Huge momentum (also hype)

Frameworks on top of primitives (DataFrame, MLLib)

Zeppelin

80% of data science/engineering is data munging

Document datasets, share steps and results

Killer app to work with and collaborate on datasets

Familiar REPL experience