Classical Approach
Hadoop Approach
Distributed File System
Highly scalable (storage, processing, bandwidth)
Runs on commodity servers
Allow parallel read/processing of data
Bandwidth scales linearly
Not only size matters,
time also does!
1 Hard Drive (HD) - 100MB/s
Server (12 HD) - 1.2GB/s
Rack (20 servers) - 24GB/s
Average Cluster (6 racks) - 144GB/s
Large Cluster (200 racks) - 4.8TB/s
Big Table
Key value store
(row key, column family, column, timestamp) -> value
Fast random access
Scales up to billions of rows and millions of columns
Multidimensional
Map Reduce
Programming model for parallel data processing
The mapper process a piece of data and the reducer combines all the partial results.
Based on Google papers
Started by Doug Cutting in 2007
It's open source (under the Apache license)
It is a yellow elephant toy
Schema on write VS Schema on read
Fast indexing
Fast joins
Can't add new data structure
RDBMS vs Hadoop
High fidelity data
Flexibility
Fast load of new data structure
HDFS
Files split into blocks and those blocks are replicated in the DataNodes.
Map Reduce
Extra components
Java Map Reduce: High performance and flexibility.
Streaming MapReduce: Allows implementing MapReduce in any programming language.
Crunch: Tool for creating MapReduce pipelines.
Pig Latin: High level language for data processing
Oozie: Allows creating a workflow with the previous tools.
Hive: Converts SQL to MapReduce jobs.
HBase: Big Table implementation for Hadoop.