Resilient, Cost Effective Data Analytics Architecture
OLAP vs OLTP
Motivation
Combined Compute
CAP
Scale for $$$
Separate Storage and Compute
Distributed Multi-tenant architecture
READ ONLY
Updates / Deletes/ Edits
Costs
Storage
$25/TB/Month Per Client
Compute
$5 Per TB Scanned
Network
Negligible
It's not perfect
We have to manage our own files
It's Actually Not That Bad
Data File Formats On Distributed File Systems
Why do file formats matter?
Compaction
Quicker Reads
What features do you need?
- Compact (saves space)
- Reduce scan time
- Allows for quick scans
Columnar Storage
It's Just SQL Pointed At The File
SELECT origin, count(*) AS total_departures
FROM flights_avro_example
WHERE year >= '2000'
GROUP BY origin
ORDER BY total_departures DESC
LIMIT 10;Performance
| Query | Costs | Speed | Amount Scanned |
|---|---|---|---|
| SELECT cab_type, count(*) FROM trips_parquet GROUP BY cab_type; | $.005 | 6 Seconds | 600MB |
| Query | Costs | Speed | Amount Scanned |
|---|---|---|---|
| SELECT passenger_count, avg(total_amount) FROM trips_parquet GROUP BY passenger_count; | $.50 | 6 Seconds | 102GB |
| Query | Costs | Speed | Amount Scanned |
|---|---|---|---|
| SELECT passenger_count, year(pickup_datetime), count(*) FROM trips_parquet GROUP BY passenger_count, year(pickup_datetime); | $.50 | 6 Seconds | 101GB |
Alternative To Athena
Ultimate Modularity
Operational Overhead
That's It
File Formats
By Jowanza Joseph
