Elastic Search and Apache Spark
Jowanza Joseph
@jowanza
www.jowanza.com
Agenda
- About Me ~ 1 Minute
- Data Pipelines ~ 5 Minutes
- Distributed Data ~ 5 Minutes
- Batch / Streaming ~ 3 Minutes
- Operations ~ 5 Minutes
- Elastic + Spark ~ 3 Minutes
- Code ~ 15 Minutes
About Me
- Senior Software Engineer at One Click Retail
- Scala/ Java: Spark, Flink, Kafka
- Writing a Book about Apache Spark
- Cyclist / Golfer
- Father
Data Pipelines
Distributed Data
A Brief History
Streaming
Operations
Lambda Architecture
ElasticSearch
- Master-Master
- Memory Optimizations
- Supports The Hadoop Ecosystem
- Supports Streaming & Batch
- Full-text Search
Is JSON Good?
- No Types
- No Compression
- Universal?
- Difficult to Marshall
- No suitable for joins
Marshalling
Apache Spark
- In-Memory Computing
- Rich Ecosystem
- Mature Query APIs
- Performance
Elastic + Spark
- Easily Distributed
- Batch Workloads
- Full-Text Search
- Streaming Workloads
- Large Ecosystem
- Costly (But that's OK)
Batch Demo
Streaming Demo
