Data Pipelines With Streamsets
Jowanza Joseph
@jowanza
Agenda
- About me
- The Problem Space
- Streaming
- StreamSets
- Demo
- Questions
About Me
- Software Engineer at One ClickRetail
- Scala / Spark / Mesos / Kubernetes
- Author: Apache Spark Fieldbook
- Cyclist
- Husband and father
Retail Intelligence
Data Size
Real-Time
Operational Complexity
Batch Processing
What Are Data Pipelines?
What Problems Do They Solve?
- Scalability
- Complexity
- Observability
- Extendability
Lambda Architecture
Kappa Architecture
Goals
- Data Provenance
- Guaranteed Delivery
- Configurable
- Extendable
- Multi-Protocol Support
- DAG
- Distribute
Based on Streams
Architecture
Running on Mesos
Analytics Data
Real-Time Data
Our Use Case
