Data Engineering
Di Analytic Report dan Tracker SMTP
Data Engineering
- Develop and Maintains Data Pipelines
- Design, Develop and Maintain Database Architecture
- Raw
- Not yet determined
- Highly accessible and quick to update
Data Lake
Data
Warehouse
- Processed
- Currently in use
- More complicated and costly to make changes
ScyllaDB (Cassandra2.0)
- Data Lake
- Scylla C++ VS Cassandra Java
- Retention Policy
- Strict Filtering
- https://notes.mtarget.co/doc/cassandra-documentation-BmfnQOSvXk
PostgreSQL
- Data Warehouse
- PostgreSQL vs MySQL
- Easy filtering
Queue
Kafka
For High Load Data
Rabbit
For Low Load Data
Extract Transform Load (ETL)
- Apache Spark
- Streaming
- SQL
Apache Beam
Cache
- Redis
- TTL
Thank You!
Questions?
https://slides.com/ciazhar/data-engineering