Dovetail Pipeline
Pipeline Overview
Generate
Input API
ES Connector
Output API
Output
Pipeline Completed
Generate
Input API
ES Connector
Output API
Output
Pipeline Completed
Generate
Input API
ES Connector
Output API
- CLI Tool to generate realistic sample data
- Generates many thousands of Events/sec
- Many districts, schools, and grades
- Realistic assessment data with heuristic outcomes and skills association
Pipeline Completed
Generate
Input API
ES Connector
Output API
- Dovetail entry point
- JAX-RS compliant REST framework
- CloudFormation deployed in DEV env
- Autoscaling group
Pipeline Completed
Generate
Input API
ES Connector
Output API
- Worker process based on AWS KCL
- Processes batched records from Kinesis
- Elasticsearch bulk indexing requests
- Scales with shards
- CloudFormation - Autoscaling
Pipeline Completed
Generate
Input API
ES Connector
Output API
- JAX-RS compliant REST API
- Proxies only query requests to ES
- Multithreaded with low (< 5%) overhead