Dovetail Pipeline

Pipeline Overview

Generate

Input API

ES Connector

Output API

Output

Pipeline Completed

Generate

Input API

ES Connector

Output API

Output

Pipeline Completed

Generate

Input API

ES Connector

Output API

• CLI Tool to generate realistic sample data
• Generates many thousands of Events/sec
• Many districts, schools, and grades
• Realistic assessment data with heuristic outcomes and skills association

Pipeline Completed

Generate

Input API

ES Connector

Output API

• Dovetail entry point
• JAX-RS compliant REST framework
• CloudFormation deployed in DEV env
• Autoscaling group

Pipeline Completed

Generate

Input API

ES Connector

Output API

• Worker process based on AWS KCL
• Processes batched records from Kinesis
• Elasticsearch bulk indexing requests
• Scales with shards
• CloudFormation - Autoscaling

Pipeline Completed

Generate

Input API

ES Connector

Output API

• JAX-RS compliant REST API
• Proxies only query requests to ES
• Multithreaded with low (< 5%) overhead

Dovetail Pipeline

Demo