Data Pipelines

some DevOps ideas

how to assess Pipelines 

tools to measure a Pipeline (PL) status:

  • PL Maturity Matrix
  • Service Level Objectives (SLOs) and  Indicators (SLIs)

Pipeline maturity matrix

 measures five key characteristics (Can be extended):

  • Failure tolerance
  • Scalability
  • Monitoring and debugging
  • Transparency and ease of implementation
  • Unit and integration testing

 

 

Source: https://sre.google/workbook/data-processing/#pipeline-maturity-matrix

Pipelines SLOs and SLIs

e.g. of Service Level Objectives and Indicators

  • Data freshness: e.g. X% of data processed in Y minutes, The oldest data is no older than X minutes, The PL job has completed successfully within X minutes
  • Data correctness: e.g.  On a per-job basis: less than X% of input items contain data errors; Over an X-minute moving window: less than Y% of input items contain data errors.
        Data is checked against reference data: via tests in CI/CD, end-2-end PL tests in pre-production env, monitoring PLs@prod to observe metrics related to data correctness
  • Data isolation/load balancing: e.g. High-priority jobs are processed within 10 mins, Standard-priority jobs are processed by 9 AM the next business day

source: https://cloud.google.com/solutions/building-production-ready-data-pipelines-using-dataflow-planning#defining_and_measuring_slos

Data characteristics

  • Data is immutable (today will never produce data for yesterday), time-based and independent from user

  • Data Partition: looks logical to partition per datetime (by day?), by source

An idea of a plan

Initial phases (can overlap):

  • Status: Infrastructure as it is today
  • Roadmap: define goals/plan
  • Actions: start acting towards goals

 

Goals: improve on PL Maturity Matrix, SLOs + SLIs, reduce costs (time to market, money of running Infra), CI/CD, automate operations, security everywhere

Long term vision

Reactive Services 4 key tenets:

  • Responsive: the PLs show results
  • Resilient: recover from errors (an external API failed, workers died, etc)
  • Elastic: grows with load, shrinks on idle times
  • Message Driven: communication is asynchronous (Redis + Sidekiq is already there)

Achieve the vision

Divide and Conquer

  • Parallelize
  • Isolate: separate every data flow (PL), paths for Read and Write, minimize interactions between services
  • Services only do 1 thing and do it well
  • Async Messaging: (a perfect world = non-blocking API calls with callback urls for receiving results)
  • Share nothing Architecture
  • Immutable infrastructure

 

close work with Tech Team is a must!

1. Phase: Status  - Goals

Goals: 

  • handover Operations
  • list pain points on: infrastructure (single point of failures, metrics, observability), operations, CI/CD, processes, security (CI/CD, code, infra, ops)
  • derive a backlog of existing issues, and prioritize them

 

Expected Outcome:

  • full picture of infra + PLs + documentation
  • starting work on: Current issues on Infra + CI/CD, ops
  • current and future SLOs/SLIs, Maturity Matrix of PLs

1. Phase: Status  - Actions

Actions:

  • Operations: AWS, jenkins, scripts
  • Infra:
        - cost
        - observability = logs, alerts, metrics (!) Ideas of tools: newrelic, airbrake, prometheus (far future)
        - security
        - incident processes
  • CI/CD .. speed, reliability, security, ease of use
  • Code:
        - Rails: e.g. are queue configs in code? check Writes and Reads
        - UI: e.g. API for searches (read)
  • Data: find partition strategies (e.g. by day, by source)

1. Phase: Status  - Questions

e.g. Questions:

  • Redis: what happens if we lose a sidekiq message? is Redis Highly Available (HA)? is it Load Balanced (LB)?
  • Mongo: is data partitioned? is it HA? LB? Backups?
  • Code:
    - Rails API dependency: what happens if 1 external API fails?
    - UI: read queries are coupled to Mongo's APIs? (low prio)

2. Phase: RoadMap

Goals:

  • refine SLOs and SLIs
  • work on the Maturity Matrix improvements by priority
  • Performance improvements:
      - Infra: Load Balance queues+workers, new server creation: cost? time needed? impediments? Linux servers performance, Linux Image (idea for the future: containerize app)
      - CI/CD, processes and ops
  • PL best practices and design, tech exchanges with team

a Rails worker...

Given: a Rails sidekiq worker that is fully configurable (via a Configuration System) at server creation or reconfigured+restarted

Then: configure Workers to
   - read from queue x, write to queue y => Event
   - read from source n, write to sink m => Data
 

Question: where are the queues configuration now?
   

This achieves Isolation of jobs/PLs: Load Balance jobs (add more workers for a high-priority queue), monitor/observe separately, etc

Some food for thought

thanks for reading :-)

Made with Slides.com