What Statistical Process Control Taught Me About DevOps
Jowanza Joseph
@jowanza
Agenda
- My Problem
- Statistical Process Control
- DevOps
- Questions
A Story
Architecture
Questions
- How long does it take to get from streaming data to the data lake?
- What's the worst case scenario for my query?
- Can you send me an update every time a certain table changes?
- What do I have to do to get my own copy of the data lake?
- What are the foreign key constraints on these tables?
- Who owns this data?
- Can I get alerts when this cron job kicks off?
The Book
SPC
- Understanding the process
- Eliminate specialized processes
- Monitor and improve the process
SPC
- Data Driven
- Process Oriented
- Understanding Boundaries
Distributions
The average return time of that query is 12 seconds
The average return time of that query is 12 seconds, 99% of queries return faster than 15 seconds
Variations
customers_with_orders
customers_with_orders_timestamp
orders_with_customer_names
orders_streaming
new_customers_stream
Run Charts
USE Method
Utilization
Saturation
Errors
RED Method
Request Rate
Request Errors
Request Duration
Control Charts
Design of Experiments
Continuous Improvement
Check Lists
Process Charts
Bottlenecks
Boundaries
- Limitations are good
- Say no to special requests
- Spend more time thinking
- Make more service oriented pieces
DevOps
Takeaways
- Build metrics and monitoring into your process
- Document every process from end-to-end
- Remove as many "special" processes as you can
- Give context to everything
- Speak in metrics
- Find more reasons to change
Thank You
What Statistical Process Control Taught Me About DevOps
By Jowanza Joseph
