Hacking Consul for fun & profit
Avishai Ish-Shalom (@nukemberg)
Da faq is Consul
- Service discovery
- Consistent, redundant DB
- Events, distributed locks,
group membership,
health monitoring
Architecture
Thanks Alex :)
How do we use it?
- Client side load-balancing
- HAProxy
- Prometheus
Sooo.... What's wrong with that picture?
Problems ahoy!
- Central coupling
- Limited scaleability
- Young
We experienced
(go ask Alex...)
- Extremely high CPU
- Excessive status updates
(health checks outputs) - Excessive client triggers
Eran was not happy...
Alex steps in...
- Finds Consul bugs
- Upgrades Consul
- Adds DNS Caching
- Creates static fallback
Then we all stepped in
- Fix health checks output
- Improve OB1k client
- Static fallback
- Sanity restriction
- Improved update detection
But were was the noise coming from???
Consul RAFT DB reader
github.com/outbrain/consul-raftdb-reader
A tool to read Consul's commit log
- Outputs all commits as JSON
- We loaded into ELK
- Profit!!!!
Help us make Eran happy!
(Seriously, we have to sit in the same room with him)
Things that make Eran happy
- Stable health checks (no flapping)
- Consistent output in health checks
- Lightweight health checks
We can make you happy too!
I need to get back to work now
