Hacking Consul for fun & profit

Avishai Ish-Shalom (@nukemberg)

Da faq is Consul

• Service discovery
• Consistent, redundant DB
• Events, distributed locks,
  group membership,
  health monitoring

Architecture

Thanks Alex :)

How do we use it?

• Client side load-balancing
• HAProxy
• Prometheus

Sooo.... What's wrong with that picture?

Problems ahoy!

• Central coupling
• Limited scaleability
• Young

We experienced

(go ask Alex...)

• Extremely high CPU
• Excessive status updates
  (health checks outputs)
• Excessive client triggers

Eran was not happy...

Alex steps in...

• Finds Consul bugs
• Upgrades Consul
• Adds DNS Caching
• Creates static fallback

Then we all stepped in

• Fix health checks output
• Improve OB1k client
  • Static fallback
  • Sanity restriction
  • Improved update detection

Help us make Eran happy!

(Seriously, we have to sit in the same room with him)

Things that make Eran happy

• Stable health checks (no flapping)
• Consistent output in health checks
• Lightweight health checks

We can make you happy too!

github.com/outbrain/consult

Small CLI tool

• Query Consul
• SSH into servers
• HTTP query endpoints

I need to get back to work now