Hacking Consul for fun & profit

Avishai Ish-Shalom (@nukemberg)

Da faq is Consul

  • Service discovery
  • Consistent, redundant DB
  • Events, distributed locks,
    group membership,
    health monitoring

Architecture

Thanks Alex :)

How do we use it?

  • Client side load-balancing
  • HAProxy
  • Prometheus

Sooo.... What's wrong with that picture?

Problems ahoy!

  • Central coupling
  • Limited scaleability
  • Young

We experienced

(go ask Alex...)

  • Extremely high CPU
  • Excessive status updates
    (health checks outputs)
  • Excessive client triggers

Eran was not happy...

Alex steps in...

  • Finds Consul bugs
  • Upgrades Consul
  • Adds DNS Caching
  • Creates static fallback

Then we all stepped in

  • Fix health checks output
  • Improve OB1k client
    • Static fallback
    • Sanity restriction
    • Improved update detection

But were was the noise coming from???

Consul RAFT DB reader

github.com/outbrain/consul-raftdb-reader

A tool to read Consul's commit log

  • Outputs all commits as JSON
  • We loaded into ELK
  • Profit!!!!

Help us make Eran happy!

(Seriously, we have to sit in the same room with him)

Things that make Eran happy

  • Stable health checks (no flapping)
  • Consistent output in health checks
  • Lightweight health checks

We can make you happy too!

github.com/outbrain/consult

Small CLI tool

  • Query Consul
  • SSH into servers
  • HTTP query endpoints

I need to get back to work now

Made with Slides.com