The Challenges of

Containerizing your Data Center

Daniel Hall (@smarthall)

About Me

  • Systems Engineer at LIFX
  • Making the 'Internet' in the Internet of Things

How we did it at LIFX

  • All our applications are stateless
  • This made them all easy to Dockerise
  • Mesos manages the resources
  • Zookeeper helps Mesos choose a master
  • Marathon makes sure daemons are running
  • Chronos runs scheduled and repeating jobs
  • Databases and other things storing state run outside

What does that look like?

Docker Security

  • chroot < Docker < kvm
  • root inside container != root on host
  • Untrusted Docker images are a BAD IDEA(tm)

Improving Docker Security

  • Don't run as root inside the container
  • Use a a recent kernel
  • Run as little as possible inside each container
    • If possible a single statically compiled executable
  • Run SELinux on the host

Finding things

  • You have lots of microservices
  • Marathon keeps moving them
  • Whole machines are going up and down
  • Where is this API running?
  • Which copy of the API do I connect to?

Service Discovery

  • etcd, consul, synapse
  • Marathon comes with an example
    • Marathon knows where things are running
    • Uses HAProxy as load balancer to serivices
    • You run HAProxy on every slave and configure everything to use localhost
    • Not always perfect
  • We use a custom script
    • HTTP routing by putting hostnames in environment variables

Collecting Logs

  • Docker currently has no logging solution
  • You can mount /dev/log but don't restart rsyslog
  • Mesos collects stdout, stderr
    • No easy way to access it
    • No timestamps
  • Correlating logs is great for debugging

Centralised Logs

  • Make rsyslog log to 127.0.0.1
  • Configure a queue to store messages, but drop if full
  • Mount /dev/log into the container
    • You'll need systemd
  • Run several marathon logstash tasks
  • Run elasticsearch on mesos
  • Setup a few small nginx tasks running Kibana
  • TADA! Centralised fault tolerant logs

What it looks like

Troubleshooting

  • Similar to the service discovery problem
  • Breaking in is easier than breaking out
  • Logs inside the image can be hard to get to

Debugging Techniques

  • Find a container in Marathon
  • Use docker exec to run a shell in the container
    • Old versions of docker can use nsenter
    • This won't work for a single executable container
    • You also need tools in there
  • Some debugging tools work from outside
    • pprof for Go
    • jconsole for Java
    • gdb, strace for almost anything

Thank you

Any Questions?

Made with Slides.com