The Challenges of

Containerizing your Data Center

Daniel Hall (@smarthall)

About Me

• Systems Engineer at LIFX
• Making the 'Internet' in the Internet of Things

How we did it at LIFX

• All our applications are stateless
• This made them all easy to Dockerise
• Mesos manages the resources
• Zookeeper helps Mesos choose a master
• Marathon makes sure daemons are running
• Chronos runs scheduled and repeating jobs
• Databases and other things storing state run outside

Docker Security

• chroot < Docker < kvm
• root inside container != root on host
• Untrusted Docker images are a BAD IDEA(tm)

Improving Docker Security

• Don't run as root inside the container
• Use a a recent kernel
• Run as little as possible inside each container
  • If possible a single statically compiled executable
• Run SELinux on the host

Finding things

• You have lots of microservices
• Marathon keeps moving them
• Whole machines are going up and down
• Where is this API running?
• Which copy of the API do I connect to?

Service Discovery

• etcd, consul, synapse
• Marathon comes with an example
  • Marathon knows where things are running
  • Uses HAProxy as load balancer to serivices
  • You run HAProxy on every slave and configure everything to use localhost
  • Not always perfect
• We use a custom script
  • HTTP routing by putting hostnames in environment variables

Collecting Logs

• Docker currently has no logging solution
• You can mount /dev/log but don't restart rsyslog
• Mesos collects stdout, stderr
  • No easy way to access it
  • No timestamps
• Correlating logs is great for debugging

Centralised Logs

• Make rsyslog log to 127.0.0.1
• Configure a queue to store messages, but drop if full
• Mount /dev/log into the container
  • You'll need systemd
• Run several marathon logstash tasks
• Run elasticsearch on mesos
• Setup a few small nginx tasks running Kibana
• TADA! Centralised fault tolerant logs

Troubleshooting

• Similar to the service discovery problem
• Breaking in is easier than breaking out
• Logs inside the image can be hard to get to

Debugging Techniques

• Find a container in Marathon
• Use docker exec to run a shell in the container
  • Old versions of docker can use nsenter
  • This won't work for a single executable container
  • You also need tools in there
• Some debugging tools work from outside
  • pprof for Go
  • jconsole for Java
  • gdb, strace for almost anything