Building a Docker PaaS

A Docker networking case study

@lukeb0nd

http://yld.io

ContainerCamp 2015

San Francisco

@lukeb0nd

http://yld.io

ContainerCamp San Francisco 2015

In the next 25 mins...

• Problems in building a Docker PaaS

• Deep dive into service discovery, using Paz as a reference point

• Along the way discuss the Docker ecosystem, and Docker networking in particular

@lukeb0nd

http://yld.io

ContainerCamp San Francisco 2015

Who am I?

• I'm a server developer at YLD.io
  • Software engineering consultency
  • We do mostly Node.js and Docker
• Built a Docker PaaS called "Paz"
• • ​http://paz.sh
  • http://paz.readme.io

@lukeb0nd

luke@yld.io

What's in a Docker PaaS?

Some problems that may need solving in a Docker PaaS:

• Workflow

• Scheduling

• Service Discovery

  • Routing

• DNS

• Zero-downtime deployments

ContainerCamp San Francisco 2015

@lukeb0nd

luke@yld.io

What is Service Discovery?

Service A needs to speak to service B.

1. How does service A know how to address service B?
2. How are requests from service A routed to service B?

Best explained with an example...

ContainerCamp San Francisco 2015

@lukeb0nd

http://yld.io

ContainerCamp San Francisco 2015

Service Discovery

Requirements:

• "web" containers need to talk to "api" containers
• "api" containers need to talk to "db" containers
• Each service takes URL of other service as envvars
• We can't assume service is on the same host
• Ports are random
• Multiple instances of each service type

@lukeb0nd

http://yld.io

ContainerCamp San Francisco 2015

Service Discovery

Sounds like a job for DNS, right?

Traditional* DNS has a few limitations:

• Containers don't have a unique IP
  • ∴ SRV records needed
• TTL issues
  • Internally we could get around this
  • But not externally

@lukeb0nd

http://yld.io

ContainerCamp San Francisco 2015

Service Discovery

IMHO two options:

1. A non-traditional DNS solution (e.g. WeaveDNS) with round-robining
2. A load-balancing dynamic proxy

The latter is what Paz does, but I'll talk about both.

@lukeb0nd

http://yld.io

ContainerCamp San Francisco 2015

Service Discovery - WeaveDNS

• Weave is a SDN
• Each container gets its own IP address
• So DNS is usable again without SRV records

• WeaveDNS is DNS for addresses on a Weave network
• You can register multiple IPs against a name

@lukeb0nd

http://yld.io

ContainerCamp San Francisco 2015

Service Discovery - Dynamic Proxy

Logical View

• This is the "ambassador pattern"
• Part proxy, part dynamic load-balancer

• You could apply this logical model to various protocols
• e.g. HTTP requests proxied by host header

@lukeb0nd

http://yld.io

ContainerCamp San Francisco 2015

Service Discovery - Dynamic Proxy

• All containers' DNS point to Dnsmasq on same host
• All *.mydomain forwarded to HAProxy
• All else to the Internet

@lukeb0nd

http://yld.io

ContainerCamp San Francisco 2015

Service Discovery - Dynamic Proxy

• All running on CoreOS, using Fleet and hence systemd
• "Sidekick" announce units using systemd BindsTo directive
• Writes IP:port for service (from docker inspect) to Etcd
• Writes with TTL, sleeps, loop
• Will stop when service stops

@lukeb0nd

http://yld.io

ContainerCamp San Francisco 2015

Service Discovery - Dynamic Proxy

frontend http-in
    bind *:80
    acl subdom_myservice hdr(host) -i myservice.paz hdr(host) -i myservice.mydomain.com
    use_backend backend-myservice if subdom_myservice
    acl subdom_paz-web hdr(host) -i paz-web.paz hdr(host) -i paz-web.mydomain.com
    use_backend backend-paz-web if subdom_paz-web

backend backend-myservice
    balance roundrobin
    server myservice-v1-1 10.131.238.241:49177
    server myservice-v1-2 10.131.238.242:49165

backend backend-paz-web
    server paz-web 10.131.238.233:49164

HAProxy config example excerpt:

This gets written by Confd.

Confd watches Etcd directories and renders values against a template into the HAProxy config file.

@lukeb0nd

luke@yld.io

Docker Multi-host Networking

• Docker has been criticised for multi-host networking being hard in Docker
• IMHO this is unfair
  • This is what happens when you make things easier
• Also criticised for providing their own solution
  • ie. Docker Swarm
• Also unfair

ContainerCamp San Francisco 2015

@lukeb0nd

luke@yld.io

Difficult Doesn't Mean Bad

• Multi-host LXC networking is complex
  • That's not Docker's fault
• Docker (pre-Swarm/Compose) is a low-level building block for a platform
• My issues with Docker lie elsewhere...

ContainerCamp San Francisco 2015

@lukeb0nd

luke@yld.io

The Docker Daemon

• A PaaS can't start/stop Docker containers
  • It asks the Docker daemon to do so
• My issue with Docker is the lack of process control and composability
• The daemon/platform is in control not me or a platform I build
• I love Docker but this is something about appc (rkt) that I prefer

ContainerCamp San Francisco 2015

@lukeb0nd

luke@yld.io

Summary

• A PaaS designed for 12-factor apps needs to deal with some complex realities
• A good Docker PaaS should be fully dynamic
• Service Discovery IMHO needs to be either WeaveDNS or a dynamically-configured load-balancing proxy
• Docker networking can be difficult
• Docker networking is not a problem
• Docker is awesome & production-ready

ContainerCamp San Francisco 2015