Building a Docker PaaS

A Docker networking case study

@lukeb0nd

http://yld.io

ContainerCamp 2015

San Francisco

@lukeb0nd

http://yld.io

ContainerCamp San Francisco 2015

In the next 25 mins...

  • Problems in building a Docker PaaS

  • Deep dive into service discovery, using Paz as a reference point

  • Along the way discuss the Docker ecosystem, and Docker networking in particular

@lukeb0nd

http://yld.io

ContainerCamp San Francisco 2015

Who am I?

  • I'm a server developer at YLD.io
    • Software engineering consultency
    • We do mostly Node.js and Docker
  • Built a Docker PaaS called "Paz"

@lukeb0nd

luke@yld.io

What's in a Docker PaaS?

Some problems that may need solving in a Docker PaaS:

  • Workflow

  • Scheduling

  • Service Discovery

    • Routing

  • DNS

  • Zero-downtime deployments

ContainerCamp San Francisco 2015

@lukeb0nd

luke@yld.io

What is Service Discovery?

Service A needs to speak to service B.

  1. How does service A know how to address service B?
  2. How are requests from service A routed to service B?

 

Best explained with an example...

ContainerCamp San Francisco 2015

@lukeb0nd

http://yld.io

ContainerCamp San Francisco 2015

Service Discovery

Requirements:

  • "web" containers need to talk to "api" containers
  • "api" containers need to talk to "db" containers
  • Each service takes URL of other service as envvars
  • We can't assume service is on the same host
  • Ports are random
  • Multiple instances of each service type

@lukeb0nd

http://yld.io

ContainerCamp San Francisco 2015

Service Discovery

Sounds like a job for DNS, right?

 

Traditional* DNS has a few limitations:

  • Containers don't have a unique IP
    • ∴ SRV records needed
  • TTL issues
    • Internally we could get around this
    • But not externally

@lukeb0nd

http://yld.io

ContainerCamp San Francisco 2015

Service Discovery

IMHO two options:

  1. A non-traditional DNS solution (e.g. WeaveDNS) with round-robining
  2. A load-balancing dynamic proxy

 

The latter is what Paz does, but I'll talk about both.

@lukeb0nd

http://yld.io

ContainerCamp San Francisco 2015

Service Discovery - WeaveDNS

  • Weave is a SDN
  • Each container gets its own IP address
  • So DNS is usable again without SRV records
  • WeaveDNS is DNS for addresses on a Weave network
  • You can register multiple IPs against a name

@lukeb0nd

http://yld.io

ContainerCamp San Francisco 2015

Service Discovery - Dynamic Proxy

Logical View

  • This is the "ambassador pattern"
  • Part proxy, part dynamic load-balancer
  • You could apply this logical model to various protocols
  • e.g. HTTP requests proxied by host header

@lukeb0nd

http://yld.io

ContainerCamp San Francisco 2015

Service Discovery - Dynamic Proxy

  • All containers' DNS point to Dnsmasq on same host
  • All *.mydomain forwarded to HAProxy
  • All else to the Internet

@lukeb0nd

http://yld.io

ContainerCamp San Francisco 2015

Service Discovery - Dynamic Proxy

  • All running on CoreOS, using Fleet and hence systemd
  • "Sidekick" announce units using systemd BindsTo directive
  • Writes IP:port for service (from docker inspect) to Etcd
  • Writes with TTL, sleeps, loop
  • Will stop when service stops

@lukeb0nd

http://yld.io

ContainerCamp San Francisco 2015

Service Discovery - Dynamic Proxy 

frontend http-in
    bind *:80
    acl subdom_myservice hdr(host) -i myservice.paz hdr(host) -i myservice.mydomain.com
    use_backend backend-myservice if subdom_myservice
    acl subdom_paz-web hdr(host) -i paz-web.paz hdr(host) -i paz-web.mydomain.com
    use_backend backend-paz-web if subdom_paz-web

backend backend-myservice
    balance roundrobin
    server myservice-v1-1 10.131.238.241:49177
    server myservice-v1-2 10.131.238.242:49165

backend backend-paz-web
    server paz-web 10.131.238.233:49164

HAProxy config example excerpt:

This gets written by Confd.

Confd watches Etcd directories and renders values against a template into the HAProxy config file.

@lukeb0nd

luke@yld.io

Docker Multi-host Networking

  • Docker has been criticised for multi-host networking being hard in Docker
  • IMHO this is unfair
    • This is what happens when you make things easier
  • Also criticised for providing their own solution
    • ie. Docker Swarm
  • Also unfair

ContainerCamp San Francisco 2015

@lukeb0nd

luke@yld.io

Difficult Doesn't Mean Bad

  • Multi-host LXC networking is complex
    • That's not Docker's fault
  • Docker (pre-Swarm/Compose) is a low-level building block for a platform
  • My issues with Docker lie elsewhere...

ContainerCamp San Francisco 2015

@lukeb0nd

luke@yld.io

The Docker Daemon

  • A PaaS can't start/stop Docker containers
    • It asks the Docker daemon to do so
  • My issue with Docker is the lack of process control and composability
  • The daemon/platform is in control not me or a platform I build
  • I love Docker but this is something about appc (rkt) that I prefer

ContainerCamp San Francisco 2015

@lukeb0nd

luke@yld.io

Summary

  • A PaaS designed for 12-factor apps needs to deal with some complex realities
  • A good Docker PaaS should be fully dynamic
  • Service Discovery IMHO needs to be either WeaveDNS or a dynamically-configured load-balancing proxy
  • Docker networking can be difficult
  • Docker networking is not a problem
  • Docker is awesome & production-ready

ContainerCamp San Francisco 2015

Made with Slides.com