Docker Swarm

Ron Kurr

September 2016

What

• Docker 1.12.0's embedded Swarm, aka Swarm mode
• What is Swarm?
• Why use Swarm?
• How to use Swarm?

WhY

• Docker, Inc. is competing in the same space as Kubernetes, Mesos, CoreOS and Rancher
• Some rumblings in the community about Docker, Inc. creating breaking changes, making it harder for others to build businesses on top of Docker
• Using Docker, Inc. only tools might avoid issues if a fork of the Docker source code happens
• Let's see how far we can get with a Docker, Inc. only stack

WhY

Discussions about a split from Docker are now underway among several Docker ecosystem vendors and end users. Expressing frustration of Docker’s management of Docker Engine, the technologists with the companies are exploring ways to address various issues around supporting enterprise Docker deployments.

 

Several options are under consideration, including the possibility of forking the open source Docker engine altogether. According to a number of sources close to recent discussions, representatives are involved from companies such Red Hat, Google, CoreOS, Huawei and two large end-user customers.

http://thenewstack.io/docker-fork-talk-split-now-table/

how

• Use Vagrant to set up a simulated environment
• Available at github.com/kurron/docker-full-stack
• Use Docker Machine to install Docker Engine
• Use Docker Swarm to:

• install global services
• install load balanced services
• install "pinned" services
• perform rolling upgrades
• manage service discovery
• manage load balancing

environment

• Ubuntu 16.04
• alpha - console machine where all command line operations occur
• bravo - Swarm Manager
• charlie, delta, echo - Swam workers
• all boxes on a private network set up by Vagrant
• forces us to open up ports when exposing services to the public

Docker Machine

Docker Machine is a tool that lets you install Docker Engine on virtual hosts, and manage the hosts with the docker-machine command.

• Amazon Web Services
• Digital Ocean
• Google Compute Engine
• Microsoft Hyper-V
• Rackspace
• Oracle VirtualBox
• VMware Fusion

• Microsoft Azure
• Exoscale
• Generic (SSH)
• OpenStack
• IBM Softlayer
• VMware vCloud Air
• VMware vSphere

machine filesystem

df -Th
Filesystem     Type      Size  Used Avail Use% Mounted on
udev           devtmpfs  492M     0  492M   0% /dev
tmpfs          tmpfs     100M  3.1M   97M   4% /run
/dev/sda5      ext4       30G  1.5G   27G   6% /
tmpfs          tmpfs     497M     0  497M   0% /dev/shm
tmpfs          tmpfs     5.0M     0  5.0M   0% /run/lock
tmpfs          tmpfs     497M     0  497M   0% /sys/fs/cgroup
/dev/sda1      ext4      232M   44M  172M  21% /boot
/dev/sda6      xfs        50G   33M   50G   1% /opt/docker  <====== XFS to avoid inode issues
vagrant        vboxsf    233G  132G  101G  57% /vagrant

install Docker Engine

CMD="docker-machine create --driver generic \
                           --generic-ip-address=10.10.10.20 \
                           --generic-ssh-key ${BRAVO_SECURE_KEY} \
                           --engine-storage-driver overlay \ <==== engine configuration switch
                           --engine-opt graph=/opt/docker \ <==== engine configuration switch
                           --engine-label size=small \ <==== arbitrary label
                           --engine-label role=manager \ <==== arbitrary label
                           bravo"

CMD="docker-machine create --driver generic \
                           --generic-ip-address=${IP} \
                           --generic-ssh-key ${SECURE_KEY} \
                           --engine-storage-driver overlay \ <==== engine configuration switch
                           --engine-opt graph=/opt/docker \ <==== engine configuration switch
                           --engine-label size=medium \ <==== arbitrary label
                           --engine-label role=worker \ <==== arbitrary label
                           ${NAME}"

• docker-machine version 0.8.1, build 41b3b25
• can handle differences between Ubuntu releases

install results

docker-machine ls

NAME      ACTIVE   DRIVER    STATE     URL                      SWARM   DOCKER    ERRORS
alpha     -        generic   Running   tcp://10.10.10.10:2376           v1.12.1
bravo     -        generic   Running   tcp://10.10.10.20:2376           v1.12.1
charlie   -        generic   Running   tcp://10.10.10.30:2376           v1.12.1
delta     -        generic   Running   tcp://10.10.10.40:2376           v1.12.1
echo      -        generic   Running   tcp://10.10.10.50:2376           v1.12.1

• do not destroy the Docker Machine box
• it has local knowledge about where it installed engines
• useful when upgrading a fleet of boxes
• does not add accounts to the docker group so you have to do it by hand or use sudo

machine Commands

docker-machine --help

[[ edited out for space ]]

Commands:
  active                Print which machine is active
  config                Print the connection config for machine
  create                Create a machine
  env                   Display the commands to set up the environment for the Docker client
  inspect               Inspect information about a machine
  ip                    Get the IP address of a machine
  kill                  Kill a machine
  ls                    List machines
  provision             Re-provision existing machines
  regenerate-certs      Regenerate TLS Certificates for a machine
  restart               Restart a machine
  rm                    Remove a machine
  ssh                   Log into or run a command on a machine with SSH.
  scp                   Copy files between machines
  start                 Start a machine
  status                Get the status of a machine
  stop                  Stop a machine
  upgrade               Upgrade a machine to the latest version of Docker
  url                   Get the URL of a machine
  version               Show the Docker Machine version or a machine docker version
  help                  Shows a list of commands or help for one command

Docker Swarm

A swarm is a cluster of Docker Engines where you deploy services.  The Docker Engine CLI includes the commands for swarm management, such as adding and removing nodes.  The CLI also includes the commands you need to deploy services to the swarm and manage service orchestration.  A node is an instance of the Docker Engine participating in the swarm.

• legacy Swarm used containers to manage things
• embedded Swarm is baked into the docker command
• the two forms appear to be incompatible

Docker Swarm

• cluster management integrated with Docker Engine
• decentralized design
• declarative service model
• scaling
• desired state reconciliation
• multi-host networking
• service discovery
• load balancing
• secure by default
• rolling updates

Docker Swarm

Docker Swarm

Docker Swarm

Docker Swarm

Docker Swarm

# inspect a swarm node
docker-machine ssh bravo sudo docker node inspect echo

[
    {
        "ID": "awi2o2zrso99aj4kxoodas95u",
        "Version": {
            "Index": 126
        },
        "CreatedAt": "2016-08-31T16:25:44.238584068Z",
        "UpdatedAt": "2016-08-31T19:00:21.329507138Z",
        "Spec": {
            "Role": "worker",
            "Availability": "active"
        },
        "Description": {
            "Hostname": "echo",
            "Platform": {
                "Architecture": "x86_64",
                "OS": "linux"
            },
            "Resources": {
                "NanoCPUs": 1000000000,
                "MemoryBytes": 1040699392
            },
            "Engine": {
                "EngineVersion": "1.12.1",
                "Labels": {
                    "provider": "generic",
                    "role": "worker",
                    "size": "medium"
                },
                "Plugins": [
                    {
                        "Type": "Network",
                        "Name": "bridge"
                    },
                    {
                        "Type": "Network",
                        "Name": "host"
                    },
                    {
                        "Type": "Network",
                        "Name": "null"
                    },
                    {
                        "Type": "Network",
                        "Name": "overlay"
                    },
                    {
                        "Type": "Volume",
                        "Name": "local"
                    }
                ]
            }
        },
        "Status": {
            "State": "ready"
        }
    }
]

Creating A Swarm

# we are on alpha and running commands on bravo via Docker Machine

# create a new swarm with bravo as the manager
docker-machine ssh bravo sudo docker swarm init --advertise-addr 10.10.10.20

# capture the manager and worker tokens that are specific to this swarm
MANAGER_TOKEN=$(docker-machine ssh bravo sudo docker swarm join-token --quiet manager)
WORKER_TOKEN=$(docker-machine ssh bravo sudo docker swarm join-token --quiet worker)

# join the remaining nodes to the swarm as workers
docker-machine ssh charlie sudo docker swarm join --token $WORKER_TOKEN 10.10.10.20:2377
docker-machine ssh delta sudo docker swarm join --token $WORKER_TOKEN 10.10.10.20:2377
docker-machine ssh echo sudo docker swarm join --token $WORKER_TOKEN 10.10.10.20:2377

docker-machine ssh bravo sudo docker node ls

ID                           HOSTNAME  STATUS  AVAILABILITY  MANAGER STATUS
02zjanuyc0fqxg5rlqdj5imxg    delta     Ready   Active
1rrhfx76hlh09y6lh09kvgvmr *  bravo     Ready   Active        Leader
39t9mf1mj9h1nsrba7b7iud0s    charlie   Ready   Active
awi2o2zrso99aj4kxoodas95u    echo      Ready   Active

overlay network

• aka multi-host networking
• based on Virtual Extensible LAN (VXLAN)
• each container gets its own private ip address
• only routable to containers in the same network
• unmanged containers cannot see any nodes on the network
• good bye port collisions
• containers can participate in multiple networks
• network is created on the swarm manager and automatically made available to all swarm workers
• legacy multi-host networking required external consensus server such as Consul or etcd

overlay network

# create the networked named showcase on a randomly selected network segment
docker-machine ssh bravo sudo docker network create --driver overlay showcase

477x0fr2gbdmp7xu1wzebyqlz


# list existing networks
docker-machine ssh bravo sudo docker network ls

NETWORK ID          NAME                DRIVER              SCOPE
2d205b99d138        bridge              bridge              local
4434086683ad        docker_gwbridge     bridge              local
1103400b1f3e        host                host                local
c9p8v2mo5uc5        ingress             overlay             swarm
debb3f27bc6d        none                null                local
477x0fr2gbdm        showcase            overlay             swarm

overlay network

# examine the showcase network
docker-machine ssh bravo sudo docker network inspect showcase

[
    {
        "Name": "showcase",
        "Id": "477x0fr2gbdmp7xu1wzebyqlz",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": []
        },
        "Internal": false,
        "Containers": null,
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "257"
        },
        "Labels": null
    }
]

network COMMANDS

docker-machine ssh bravo sudo docker network --help

Usage:  docker network COMMAND

Manage Docker networks

Options:
      --help   Print usage

Commands:
  connect     Connect a container to a network
  create      Create a network
  disconnect  Disconnect a container from a network
  inspect     Display detailed information on one or more networks
  ls          List networks
  rm          Remove one or more networks

Run 'docker network COMMAND --help' for more information on a command.

network COMMANDS

docker-machine ssh bravo sudo docker network create --help

Usage:  docker network create [OPTIONS] NETWORK

Create a network

Options:
      --aux-address value    Auxiliary IPv4 or IPv6 addresses used by Network driver (default map[])
  -d, --driver string        Driver to manage the Network (default "bridge")
      --gateway value        IPv4 or IPv6 Gateway for the master subnet (default [])
      --help                 Print usage
      --internal             Restrict external access to the network
      --ip-range value       Allocate container ip from a sub-range (default [])
      --ipam-driver string   IP Address Management Driver (default "default")
      --ipam-opt value       Set IPAM driver specific options (default map[])
      --ipv6                 Enable IPv6 networking
      --label value          Set metadata on a network (default [])
  -o, --opt value            Set driver specific options (default map[])
      --subnet value         Subnet in CIDR format that represents a network segment (default [])

GLOBAL Service

# deploy a global service
docker-machine ssh bravo sudo docker service create --mode global 
                                                    --name hello-global 
                                                    alpine ping docker.com

# inspect the service
docker-machine ssh bravo sudo docker service inspect --pretty hello-global

ID:             6e86pve42a7yd764lkbaxq4h2
Name:           hello-global
Mode:           Global
Placement:
UpdateConfig:
 Parallelism:   1
 On failure:    pause
ContainerSpec:
 Image:         alpine
 Args:          ping docker.com
Resources:

GLOBAL Service

# list the logical status of the global service
docker-machine ssh bravo sudo docker service ls

ID            NAME          REPLICAS  IMAGE   COMMAND
6e86pve42a7y  hello-global  global    alpine  ping docker.com

# list where the service is running
docker-machine ssh bravo sudo docker service ps hello-global

ID                         NAME              IMAGE   NODE     DESIRED STATE  CURRENT STATE        
04xswl9iwxv3p0x8k2ewwruun  hello-global      alpine  echo     Running        Running 3 minutes ago
804eybfl1awo8g0n511hgv8co   \_ hello-global  alpine  delta    Running        Running 3 minutes ago
cklpu46qf1wrajt5l0teuzulf   \_ hello-global  alpine  charlie  Running        Running 3 minutes ago
10u64dg0z76l8aj8gr0q05i7s   \_ hello-global  alpine  bravo    Running        Running 3 minutes ago

• global services run on every node in the swarm
• Consul or Data Dog are examples of something you might want to run as a global service

replicated Service

# deploy a replicated service
docker-machine ssh bravo sudo docker service create --mode replicated 
                                                    --replicas=2 
                                                    --name hello-replicated 
                                                    alpine ping docker.com

# inspect the service
docker-machine ssh bravo sudo docker service inspect --pretty hello-replicated

ID:             7b41bx4f2ueqa5f9ztxzgs4no
Name:           hello-replicated
Mode:           Replicated
 Replicas:      2
Placement:
UpdateConfig:
 Parallelism:   1
 On failure:    pause
ContainerSpec:
 Image:         alpine
 Args:          ping docker.com
Resources:

replicated Service

# list the logical status of the replicated service
docker-machine ssh bravo sudo docker service ls

ID            NAME              REPLICAS  IMAGE   COMMAND
6e86pve42a7y  hello-global      global    alpine  ping docker.com
7b41bx4f2ueq  hello-replicated  2/2       alpine  ping docker.com

# list where the service is running
docker-machine ssh bravo sudo docker service ps hello-replicated

ID                         NAME                IMAGE   NODE   DESIRED STATE  CURRENT STATE        
dkat3ntbupfl9z68lmaj2eyv8  hello-replicated.1  alpine  echo   Running        Running 3 minutes ago
36zo00ycklszgoe977zubmoep  hello-replicated.2  alpine  bravo  Running        Running 3 minutes ago

• replicated services can run on any node
• only as many instances as desired
• failed containers will be replaced as needed

constrained Service

# Deploy a constrained service
docker-machine ssh bravo sudo docker service create --mode replicated \
                                                    --replicas=3 \
                                                    --name hello-constrained \
                                                    --constraint 'node.role==worker' \
                                                    alpine ping docker.com
dw86g0jgh7jh2tktizqok1l68

# inspect the service
docker-machine ssh bravo sudo docker service inspect --pretty hello-constrained

ID:             dw86g0jgh7jh2tktizqok1l68
Name:           hello-constrained
Mode:           Replicated
 Replicas:      3
Placement:
 Constraints    : node.role==worker
UpdateConfig:
 Parallelism:   1
 On failure:    pause
ContainerSpec:
 Image:         alpine
 Args:          ping docker.com
Resources:

constrained Service

# list the logical status of the constrained service
docker-machine ssh bravo sudo docker service ls

ID            NAME               REPLICAS  IMAGE   COMMAND
6e86pve42a7y  hello-global       global    alpine  ping docker.com
7b41bx4f2ueq  hello-replicated   2/2       alpine  ping docker.com
dw86g0jgh7jh  hello-constrained  3/3       alpine  ping docker.com

# list where the service is running
docker-machine ssh bravo sudo docker service ps hello-constrained

ID                         NAME                 IMAGE   NODE     DESIRED STATE  CURRENT STATE        
b14avy9hcwy8pft3roi36jhs9  hello-constrained.1  alpine  charlie  Running        Running 4 minutes ago
5zhc0ab8rht63v4iqcpcbkuz0  hello-constrained.2  alpine  delta    Running        Running 4 minutes ago
5q39tbwovu0a9lxy09al7iqr4  hello-constrained.3  alpine  charlie  Running        Running 4 minutes ago

• only deployed to nodes that match the restriction
• can apply simple boolean expressions

scaling down

# scale down the constrained service
docker-machine ssh bravo sudo docker service scale hello-constrained=1

hello-constrained scaled to 1

# inspect the service
docker-machine ssh bravo sudo docker service inspect --pretty hello-constrained

ID:             dw86g0jgh7jh2tktizqok1l68
Name:           hello-constrained
Mode:           Replicated
 Replicas:      1
Placement:
 Constraints    : node.role==worker
UpdateConfig:
 Parallelism:   1
 On failure:    pause
ContainerSpec:
 Image:         alpine
 Args:          ping docker.com
Resources:

scaling down

# list the logical status of the constrained service
docker-machine ssh bravo sudo docker service ls

ID            NAME               REPLICAS  IMAGE   COMMAND
6e86pve42a7y  hello-global       global    alpine  ping docker.com
7b41bx4f2ueq  hello-replicated   2/2       alpine  ping docker.com
dw86g0jgh7jh  hello-constrained  1/1       alpine  ping docker.com

# monitor the transition
docker-machine ssh bravo sudo docker service ps hello-constrained

ID                         NAME                 IMAGE   NODE     DESIRED STATE  CURRENT STATE         
b14avy9hcwy8pft3roi36jhs9  hello-constrained.1  alpine  charlie  Shutdown       Shutdown 2 minutes ago
5zhc0ab8rht63v4iqcpcbkuz0  hello-constrained.2  alpine  delta    Shutdown       Shutdown 2 minutes ago
5q39tbwovu0a9lxy09al7iqr4  hello-constrained.3  alpine  charlie  Running        Running 10 minutes ago

service removal

# removing the global service
docker-machine ssh bravo sudo docker service rm hello-global

hello-global

# list the logical status of the global service
docker-machine ssh bravo sudo docker service ls

ID            NAME               REPLICAS  IMAGE   COMMAND
7b41bx4f2ueq  hello-replicated   2/2       alpine  ping docker.com
dw86g0jgh7jh  hello-constrained  1/1       alpine  ping docker.com

rolling update

# install an "old" version of the container
docker-machine ssh bravo sudo docker service create --mode replicated \
                                                    --replicas 4 \
                                                    --name redis \
                                                    --update-delay 10s \
                                                    --network showcase \
                                                    redis:3.0.6
bgv25hgzz40dkxcwzsz6752o8

# inspect the service
docker-machine ssh bravo sudo docker service inspect --pretty redis

ID:             bgv25hgzz40dkxcwzsz6752o8
Name:           redis
Mode:           Replicated
 Replicas:      4
Placement:
UpdateConfig:
 Parallelism:   1
 Delay:         10s
 On failure:    pause
ContainerSpec:
 Image:         redis:3.0.6
Resources:

rolling update

# list the logical status of the replicated service
docker-machine ssh bravo sudo docker service ls

ID            NAME               REPLICAS  IMAGE        COMMAND
7b41bx4f2ueq  hello-replicated   2/2       alpine       ping docker.com
bgv25hgzz40d  redis              0/4       redis:3.0.6
dw86g0jgh7jh  hello-constrained  1/1       alpine       ping docker.com

# list where the service is running
docker-machine ssh bravo sudo docker service ps redis

ID                         NAME     IMAGE        NODE     DESIRED STATE  CURRENT STATE          ERROR
0uwkk79qrr7x2x079sgzixtzp  redis.1  redis:3.0.6  delta    Running        Running 2 minutes ago
5lft6lmre9iqidplf0vrthyc7  redis.2  redis:3.0.6  delta    Running        Running 2 minutes ago
34o0xfsjk6z17kdytq5uqofgi  redis.3  redis:3.0.6  charlie  Running        Running 2 minutes ago
arjitimy3n9gumhneissek58p  redis.4  redis:3.0.6  echo     Running        Running 2 minutes ago

• 4 copies of Redis 3.0.6 deployed
• delta is running 2 of them

rolling update

# upgrade to a newer version of the container
docker-machine ssh bravo sudo docker service update --image redis:3.0.7 redis

redis

# inspect the service
docker-machine ssh bravo sudo docker service inspect --pretty redis

ID:             bgv25hgzz40dkxcwzsz6752o8
Name:           redis
Mode:           Replicated
 Replicas:      4
Update status:
 State:         updating
 Started:       less than a second ago
 Message:       update in progress
Placement:
UpdateConfig:
 Parallelism:   1
 Delay:         10s
 On failure:    pause
ContainerSpec:
 Image:         redis:3.0.7
Resources:

rolling update

maintenance mode

# put echo into maintenance mode
docker-machine ssh bravo sudo docker node update --availability drain delta

echo

# inspect the node
docker-machine ssh bravo sudo docker node inspect --pretty delta

ID:                     02zjanuyc0fqxg5rlqdj5imxg
Hostname:               delta
Joined at:              2016-08-31 16:25:44.042049859 +0000 utc
Status:
 State:                 Ready
 Availability:          Drain
Platform:
 Operating System:      linux
 Architecture:          x86_64
Resources:
 CPUs:                  1
 Memory:                992.5 MiB
Plugins:
  Network:              bridge, host, null, overlay
  Volume:               local
Engine Version:         1.12.1
Engine Labels:
 - provider = generic - role = worker - size = medium

maintenance mode

# inspect the status of the Swarm
docker-machine ssh bravo sudo docker node ls

ID                           HOSTNAME  STATUS  AVAILABILITY  MANAGER STATUS
02zjanuyc0fqxg5rlqdj5imxg    delta     Ready   Drain
1rrhfx76hlh09y6lh09kvgvmr *  bravo     Ready   Active        Leader
39t9mf1mj9h1nsrba7b7iud0s    charlie   Ready   Active
awi2o2zrso99aj4kxoodas95u    echo      Ready   Active

# monitor the transition between versions
docker-machine ssh bravo sudo docker service ps redis

ID                         NAME         IMAGE        NODE     DESIRED STATE  CURRENT STATE          
8amieyaaw80pjys34xpwvy2ml  redis.1      redis:3.0.7  bravo    Running        Running 18 minutes ago
0uwkk79qrr7x2x079sgzixtzp   \_ redis.1  redis:3.0.6  delta    Shutdown       Shutdown 18 minutes ago
bd43cpi0elyczxi2tjuyy01oh  redis.2      redis:3.0.7  charlie  Running        Running 17 minutes ago
5lft6lmre9iqidplf0vrthyc7   \_ redis.2  redis:3.0.6  delta    Shutdown       Shutdown 17 minutes ago
bzy8prouu0yvcgbylqrp06ngr  redis.3      redis:3.0.7  echo     Running        Running about a minute ago
bvqbnuuazqh6myyrebz7lv8pp   \_ redis.3  redis:3.0.7  delta    Shutdown       Shutdown 2 minutes ago
34o0xfsjk6z17kdytq5uqofgi   \_ redis.3  redis:3.0.6  charlie  Shutdown       Shutdown 18 minutes ago
4oocd1d0acrneq5nwy7hn959x  redis.4      redis:3.0.7  echo     Running        Running about a minute ago
040rx6mhtt6pctfzle4he7oiy   \_ redis.4  redis:3.0.7  delta    Shutdown       Shutdown 2 minutes ago
arjitimy3n9gumhneissek58p   \_ redis.4  redis:3.0.6  echo     Shutdown       Shutdown 17 minutes ago

workload has shifted from delta to echo

maintenance mode

# put delta into active mode
docker-machine ssh bravo sudo docker node update --availability active echo

delta

# inspect the node
docker-machine ssh bravo sudo docker node inspect --pretty echo

ID:                     02zjanuyc0fqxg5rlqdj5imxg
Hostname:               delta
Joined at:              2016-08-31 16:25:44.042049859 +0000 utc
Status:
 State:                 Ready
 Availability:          Active
Platform:
 Operating System:      linux
 Architecture:          x86_64
Resources:
 CPUs:                  1
 Memory:                992.5 MiB
Plugins:
  Network:              bridge, host, null, overlay
  Volume:               local
Engine Version:         1.12.1
Engine Labels:
 - size = medium - provider = generic - role = workerInspect the status of the Swarm

maintenance mode

# inspect the status of the Swarm
docker-machine ssh bravo sudo docker node ls

ID                           HOSTNAME  STATUS  AVAILABILITY  MANAGER STATUS
02zjanuyc0fqxg5rlqdj5imxg    delta     Ready   Active
1rrhfx76hlh09y6lh09kvgvmr *  bravo     Ready   Active        Leader
39t9mf1mj9h1nsrba7b7iud0s    charlie   Ready   Active
awi2o2zrso99aj4kxoodas95u    echo      Ready   Active

# see where the containers are deployed
docker-machine ssh bravo sudo docker service ps redis

ID                         NAME         IMAGE        NODE     DESIRED STATE  CURRENT STATE   
8amieyaaw80pjys34xpwvy2ml  redis.1      redis:3.0.7  bravo    Running        Running 22 minutes ago
0uwkk79qrr7x2x079sgzixtzp   \_ redis.1  redis:3.0.6  delta    Shutdown       Shutdown 22 minutes ago
bd43cpi0elyczxi2tjuyy01oh  redis.2      redis:3.0.7  charlie  Running        Running 21 minutes ago
5lft6lmre9iqidplf0vrthyc7   \_ redis.2  redis:3.0.6  delta    Shutdown       Shutdown 21 minutes ago
bzy8prouu0yvcgbylqrp06ngr  redis.3      redis:3.0.7  echo     Running        Running 6 minutes ago
bvqbnuuazqh6myyrebz7lv8pp   \_ redis.3  redis:3.0.7  delta    Shutdown       Shutdown 6 minutes ago
34o0xfsjk6z17kdytq5uqofgi   \_ redis.3  redis:3.0.6  charlie  Shutdown       Shutdown 22 minutes ago
4oocd1d0acrneq5nwy7hn959x  redis.4      redis:3.0.7  echo     Running        Running 6 minutes ago
040rx6mhtt6pctfzle4he7oiy   \_ redis.4  redis:3.0.7  delta    Shutdown       Shutdown 6 minutes ago
arjitimy3n9gumhneissek58p   \_ redis.4  redis:3.0.6  echo     Shutdown       Shutdown 21 minutes ago

workload has not shifted back to delta

service discovery

service discovery

# install Nginx
docker-machine ssh bravo sudo docker service create --mode replicated \
                                                    --replicas 3 \
                                                    --name nginx \
                                                    --update-delay 10s \
                                                    --network showcase \
                                                    --endpoint-mode dnsrr \ <===== enable DNS round-robin
                                                    nginx:latest

8hxho7lkeodd7w9959w2ecn9j

# inspect the service
docker-machine ssh bravo sudo docker service inspect --pretty nginx

ID:             8hxho7lkeodd7w9959w2ecn9j
Name:           nginx
Mode:           Replicated
 Replicas:      3
Placement:
UpdateConfig:
 Parallelism:   1
 Delay:         10s
 On failure:    pause
ContainerSpec:
 Image:         nginx:latest
Resources:
Networks: 477x0fr2gbdmp7xu1wzebyqlz

service discovery

# list the logical status of the replicated service
docker-machine ssh bravo sudo docker service ls

ID            NAME   REPLICAS  IMAGE         COMMAND
bpu620z6ukt5  nginx  3/3       nginx:latest

# list where the service is running
docker-machine ssh bravo sudo docker service ps nginx

ID                         NAME     IMAGE         NODE     DESIRED STATE  CURRENT STATE     
687kp8xeedxeyt8sfxz8r5dif  nginx.1  nginx:latest  bravo    Running        Running 58 seconds ago
6ip4lj89g2kc3orrs7vyuowr5  nginx.2  nginx:latest  echo     Running        Running about a minute ago
8ib2xtanceioqwyxp95znmx8i  nginx.3  nginx:latest  charlie  Running        Running 58 seconds ago

bravo, charlie and echo are all running instances of the nginx service

service discovery

# install Busybox
docker-machine ssh bravo sudo docker service create --mode replicated \
                                                    --replicas 1 \
                                                    --name busybox \
                                                    --update-delay 10s \
                                                    --network showcase \ <===== on the same network
                                                    busybox:latest sleep 3000

1dk4k1pbhzrtjci1j350vane5

# inspect the service
docker-machine ssh bravo sudo docker service inspect --pretty busybox

ID:             1dk4k1pbhzrtjci1j350vane5
Name:           busybox
Mode:           Replicated
 Replicas:      1
Placement:
UpdateConfig:
 Parallelism:   1
 Delay:         10s
 On failure:    pause
ContainerSpec:
 Image:         busybox:latest
 Args:          sleep 3000
Resources:
Networks: 477x0fr2gbdmp7xu1wzebyqlz

service discovery

# list the logical status of the replicated service
docker-machine ssh bravo sudo docker service ls

ID            NAME     REPLICAS  IMAGE           COMMAND
1dk4k1pbhzrt  busybox  1/1       busybox:latest  sleep 3000
bpu620z6ukt5  nginx    3/3       nginx:latest

# list where the service is running
docker-machine ssh bravo sudo docker service ps busybox

ID                         NAME       IMAGE           NODE   DESIRED STATE  CURRENT STATE
3qhul4bmaq80q23v516oztawy  busybox.1  busybox:latest  delta  Running        Running 2 minutes ago

# get the container id so we can exec into it
docker-machine ssh delta sudo docker ps

CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
f4ccf7e4afe5        busybox:latest      "sleep 3000"        15 minutes ago      Up 15 minutes                           busybox.1.3qhul4bmaq80q23v516oztawy

we have a shell container running on the same network that the nginx service is running on

service discovery

# ssh into the node running busybox
docker-machine ssh delta

# exec into the busybox container
sudo docker exec --interactive --tty f4ccf7e4afe5 /bin/sh

# get the virtual ip of a load-balanced, replicated containers
nslookup nginx-lb

Server:    127.0.0.11
Address 1: 127.0.0.11

Name:      nginx-lb
Address 1: 10.0.0.3 <== this is address of the load balancer

# can use a special form to get addresses that the balancer forwards to
nslookup tasks.nginx-lb

Server:    127.0.0.11
Address 1: 127.0.0.11

Name:      tasks.nginx-lb
Address 1: 10.0.0.7 nginx-lb.3.1255b47ax3gwz04zhi24pc2f2.showcase
Address 2: 10.0.0.6 nginx-lb.2.dzzjgwjce6s8o531q3j58uo66.showcase
Address 3: 10.0.0.4 nginx-lb.1.268tdndlkyepiu2fqfqor5gsh.showcase

service discovery

# ssh into the node running busybox
docker-machine ssh delta

# get the virtual ip of nginx service where DNS round-robin is enabled
nslookup nginx

Server:    127.0.0.11
Address 1: 127.0.0.11

Name:      nginx
Address 1: 10.0.0.10 nginx.3.1ykxwuiguqb9w7ng4ltwz6bew.showcase <=== you get container addresses,
Address 2: 10.0.0.8 nginx.1.87mcscgoi4154sb4s54jx4bed.showcase  <=== not the load balancer address
Address 3: 10.0.0.11 nginx.2.ekfaf069ev8xr7eyymmfnzz1b.showcase

service discovery

# connect to nginx
wget -O- nginx

Connecting to nginx (10.0.0.2:80)

wget -O- nginx

Connecting to nginx (10.0.0.3:80)

wget -O- nginx

Connecting to nginx (10.0.0.4:80)

wget -O- nginx

Connecting to nginx (10.0.0.2:80)

using the service name resolves to a different ip each time due to round-robin DNS

service discovery

# show containers on the bravo node that are part of the showcase network
docker-machine ssh bravo sudo docker network inspect showcase

[
    {
        "Name": "showcase",
        "Id": "477x0fr2gbdmp7xu1wzebyqlz",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.0.0.0/24",
                    "Gateway": "10.0.0.1"
                }
            ]
        },
        "Internal": false,
        "Containers": {
            "4bfa541c612601560407815deab3e649b396037537a8784adb63e5141009ea38": {
                "Name": "nginx.3.1ykxwuiguqb9w7ng4ltwz6bew",
                "EndpointID": "cc96894a7e115e960babc29f6b7aa62a7f40e725fa21d6e88bd345970ba34f5a",
                "MacAddress": "02:42:0a:00:00:0a",
                "IPv4Address": "10.0.0.10/24",
                "IPv6Address": ""
            },
            "dbeeccd92cff84555ecdef4e78f08d163b331c5e739a67472e2baede3801fb1e": {
                "Name": "nginx.2.ekfaf069ev8xr7eyymmfnzz1b",
                "EndpointID": "c28952fa3bf356086c584e9d1c81e522b401d746993b9b683b2dfbb43fe0e45a",
                "MacAddress": "02:42:0a:00:00:0b",
                "IPv4Address": "10.0.0.11/24",
                "IPv6Address": ""
            },
            "e8565babe8a6f053a32ff7e41b052955ffc86409dc5ae6577a24f0b23db9e168": {
                "Name": "nginx.1.87mcscgoi4154sb4s54jx4bed",
                "EndpointID": "363a3039c0e7e50803f0191eba040f7ddab0abe8fb1d8e3f559e548019e47dac",
                "MacAddress": "02:42:0a:00:00:08",
                "IPv4Address": "10.0.0.8/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "257"
        },
        "Labels": {}
    }
]

external access

• same port used each ip address
• automagically routes you to an active container
• round-robin load balanced

known unknowns

• health checking
• Docker Compose support
• network latency of load balancing and the SDN fabric in general
• stability of the swarm
• UI for Operations
• can we create applications that operate correctly both in a multi-host network and out?

known unkowns

• Docker files now have a place to declare how to check the health of the container
• container has a health status of either  healthy or unhealthy
• a health_status event is generated when the status changes
• the inspect command will report the current status
• exit code 0 signals a healthy container
• exit code 1 signals an unhealthy container

# enable health checking from within the container
HEALTHCHECK --interval=30s --timeout=30s --retries=3 CMD curl -f http://localhost/ || exit 1

# check the health via inspect command
sudo docker inspect --format '{{.State.Health.Status}}' nginx

# examine stdout resulting from the health check command
sudo docker inspect --format '{{range .State.Health.Log}} {{.ExitCode}} {{.Output}} {{end}}' nginx

0 Hi, I am in your container

known unkowns

# notice pon-e is marked as unhealthy
docker ps
CONTAINER ID        IMAGE                 COMMAND                  CREATED             STATUS                     PORTS                                                         NAMES
3328d927f5d8        docker_pon-e          "/sbin/my_init"          2 minutes ago       Up 2 minutes (unhealthy)   0.0.0.0:60000->8080/tcp   pon-e
1fa1966149a3        redis                 "docker-entrypoint.sh"   2 hours ago         Up 2 hours                                                                               redis

# check the status by hand
docker inspect --format '{{.State.Health.Status}}' pon-e
unhealthy

# check the health check output
docker inspect --format '{{range .State.Health.Log}} {{.ExitCode}} {{.Output}} {{end}}' pon-e
1   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current Dload  Upload   Total   Spent    Left  Speed
    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to localhost port 8080: Connection refused

known unkowns

• Docker Compose does not appear to play nice with the new embedded Swarm
• documentation still describes interacting with the legacy mode
• reached out to the community but haven't gotten a response
• does not know about global, replicated or restricted services

known unkowns

# should be able to do this, since Compose just uses the Docker Engine API
eval "$(docker-machine env charlie)"

# which is equivalent to this:
export DOCKER_TLS_VERIFY="1"
export DOCKER_HOST="tcp://10.10.10.30:2376"
export DOCKER_CERT_PATH="/home/vagrant/.docker/machine/machines/charlie"
export DOCKER_MACHINE_NAME="charlie"
# Run this command to configure your shell:
# eval $(docker-machine env charlie)

# you get this when you attempt to run Compose
docker-compose up
WARNING: The Docker Engine you're using is running in swarm mode.

Compose does not use swarm mode to deploy services to multiple nodes in a swarm. All containers will be scheduled on the current node.

To deploy your application across the swarm, use the bundle feature of the Docker experimental build.

More info:
https://docs.docker.com/compose/bundles

known unkowns

• new multi-host networking is not compatible with legacy multi-host networking
• containers deployed via docker run cannot participate in the new multi-host networks
• legacy multi-host network requires consensus servers, such as Consul while the new network does not
• unclear how to make containers running in both networks cooperate

known unkowns

Docker Universal Control Plane (UCP) is the enterprise-grade cluster management solution from Docker. You install it behind your firewall, and it helps you manage your whole cluster from a single place.

known unkowns

known unkowns

known unkowns

known unkowns

CS Engine is short for Commercially Supported

known unkowns

thoughts

• service discovery is super easy
• baked-in health checking is a good idea
• round-robin balancing seems to be the only choice
• don't put services that already have their own replication story, like MongoDB, into a swarm
• upgrades are convenient and powerful as is scaling
• haven't seen unhealthy containers replaced -- just theory at this point
• command-line is free, unsure what the DCP costs
• non-swarmed containers probably require their own infrastructure -- discovery, health checking and monitoring
• very little control over swarm networking

questions