27.4.

2016

Docker meetup

Petr Michalec

Developer. IBMer. Vi(m) lover. DevOps kid. Runner. Performing cloud infrastructure and application architecture with passion for the edge thing.

 

Reach me on Twitter as @epcim.

 

Pavel Čižinský, @LotharKAtt

Cloud Junior at tcp cloud. Proud debian user.  

Swarm

Swarm

  • Native clustering for Docker
  • Version 1.0 announced 11/2015
  • Turns a pool of Docker hosts into a single, virtual Docker host, uff ...
  • Pluggable backend for orchestration
  • Uses extended Docker remote API

Engine related features

  • docker 1.10  (Feb 4)
    • New Compose file
    • Temporary filesystems
    • Live update container resource constraints
    • Event streams updated
    • DNS based discovery
  • docker 1.11  (April 13)
    • built on runC and containerd
    • DNS round robin load balancing
    • IPv6 service discovery

Swarm 1.0 +

  • Works with engine >= 1.9
    • new networking
    • persistent storage (flocker, cepth)
  • Version 1.1 (Feb 4)
    • experimental reschedule if host fails
  • Version 1.2 (April 13, current)
    • reschedule containers no longer experimental
    • fixes, better errors

 

On board features

On board features

  • API compatibility
  • Scheduler
  • Labels, Afinities, Constrains
  • Networking
    • Support multi-host networking
    • DNS based service discovery, round-robin

Off the board

  • Volume orchestration
  • Image management
  • Rebalance on recovery, etc.
  • Jobs, task, etc.
  • . . . 

Roadmap (May/'15)

  • Scheduler
    • rebalancing, global scheduller
  • HA
    • shared states between multiple "soon-to-be-master"
  • API
    • Bring Swarm API on par with Docker API
    • Use labels for constraints and affinity
  • Extensibility
    • cluster drivers (Mesos #393, Kubernetes)

Architecture

Architecture

  • Available as container or binnary
  • Plugable KV store
    • etcd, consul.io, zookeper, file, hosted
  • HA
    • failover of the manager instance
    • single primary, multiple replicas
  • Reschedule containers on node failure

Scheduling

Filters

Strategies

Rescheduling

Filters

Placement decisions

  • As container
    • constraint
    • health
  • As container config 
    • Afinity
    • Dependency
    • Port

Strategies

Right now

  • Spread                 
  • Binpack

 

Under Dev

  • Mesos, ...
  • Balanced
  • <fully pluggable>

Reschedule on failure

##
# Default off

# evironment variable
docker run -d -e reschedule:on-node-failure redis

# or as label
docker run -d -l 'com.docker.swarm.reschedule-policy=["on-node-failure"]' redis



##
# Swarm manager logs:

ERRO[2173] Flagging engine as unhealthy. Connect failed 3 times  id=VD3Q:XLOA:U2FH:TW7F:LMDM:YK3I:ZF4L:LRMM:X2GF:EI5X:NUEY:TQMP 
  name=swarm-node-1

INFO[2191] Rescheduled container d96a624fcf8b52488b5f057f7032b8666f1c338c7d62ac5a9f6c8aa3cb6d330c 
  from swarm-node-1 to swarm-node-2 as e85a6056d533d21c4334738a866d4349a1ce72abbf2917735f976ef469debd49 

INFO[2191] Container d96a624fcf8b52488b5f057f7032b8666f1c338c7d62ac5a9f6c8aa3cb6d330c
  was running, starting container e85a6056d533d21c4334738a866d4349a1ce72abbf2917735f976ef469debd49 

Filter examples

# Start a Nginx node on the edge
$ docker run -d -p 80:80 \
    --name edge_webserver \
    -e constraint:zone==external \
    -e constraint:disk==ssd \
    -t nginx:latest

# Start an app container on the same Docker host
$ docker run -d -p 5000 \
    --name app_1 \
    -e affinity:container==edge_webserver \
    -t app:latest


# Or with labels
$ docker run -itd --label 'com.docker.swarm.affinities=["container==c1"]' busybox
$ docker run -itd --label 'com.docker.swarm.constraints=["disk==ssd"]' busybox

Filter expressions


# Some possible expressions:

constraint:node==node1 # matches node node1.
constraint:node!=node1 # matches all nodes, except node1.
constraint:region!=us* # matches all nodes outside with a region tag prefixed with us.
constraint:node==/node[12]/ # matches nodes node1 and node2.
constraint:node==/node\d/ # matches all nodes with node + 1 digit.
constraint:node!=/node-[01]/  # matches all nodes, except node-0 and node-1.
constraint:node!=/foo\[bar\]/ # matches all nodes, except foo[bar]. You can see the use of escape characters here.
constraint:node==/(?i)node1/  # matches node node1 case-insensitive. So NoDe1 or NODE1 also match.
affinity:image==~redis  # tries to match for nodes running container with a redis image
constraint:region==~us* # searches for nodes in the cluster belonging to the us region
affinity:container!=~redis* # schedule a new redis5 container to a node without a container that satisfies redis*
  • globbing pattern, abc*.
  • regular expression in the form of /regexp/

Ship IT

Run a cluster #1

$ docker-machine stop default
$ docker-machine create -d virtualbox manager
$ docker-machine create -d virtualbox agent1
$ docker-machine create -d virtualbox agent2

$ eval $(docker-machine env manager)
$ TOKEN=$(docker run --rm swarm create) # using discovery service on docker hub
$ docker run -d -p 3376:3376 -t -v /var/lib/boot2docker:/certs:ro swarm manage -H 0.0.0.0:3376 \
  --tlsverify --tlscacert=/certs/ca.pem --tlscert=/certs/server.pem --tlskey=/certs/server-key.pem \
  token://$TOKEN

$ eval $(docker-machine env agent1)
$ docker run -d swarm join --addr=$(docker-machine ip agent1):2376 token://$TOKEN

$ eval $(docker-machine env agent2)
$ docker run -d swarm join --addr=$(docker-machine ip agent2):2376 token://$TOKEN


$ DOCKER_HOST=$(docker-machine ip manager):3376
$ docker info
$ docker ps

Run a cluster #2

# get token from hosted service
export TOKEN=$(docker run swarm create)

# start first node with manager
docker-machine create \
  --engine-opt cluster-store=token://$TOKEN \
  -d virtualbox \
  --swarm \
  --swarm-master \
  --swarm-discovery token://$TOKEN \
  --swarm-strategy spread \
  --swarm-opt heartbeat=5s \
  --engine-label foo=bar \
  --engine-label spam=eggs \
  swarm-manager


# start other nodes of cluster
docker-machine create \
  --engine-opt cluster-store=token://$TOKEN \
  -d virtualbox \
  --swarm \
  --swarm-discovery token://$TOKEN \
  --swarm-opt heartbeat=5s \
  --engine-label foo=baz \
  swarm-node-1

docker-machine create \
  --engine-opt cluster-store=token://$TOKEN \
  -d virtualbox \
  --swarm \
  --swarm-discovery token://$TOKEN \
  --swarm-opt heartbeat=5s \
  --engine-label foo=baz \
  swarm-node-2

# use cluster
 eval "$(docker-machine env -swarm swarm-manager)"
 docker info
 docker ps

Run a cluster #3

# get local kv store
docker run -d -p 8500:8500 --name=consul --restart=always progrium/consul -server -bootstrap
CONSUL=$(hostname -I | awk '{print $1}'):8500

MNGRS=(manager-1 manager-2)
NODES=(front-node-1 app-node-2 data-node-3)

# start manager nodes
for n in ${MNGRS[@]}; do
  docker-machine create \
    --engine-opt cluster-store=consul://$CONSUL \
    --engine-opt cluster-advertise=eth1:2376 \ # optional
    -d virtualbox \
    --swarm \
    --swarm-master \
    --swarm-discovery consul://$CONSUL \
    --swarm-strategy spread \               # optional
    --swarm-opt heartbeat=5s \              # optional, customize
    --swarm-opt replication \
    --swarm-opt advertise=eth1:3376 \
    --engine-label foo=bar \                # customize
    --engine-label host=${n//-*/} \         # eg: manager, front, app, data
    $n
done


# start workload nodes of cluster
for n in ${NODES[@]}; do
  docker-machine create \
    --engine-opt cluster-store=consul://$CONSUL \
    -d virtualbox \
    --swarm \
    --swarm-discovery consul://$CONSUL \
    --swarm-opt heartbeat=5s \              # optional, customize
    --engine-label foo=baz \                # customize
    --engine-label spam=eggs \              # customize
    --engine-label host=${n//-*/} \         # eg: manager, front, app, data
    # add some driver options for CPU, RAM, STORAGE: https://docs.docker.com/machine/drivers/
    $n
  # add registrator at each node
  eval "$(docker-machine env ${n})"
  docker run -d --name=registrator -h $(docker-machine ip ${n}) \
    -v=/var/run/docker.sock:/tmp/docker.sock \
    gliderlabs/registrator:latest \
    consul://$CONSUL
done


# use cluster
 eval "$(docker-machine env -swarm manager-1)"
 docker info
 docker network create --subnet 10.10.10.0/24 swarm
 docker network ls
 docker ps

Finally right !

Interact with

user@my-machine $ docker info
Containers: 0
Images: 25
Storage Driver:
Role: Primary                  #<--------- manager-1 is the Primary manager
Primary: 192.168.42.200
Strategy: spread
Filters: affinity, health, constraint, port, dependency
Nodes: 3
 swarm-agent-0: 192.168.42.100:2375
  └ Containers: 0
  └ Reserved CPUs: 0 / 1
  └ Reserved Memory: 0 B / 2.053 GiB
  └ Labels: executiondriver=native-0.2, kernelversion=3.13.0-49-generic, operatingsystem=Ubuntu 14.04.2 LTS, storagedriver=aufs
 swarm-agent-1: 192.168.42.101:2375
  └ Containers: 0
  └ Reserved CPUs: 0 / 1
  └ Reserved Memory: 0 B / 2.053 GiB
  └ Labels: executiondriver=native-0.2, kernelversion=3.13.0-49-generic, operatingsystem=Ubuntu 14.04.2 LTS, storagedriver=aufs
 swarm-agent-2: 192.168.42.102:2375
  └ Containers: 0
  └ Reserved CPUs: 0 / 1
  └ Reserved Memory: 0 B / 2.053 GiB
  └ Labels: executiondriver=native-0.2, kernelversion=3

Interact with

# netwok create
docker network create -d overlay dev-www

# docker network ls
NETWORK ID          NAME                  DRIVER
a4d45b64d6b0        app-node-2/bridge     bridge              
4231de0c7baa        app-node-2/host       host                
13765007b25f        app-node-2/none       null                
ad4371891dee        data-node-3/bridge    bridge              
95ffa1b57328        data-node-3/host      host                
354cc6d91fac        data-node-3/none      null                
b3abccb078ca        front-node-1/bridge   bridge              
a0b039bdfa2d        front-node-1/host     host                
fe0509247bd8        front-node-1/none     null                
d6abd3ae9810        manager-1/bridge      bridge              
3fb5ef9bbb18        manager-1/host        host                
78c6834b6e71        manager-1/none        null                
2a2ac374f2ca        manager-2/bridge      bridge              
44d3a3b76786        manager-2/host        host                
0e91607fdf99        manager-2/none        null                
9eb82f97c6fa        dev-www               overlay  

Interact with

# docker run --net test1 -m 64m -d -p 8082:80 tutum/wordpress
bd1a41d7c1bb030f4f77abb6edd318542f2cba14deb4cc03c5ceadb74291bf7e

# docker ps
CONTAINER ID        IMAGE                           COMMAND                  CREATED             STATUS              PORTS                                   NAMES
bd1a41d7c1bb        tutum/wordpress                 "/run.sh"                2 minutes ago       Up 2 minutes        3306/tcp, 192.168.99.116:8082->80/tcp   front-node-1/goofy_williams
9e2c46278eec        redis                           "docker-entrypoint.sh"   12 minutes ago      Up 12 minutes       6379/tcp                                manager-2/testredis
11f3846446e9        busybox                         "sh"                     22 minutes ago      Up 22 minutes                                               manager-1/ecstatic_brahmagupta
c259f74e1660        gliderlabs/registrator:latest   "/bin/registrator con"   24 minutes ago      Up 24 minutes                                               data-node-3/registrator
44c16500ee75        gliderlabs/registrator:latest   "/bin/registrator con"   26 minutes ago      Up 26 minutes                                               app-node-2/registrator
84110f05cc19        gliderlabs/registrator:latest   "/bin/registrator con"   27 minutes ago      Up 27 minutes                                               front-node-1/registrator

# registrator add workdpress to consul, log entry
# 2016/04/27 14:26:37 added: 0281638846de 192.168.99.117:adoring_saha:80

# docker network ls
NETWORK ID          NAME                           DRIVER
...              
b3abccb078ca        front-node-1/bridge            bridge              
fc338f67bcc6        front-node-1/docker_gwbridge   bridge              
a0b039bdfa2d        front-node-1/host              host                
fe0509247bd8        front-node-1/none              null                
...          
9eb82f97c6fa        dev-www                        overlay             

Swarm use-case ?

 turns a pool of Docker hosts into a single, virtual Docker host

Swarm as service ?

Available as hosted service:

  • Rackspace CARINA
  • Rancher
  • . . .

Deployment

  • Deployment options
    • by hand, poor man choice
    • docker-machine
      • multiple cloud providers
      • easy to use, not for production
    • Configuration management tools
      • modules/cookbooks availability ???
  • Mind the related componetns
    • KV store, Service discovery
    • Storrage (Flocker, REX-Ray, ... )

Shi(*) happens

Reschedule on failure

##
# Default off

# evironment variable
docker run -d -e reschedule:on-node-failure redis

# or as label
docker run -d -l 'com.docker.swarm.reschedule-policy=["on-node-failure"]' redis



##
# Swarm manager logs:

ERRO[2173] Flagging engine as unhealthy. Connect failed 3 times  id=VD3Q:XLOA:U2FH:TW7F:LMDM:YK3I:ZF4L:LRMM:X2GF:EI5X:NUEY:TQMP 
  name=swarm-node-1

INFO[2191] Rescheduled container d96a624fcf8b52488b5f057f7032b8666f1c338c7d62ac5a9f6c8aa3cb6d330c 
  from swarm-node-1 to swarm-node-2 as e85a6056d533d21c4334738a866d4349a1ce72abbf2917735f976ef469debd49 

INFO[2191] Container d96a624fcf8b52488b5f057f7032b8666f1c338c7d62ac5a9f6c8aa3cb6d330c
  was running, starting container e85a6056d533d21c4334738a866d4349a1ce72abbf2917735f976ef469debd49 
ERRO[2504] Update engine specs failed: Cannot connect to the Docker daemon. 
  Is the docker daemon running on this host?  id=CRIG:LKCV:DK5O:JWHO:AHYW:LCVF:OGZW:KJT4:FSEB:FWXL:77JR:EQYB 
  name=swarm-node-2

INFO[2607] Removed Engine swarm-node-2        
          
ERRO[2607] Failed to reschedule container e85a6056d533d21c4334738a866d4349a1ce72abbf2917735f976ef469debd49: 
  Conflict: The name /serene_archimedes is already assigned. 
  You have to delete (or rename) that container to be able to 
  assign /serene_archimedes to a container again. 

Be aware of

  • No persistance - no shared state
    • If master + nodes goes offline
    • KV is rebuilded on master restart
  • Node failure
    • Do not auto-start containers
    • Stopped containers counts "spread"
    • Failed host is removed by TTL from KV
  • Node without KV breaks the cluster net.
  • Failed reschedule on first container blocked reschedule of the other ones.
  • Swarm manager: Leader not elected

#SwarmWeek picks

#SwarmWeek picks

#SwarmWeek picks

Docker Swarm exceeds Kubernetes scale, http://bit.ly/1WoxIN0 http://bit.ly/1WoxPrS

The study and article, commissioned by Docker, tested the performance of both platforms while running 30,000 containers across 1,000 node clusters

#SwarmWeek picks

BACKUP SLIDES

Dehonested kube?

Dehonested kube?

Dehonested kube?

Dehonested kube?

Dehonested kube?

Dehonested kube?

backup: multi-node

backup: multi-node

# external KV store
docker run -d -p 8500:8500 --name=consul progrium/consul -server -bootstrap
consul=$(hostname -I | awk '{print $1}'):8500

SWARM_NODE=(manager-1 manager-2 frontend-node-1 app-node-2 db-node-3)
for i ${SWARM_NODE[@]}; do 
    docker-machine create \
        --engine-opt cluster-store=consul://$consul \
        --engine-opt cluster-advertise=eth1:2376 \
        --engine-label spam=eggs --engine-label hosttype=${i} \
        -d virtualbox ${i}
done

for i ${SWARM_NODE[@]}; do 
    eval $(docker-machine env ${i})
    [[ "${i::7}" == "manager" ]] && { # manager/master
        docker network create -d overlay --subnet 10.10.10.0/24 swarm
        # with dedicated KV just for swarm
        docker run -d --net swarm --name=consul progrium/consul -server -join $consul
        docker run -d --net swarm --name=manager -p 3376:3376 -t -v /var/lib/boot2docker:/certs:ro \
            swarm manage -H 0.0.0.0:3376 \
            --tlsverify --tlscacert=/certs/ca.pem --tlscert=/certs/server.pem --tlskey=/certs/server-key.pem \
            --replication --advertise=eth1:3376 \
            consul://consul:8500
    } || {
    docker run --net=swarm -d swarm join --addr=$(docker-machine ip ${i}):2376 consul://consul:8500
    }
done

export DOCKER_HOST=$(docker-machine ip manager):3376
docker info
docker ps

# or just use external kv: "consul://$consul"

Docker Swarm

By Petr Michalec

Docker Swarm

Docker cluster with Swarm

  • 2,143
Loading comments...

More from Petr Michalec