Go, Docker & HashiCorp

2014/04/25

About Me

Software Engineer

Backend/DevOps

http://liubin.org

Source Code

https://github.com/liubin/workshop-China-pub

VirtualBox & Vagrant needed.

Golang

mascot from: https://github.com/golang-samples/gopher-vector

Interface

Simple & Pragmatic

Cross Compile

Single executable file

Goroutine

Docker

Develop, Ship and Run Any Application, Anywhere

History & background

cloud(PaaS/IaaS)

dotCloud

Docker is not virtualization

Docker = Go + Linux Kernel

Linux Kernel

NameSpace

CGroup

Union file systems

execdriver

lxc
native(libcontainer)

graphdriver

aufs
btrfs
devmapper
vfs

Docker

start from container but not only container now.

a platform of build, ship and run software

Docker platform

=

Hub + Engine

Docker's Architecture 1

Daemon
Client
Remote API

Docker's Architecture 2

Docker registries.
Docker images.
Docker containers.

Docker Registries

Docker registries hold images.
Docker registries are the distribution component of Docker.

Docker Images

A Docker image is a read-only template.
Docker images are the build component of Docker.

Docker Containers

Docker containers are similar to a directory + process(es).
Created from a Docker image.
Docker containers are the run component of Docker.

Docker Image                     Docker Container

------------->

<-------------

docker run

docker commit

Example 1

$ sudo docker run -it centos:centos7 bash
[root@e8f2d409dd27 /]# id
uid=0(root) gid=0(root) groups=0(root)
[root@e8f2d409dd27 /]# ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 16:12 ?        00:00:00 bash
root        19     1  0 16:12 ?        00:00:00 ps -ef

docker run [options] image [args]

docker run options

-p : port mapping

-v : volume

-i : interactive

-t : tty

-d : daemon

--name : naming container

--restart : --restart=on-failure:5

--rm : auto remove container

--link : link other containers

$ sudo docker run --name -d redis redis:3

$ sudo docker run --link redis:cache -it centos:centos7 bash

[root@964115430b85 /]# env
HOSTNAME=964115430b85
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/
container_uuid=96411543-0b85-e2fb-0ac5-a7a0e0bc3c86
HOME=/root
LESSOPEN=||/usr/bin/lesspipe.sh %s

CACHE_ENV_REDIS_DOWNLOAD_SHA1=c75fd32900187a7c9f9d07c412ea3b3315691c65
CACHE_PORT_6379_TCP_PORT=6379
CACHE_PORT_6379_TCP_PROTO=tcp
CACHE_PORT=tcp://172.17.0.12:6379
CACHE_ENV_REDIS_DOWNLOAD_URL=http://download.redis.io/releases/redis-3.0.0.tar.gz
CACHE_ENV_REDIS_VERSION=3.0.0
CACHE_NAME=/pensive_hodgkin/cache
CACHE_PORT_6379_TCP_ADDR=172.17.0.12
CACHE_PORT_6379_TCP=tcp://172.17.0.12:6379

# cat /etc/hosts
172.17.0.14	ccbfe5d5c4cd
127.0.0.1	localhost
... ...
172.17.0.12	cache dac176cba6c5 redis

$ pwd
/workshop

$ sudo docker run -it -v /workshop:/anywhere centos:centos7 bash

[root@5da9af015da5 /]# mount | grep any
none on /anywhere type vboxsf (rw,nodev,relatime)

[root@5da9af015da5 /]# ls /anywhere/
LICENSE  README.md  Vagrantfile  example-2  example-3  example-4  example-5

[root@5da9af015da5 /]#

docker ps

docker ps -a

docker ps -l

docker ps -q

$ sudo docker ps -a
CONTAINER ID        IMAGE                                                                     COMMAND                CREATED              STATUS                     PORTS               NAMES
e8f2d409dd27        centos:centos7                                                            "bash"                 About a minute ago   Exited (0) 9 seconds ago                       jovial_morse        
fb4ea58880d2        540c9254ff330bcc1a56f415f92139737de47e8747da59f5b96668910a0a8366:latest   "/bin/sh -c 'yum ins   6 minutes ago        Up 6 minutes                                   nostalgic_banach    
[vagrant@localhost ~]$ sudo docker ps -l
CONTAINER ID        IMAGE                                                                     COMMAND                CREATED             STATUS              PORTS               NAMES
4af352f7616a        3b527bcfe2d682b873b504f9b5dfc924529d9e34d51ebfc59379b0fe5a0fcf10:latest   "/bin/sh -c 'cd /tmp   2 minutes ago       Up 2 minutes                            romantic_wilson     
[vagrant@localhost ~]$ sudo docker ps -l -q
4af352f7616a

docker stop/restart

docker start/restart

docker stop/kill/rm

docker log [-f|t]

docker exec

docker inspect

Docker Image

docker images

$ sudo docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
ruby-redis-app      latest              4a8926e70ba2        11 minutes ago      577.3 MB
ruby-app            latest              a508fb5d4c30        17 minutes ago      568 MB
ruby                latest              66b9f2758422        21 minutes ago      556.4 MB
docker.io/redis     3                   06a1f75304ba        2 days ago          111 MB
docker.io/redis     3.0                 06a1f75304ba        2 days ago          111 MB
docker.io/redis     3.0.0               06a1f75304ba        2 days ago          111 MB
docker.io/redis     latest              06a1f75304ba        2 days ago          111 MB
docker.io/centos    latest              fd44297e2ddb        2 days ago          215.7 MB
docker.io/centos    7                   fd44297e2ddb        2 days ago          215.7 MB
docker.io/centos    centos7             fd44297e2ddb        2 days ago          215.7 MB

docker rmi

docker history

$ sudo docker history redis:3
IMAGE               CREATED             CREATED BY                                      SIZE
06a1f75304ba        2 days ago          /bin/sh -c #(nop) CMD ["redis-server"]          0 B
54ca92b7c8d7        2 days ago          /bin/sh -c #(nop) EXPOSE 6379/tcp               0 B
6755f61be70b        2 days ago          /bin/sh -c #(nop) ENTRYPOINT ["/entrypoint.sh   0 B
40980abbab9f        2 days ago          /bin/sh -c #(nop) COPY file:c4bbead0efd18835c   109 B
e501d0146d1d        2 days ago          /bin/sh -c #(nop) WORKDIR /data                 0 B
d315f0a01142        2 days ago          /bin/sh -c #(nop) VOLUME [/data]                0 B
3a8cd27bb3d5        2 days ago          /bin/sh -c mkdir /data && chown redis:redis /   0 B
4454da7c7dbc        2 days ago          /bin/sh -c buildDeps='gcc libc6-dev make';      s                   9.373 MB
7ebc2ece510e        2 days ago          /bin/sh -c #(nop) ENV REDIS_DOWNLOAD_SHA1=c75   0 B
1c255a1b1254        2 days ago          /bin/sh -c #(nop) ENV REDIS_DOWNLOAD_URL=http   0 B
130c4eb9410a        2 days ago          /bin/sh -c #(nop) ENV REDIS_VERSION=3.0.0       0 B
8d9a45a71a91        2 days ago          /bin/sh -c curl -o /usr/local/bin/gosu -SL "h   2.141 MB
c215cb712b89        2 days ago          /bin/sh -c gpg --keyserver pool.sks-keyserver   98.87 kB
b720af9a6508        2 days ago          /bin/sh -c apt-get update                       && apt-get install   14.1 MB
29809ed33dfd        2 days ago          /bin/sh -c groupadd -r redis && useradd -r -g   328.3 kB
b3d362b23ec1        2 days ago          /bin/sh -c #(nop) CMD ["/bin/bash"]             0 B
21e4345e9035        2 days ago          /bin/sh -c #(nop) ADD file:20cd6318f68d34ca8e   84.96 MB

Build Image

docker commit

Dockerfile

FROM centos:centos7

MAINTAINER Bin Liu <liubin0329@gmail.com>

RUN yum update -y && yum install -y unzip wget

RUN cd /tmp \
  && wget https://dl.bintray.com/mitchellh/serf/0.6.4_linux_amd64.zip

RUN cd /tmp \
  && unzip 0.6.4_linux_amd64.zip \
  && chmod +x serf \
  && mv serf /usr/bin/serf


ADD handler.sh /handler.sh

RUN chmod +x /handler.sh

WORKDIR /

docker build -t imagename .

RUN

instruction granularity

CMD & ENTRYPOINT

ENTRYPOINT ["/usr/sbin/nginx"]

CMD ["-h"]

dockuer run --entrypoint

Example 2

builde a Ruby runtime

Tips

$ sudo docker rm $(sudo docker ps -a -q)

$ sudo docker inspect --format '{{ .NetworkSettings.IPAddress }}' f007c401f577

$ sudo docker images | grep none | awk '{ print $3 }' | xargs sudo docker rmi

Docker Hub & AutoBuild

Example-3

Dockerizing a Sinatra app

Docker compose

A tool for defining and running multi-container applications by one file and one command.

Example-4

# docker-compose.yml
web:
  build: .
  links:
   - redis
  ports:
   - "4567:4567"
redis:
  image: redis:3

# app.rb
$redis = Redis.new(:host => "redis", :port => 6379)

Docker Machine

Machine is currently in beta

http://docs.docker.com/machine/

It creates servers, installs Docker on them, then configures the Docker client to talk to them.

Docker Swarm

Manager

Node

Scheduler

Discovery

Pluggable Scheduler

Filter
Strategy

Constraint
Affinity
Port
Dependency
Health

Filter

spread
binpack
random

Strategy

Hosted discovery service
Static file
Etcd
Consul
Zookeeper
IPs(list or range pattern)

Discovery

Orchestration

Docker production pattern

1/n

1 / n

credit: http://techlife.cookpad.com/entry/2015/04/20/134758

Use Case

Anywhere

CI/CD

Dev/Prod

PaaS

Delivery

... ...

SOA/Micro service

No installations,

Just docker run

HashiCorp

The MOST sexy tools/company

about

DevOps, Docker, Go,

Immutable Infrastructure,
Infrastructure as Code,

-- by bin liu :-)

Vagrant

Development environments made easy

https://www.vagrantup.com/

Create and configure lightweight, reproducible, and portable development environments.

$ vagrant init hashicorp/precise32
$ vagrant up

#!/usr/bin/env bash

apt-get update
apt-get install -y apache2
if ! [ -L /var/www ]; then
  rm -rf /var/www
  ln -fs /vagrant /var/www
fi

Vagrant.configure("2") do |config|
  config.vm.box = "hashicorp/precise32"
  config.vm.provision :shell, path: "bootstrap.sh"
end

Provision

Packer

https://www.packer.io/

Packer is a tool for creating identical machine images for multiple platforms from a single source configuration.

Super fast infrastructure deployment.

Multi-provider portability.

Improved stability.

Greater testability.

Builders

Amazon EC2 (AMI)
DigitalOcean
Docker
Google Compute Engine
Null
OpenStack
Parallels
QEMU
VirtualBox
VMware

Provisioners

Shell Scripts
File Uploads
Ansible
Chef Client
Chef Solo
Puppet Masterless
Puppet Server
SaltBullet Two

Post-processors

Atlas
compress
docker-import
docker-push
docker-save
docker-tag
Vagrant
Vagrant Cloud
vSphere

Serf

https://serfdom.io/

Serf

Serf is a decentralized solution for cluster membership, failure detection, and orchestration.

Serf

GOSSIP-BASED
FAILURE DETECTION
CUSTOM EVENTS

vs ZooKeeper, etcd

Client/Server
complex, not only a tool
heartbeating
long failure detection window
strongly consistent

Example 5

run Serf in Docker container

Consul

https://consul.io/

Service discovery and configuration made easy. Distributed, highly available, and datacenter-aware.

Service Discovery

Failure Detection

Multi Datacenter

Key/Value Storage

Built-in DNS server

DNS/HTTP interface

Raft

http://thesecretlivesofdata.com/raft

K/V Store

$ curl -X PUT -d 'test' http://localhost:8500/v1/kv/web/key1
true
$ curl -X PUT -d 'test' http://localhost:8500/v1/kv/web/key2?flags=42
true
$ curl -X PUT -d 'test'  http://localhost:8500/v1/kv/web/sub/key3
true
$ curl http://localhost:8500/v1/kv/?recurse
[{"CreateIndex":97,"ModifyIndex":97,"Key":"web/key1","Flags":0,"Value":"dGVzdA=="},
 {"CreateIndex":98,"ModifyIndex":98,"Key":"web/key2","Flags":42,"Value":"dGVzdA=="},
 {"CreateIndex":99,"ModifyIndex":99,"Key":"web/sub/key3","Flags":0,"Value":"dGVzdA=="}]

Services

Service Definition
HTTP API

Create service

$ echo '{"service": {"name": "web", "tags": ["rails"], "port": 80}}' \
    >/etc/consul.d/web.json

Query services

$ dig @127.0.0.1 -p 8600 web.service.consul SRV
...
;; QUESTION SECTION:
;web.service.consul.    IN  SRV
;; ANSWER SECTION:
web.service.consul. 0   IN  SRV 1 1 80 agent-one.node.dc1.consul.
;; ADDITIONAL SECTION:
agent-one.node.dc1.consul. 0    IN  A   172.20.20.11

event, watch, exec

$ consul event deploy 4F5E234

... ...

$ consul watch -type event -name deploy /home/user/deploy.sh

... ...

$ consul exec -service web /home/user/deploy.sh

Terraform

https://terraform.io/

A tool for building, changing, and versioning infrastructure safely and efficiently.

Infrastructure as Code

Compose IaaS/PaaS/Saas

Unified view

One workflow

Physical to Container

Codes

provider "aws" {
    access_key = "ACCESS_KEY_HERE"
    secret_key = "SECRET_KEY_HERE"
    region = "us-east-1"
}
resource "aws_instance" "example" {
    ami = "ami-408c7f28"
    instance_type = "t1.micro"
}

terraform plan

$ terraform plan
...
+ aws_instance.example
    ami:               "" => "ami-408c7f28"
    availability_zone: "" => "<computed>"
    instance_type:     "" => "t1.micro"
    key_name:          "" => "<computed>"
    private_dns:       "" => "<computed>"
    private_ip:        "" => "<computed>"
    public_dns:        "" => "<computed>"
    public_ip:         "" => "<computed>"
    security_groups:   "" => "<computed>"
    subnet_id:         "" => "<computed>"

terraform apply

$ terraform apply

aws_instance.example: Creating...
  ami:           "" => "ami-408c7f28"
  instance_type: "" => "t1.micro"
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
...

terraform show

$ terraform show
aws_instance.example:
  id = i-e60900cd
  ami = ami-408c7f28
  availability_zone = us-east-1c
  instance_type = t1.micro
  key_name =
  private_dns = domU-12-31-39-12-38-AB.compute-1.internal
  private_ip = 10.200.59.89
  public_dns = ec2-54-81-21-192.compute-1.amazonaws.com
  public_ip = 54.81.21.192
  security_groups.# = 1
  security_groups.0 = default
  subnet_id =