Docker workshop

LEVEL DEVil

il

About me

Developer. IBMer. Vi(m) lover. DevOps kid. Performing cloud infrastructure and application architecture with passion for the edge thing.

 

I am not obsessed with emails and spreadsheet thing as an ordinary IBMers. Reach me on Twitter as @epcim.

 

Workshop agenda

Morning LEVEL DEVil ~ about the 'event'

DevOps, Docker intro
Docker usage in Watson team (Martin Cochner)
Learn core principles, volumes, networking, orchestration.
Develop custom container to host micro-service.
Consume community containers.
Afternoon Hack session
* Leverage the skills and build something worth!
* Drone.io/Jenkins CI pipeline, ...
* Containers on Bluemix, ...
* << bring your own ideas to work on >>
T Day 1

Something about you

Background | Practice | Expectations

Buzzword

By Definition

IBM

DevOps is an enterprise capability for continuous software delivery that enables clients to seize market opportunities and reduce time to customer feedback

Pick #1

Chef.io

A cultural and professional movement, focused on how we build and operate high velocity organizations, born from the experiences of its practitioners. 

Pick #2

Three ways

The principles that all of the DevOps patterns can be derived from

Pick #3

Forget DEV to OPS !

Practices

  • Automation
  • Continuous Delivery
  • Collaboration across teams
  • Extended visibility
  • Amplified feedback loops, rapid feedback

Feedback loop

  • Do something
  • Test, Measure
  • Analyze
  • Correct
  • Repeat

Business value

  • Enhanced customer experience
  • Increased capacity to innovate    
  • Faster time to value

Wait a while ...

What did the Waterfall, Agile, Scrum done wrong?

retired buzzwords

Design Thinking

  • Learning about the audience, customer view.
  • Aptly vs. real needs
  • Ideal vs. realistic

DevOps

  • Feedback loops, cooperation, integration     
  • Continuous Delivery

Agile

  • #12 principles, but "frequent delivery" shines
  • ...

What are agile trends?

What are agile trends?

  • From traditional IT to microservices

  • API loosely coupled architectures and micro services

  • Technology shift on frontends

  • Design for speed and simplicity

  • Cross platform requirements (Go, mobile platforms)

  • Measure for feedback, data analytics, visualization

  • Stream data

  • Dynamic configuration

  • Service discovery

  • ...

Applied           Style DevOps Kung Fu

Originaly by  Adam Jacob / CTO of Chef / @adamhjk

 

Chef Style DevOps Kung Fu

Creative Commons License

Deploy and operate all infrastructures by code, as applications are code too.

As traditional IT becomes "truck" and cloud native is a "car". Practice DevOps and apply cloud architectures to acquire expertise, tools and operational skills.

Personal picks

  • DevOps approach
  • Life-cycle in mind
  • Meet deadlines
  • Independent dependability
  • One artifact for all env.
  • Same deployment process
  • Four eyes rule; visibility
  • Team is not isolated unit
  • Share

Tools

Preffered languages and tools that fit the job

  • Consul
  • ELK stack
  • Grafana
  • Kubernetes
  • Bluemix

Backup slides

Vagrant

Kibana (ELK stack)

Grafana

Consul.io

Consul.io

Break

(~ 5 min)

Docker

Docker

 

Crash intro (slides 1-16)

 

  • Docker Engine
  • Docker Hub
  • Docker Machine
  • Docker Compose
  • Docker Toolbox
  • Docker Swarm
  • Docker Registry
  • . . .

Docker

Client

Daemon

HTTP :2376

Docker

Filesystem

Docker

VM vs Container

Docker

Architecture

Docker

# Start daemon manually
$ docker daemon -d
$ docker ps

# Expose external host
$ docker -H tcp://192.168.10.1:2376 ps
$ export DOCKER_HOST="tcp://192.168.10.1:2376"
$ docker ps


# Win/OSX - Start docker-machine
$ docker-machine create --driver virtualbox demo
$ docker-machine ls
NAME           ACTIVE   DRIVER       STATE     URL                         SWARM
demo           -        virtualbox   Running   tcp://192.168.99.100:2376   

$ docker-machine env demo
export DOCKER_TLS_VERIFY="1"
export DOCKER_HOST="tcp://192.168.99.100:2376"
export DOCKER_CERT_PATH="$HOME/.docker/machine/machines/demo"
export DOCKER_MACHINE_NAME="demo"

# Run this command to configure your shell: 
eval "$(/usr/local/bin/docker-machine env demo)"

Docker CLI

Usage: docker [OPTIONS] COMMAND [arg...]
       docker daemon [ --help | ... ]
       docker [ --help | -v | --version ]

A self-sufficient runtime for containers.

Commands:
    attach    Attach to a running container
    build     Build an image from a Dockerfile
    commit    Create a new image from a container's changes
    cp        Copy files/folders between a container and the local filesystem
    create    Create a new container
    diff      Inspect changes on a container's filesystem
    events    Get real time events from the server
    exec      Run a command in a running container
    export    Export a container's filesystem as a tar archive
    history   Show the history of an image
    images    List images
    import    Import the contents from a tarball to create a filesystem image
    info      Display system-wide information
    inspect   Return low-level information on a container or image
    kill      Kill a running container
    load      Load an image from a tar archive or STDIN
    login     Log in to a Docker registry
    logout    Log out from a Docker registry
    logs      Fetch the logs of a container
    network   Manage Docker networks
    pause     Pause all processes within a container
    port      List port mappings or a specific mapping for the CONTAINER
    ps        List containers
    pull      Pull an image or a repository from a registry
    push      Push an image or a repository to a registry
    rename    Rename a container
    restart   Restart a container
    rm        Remove one or more containers
    rmi       Remove one or more images
    run       Run a command in a new container
    save      Save one or more images to a tar archive
    search    Search the Docker Hub for images
    start     Start one or more stopped containers
    stats     Display a live stream of container(s) resource usage statistics
    stop      Stop a running container
    tag       Tag an image into a repository
    top       Display the running processes of a container
    unpause   Unpause all processes within a container
    update    Update configuration of one or more containers
    version   Show the Docker version information
    volume    Manage Docker volumes
    wait      Block until a container stops, then print its exit code

Docker machine

$ tree ~/.docker/
/home/userdir/.docker/
└── machine
    ├── certs
    │   ├── ca-key.pem
    │   ├── ca.pem
    │   ├── cert.pem
    │   └── key.pem
    └── machines
        ├── demo
        │   ├── boot2docker.iso
        │   ├── ca.pem
        │   ├── cert.pem
        │   ├── config.json
        │   ├── demo
        │   │   ├── demo.vbox
        │   │   ├── demo.vbox-prev
        │   │   └── Logs
        │   │       └── VBox.log
        │   ├── disk.vmdk
        │   ├── id_rsa
        │   ├── id_rsa.pub
        │   ├── key.pem
        │   ├── server-key.pem
        │   └── server.pem
        ├── demo2

Docker volumes

$ docker volume create --name demo_volume

$ docker run -v demo_volume:/foo ubuntu touch /foo/test

# map to another image
$ docker run -v demo_volume:/bar ubuntu ls -al /bar



# STORAGE PLUGINS 
# (flocker, nfs, convoy, glusterfs, rex-ray, ...)

$ docker volume create --driver nfs --name localhost/demo 
$ docker run --rm -v localhost/demo:/foo ubuntu ls -al /foo


$ docker inspect -f {{.Volumes}} ecbada822b
map[/data:/var/lib/docker/vfs/dir/cde167197ccc3e138a14f1a4f7c0d65b32cecbada822b0db4cc92e79059437a9] 

# touch it from the host
sudo touch /var/lib/docker/vfs/dir/cde167197...37a9/test-file

Docker volumes


# Dockerfile

VOLUME /data



# Data only container + volumes from

$ docker run --name dbdata postgres echo "Data-only container for postgres"
$ docker run -it --volumes-from dbdata debian /bin/bash


# Perform a backup

$ docker run --rm --volumes-from dbdata -v $(pwd):/backup debian \
  tar cvf /backup/backup.tar /var/lib/postgresql/data

Docker network

$ docker network

docker: "network" requires a minimum of 1 argument.
See 'docker network --help'.

Usage:	docker network [OPTIONS] COMMAND [OPTIONS]

Commands:
  ls                       List all networks
  rm                       Remove a network
  create                   Create a network
  connect                  Connect container to a network
  disconnect               Disconnect container from a network
  inspect                  Display detailed network information
$ docker network create demo

$ docker network ls
NETWORK ID          NAME                DRIVER
b7c9b6d0c133        demo                bridge  
bfc56a57191f        none                null                
0a07e021ffd5        host                host                
a69e42ce961a        bridge              bridge              

$ docker run --rm -ti \
  --name hostA \
  debian bash
$ docker run --rm -ti \
  --name hostB \
  --net demo \
  debian bash
$ docker run --rm -ti \
  --name hostC \
  --net demo \
  debian bash
$ ping -c 1 hostB
ping: unknown host
$ ping -c 1 hostC                                                                                   
PING hostC (172.21.0.4): 56 data bytes
64 bytes from 172.21.0.4: icmp_seq=0 ttl=64 time=0.069 ms
--- hostC ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.069/0.069/0.069/0.000 ms
$ ping -c 1 hostA                                                                               
ping: unknown host

Docker network

$ docker network connect demo hostA
$ ping -c 1 hostB                                                                                   
PING hostB (172.21.0.3): 56 data bytes
64 bytes from 172.21.0.3: icmp_seq=0 ttl=64 time=0.069 ms
--- hostB ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.069/0.069/0.069/0.000 ms
$ ping -c 1 hostC                                                                                   
PING hostC (172.21.0.4): 56 data bytes
64 bytes from 172.21.0.4: icmp_seq=0 ttl=64 time=0.069 ms
--- hostC ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.069/0.069/0.069/0.000 ms
$ ping -c 1 hostA
PING hostA (172.21.0.2): 56 data bytes
64 bytes from 172.21.0.2: icmp_seq=0 ttl=64 time=0.069 ms
--- hostA ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.069/0.069/0.069/0.000 ms
$ docker run --rm -ti \
  --name hostA \
  debian bash
$ docker run --rm -ti \
  --name hostB \
  --net demo \
  debian bash
$ docker run --rm -ti \
  --name hostC \
  --net demo \
  debian bash

Docker network

Docker network

#                    [app]
#                      |
# FRONT                |
# -----------------[rabbitmq]-----
# BACK                 |
#                      |
#                   [worker]



$ docker network create front
$ docker network create back

$ docker network connect front app

$ docker network connect front rabbitmq
$ docker network connect back rabbitmq

$ docker network connect back worker

Dockerfile

Dockerfile

FROM jpetazzo/dind
MAINTAINER Petr Michalec <epcim@apealive.net>

RUN apt-get update
RUN apt-get install -qqy    curl \
                            sudo \
                            git \
                            mercurial \
                            subversion \
                            ca-certificates \
                            locales \
                            jq

RUN echo 'en_US.UTF-8 UTF-8'>>/etc/locale.gen
RUN locale-gen en_US.UTF-8
ENV LANG en_US.UTF-8
ENV DEBIAN_FRONTEND noninteractive


## CHEF DK ###########################
RUN curl -L https://www.opscode.com/chef/install.sh | sudo bash -s -- -P chefdk
ENV PATH /opt/chefdk/bin:/.chefdk/gem/ruby/2.1.0/bin:/opt/chefdk/embedded/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
# Make Chef DK the primary Ruby/Chef development environment.
RUN echo 'eval "$(chef shell-init bash)"' >> ~/.bash_profile
RUN eval "$(chef shell-init bash)"


RUN chef gem install kitchen-docker
RUN chef gem install kitchen-openstack
RUN chef gem install chef-sugar
RUN chef gem install chef-rewind
RUN chef gem install serverspec
RUN chef gem install infratester


# berks pre-fetch some common soup of cookbooks
RUN mkdir /tmp/fake_cookbook; cd $_
RUN echo "name 'fake_cookbook'\nmaintainer 'fake_cookbook'\nlicense 'fake_cookbook'\ndescription 'fake_cookbook'\nversion '0.0.1'" > metadata.rb
RUN echo "source 'https://supermarket.chef.io'\nmetadata\n\n" > Berksfile
RUN echo "cookbook '7-zip'\ncookbook 'apache2'\ncookbook 'apt'\ncookbook 'ark'\ncookbook 'bluepill'\ncookbook 'build-essential'\ncookbook 'certificate'\ncookbook 'chef-client'\ncookbook 'chef_handler'\ncookbook 'chef_ruby'\ncookbook 'chef-sugar'\ncookbook 'chef-vault'\ncookbook 'cron'\ncookbook 'database'\ncookbook 'device-mapper'\ncookbook 'git'\ncookbook 'minitest-handler'\ncookbook 'modules'\ncookbook 'ncurses'\ncookbook 'nginx'\ncookbook 'ntp'\ncookbook 'ohai'\ncookbook 'openssh'\ncookbook 'openssl'\ncookbook 'packagecloud'\ncookbook 'pacman'\ncookbook 'perl'\ncookbook 'rbenv'\ncookbook 'readline'\ncookbook 'resolver'\ncookbook 'resource-control'\ncookbook 'rsyslog'\ncookbook 'ruby'\ncookbook 'ruby_build'\ncookbook 'runit'\ncookbook 'subversion'\ncookbook 'sudo'\ncookbook 'sysctl'\ncookbook 'system'\ncookbook 'ulimit'\ncookbook 'users'\ncookbook 'windows'\ncookbook 'xml'\ncookbook 'yum'\ncookbook 'yum-epel'\ncookbook 'zlib'\n" >> Berksfile
RUN chef exec berks install
RUN cd -


## FIX UP'S ##########################
RUN chmod -R 0440 /etc/sudoers
RUN chmod -R 0440 /etc/sudoers.d
# workaround (drone.io has no way yet to modify this image before git clone happens)
RUN git config --global http.sslverify false




VOLUME /var/lib/docker
CMD ["wrapdocker"]

Dockerfile

  • FROM
  • MAINTAINER
  • RUN
  • CMD
  • LABEL
  • EXPOSE
  • ENV
  • ADD
  • COPY
  • ENTRYPOINT
  • VOLUME
  • USER
  • WORKDIR
  • ARG
  • ONBUILD
  • STOPSIGNAL

Dockerfile Build

$ docker build -t myapp .

Sending build context to Docker daemon 92.67 kB
Step 1 : FROM ruby:2.3.0
 ---> 7ca70eb2dfea
Step 2 : MAINTAINER luka.dornhecker@gmail.com
 ---> Running in 61ffc257656d
 ---> 824c72ddfb87
Removing intermediate container 61ffc257656d
Step 3 : RUN apt-get update -qq && apt-get install -y   build-essential   nodejs   && apt-get clean   && rm -rf /var/lib/apt/lists/*
 ---> Running in 3ee402db540d
Reading package lists...
[...]
 ---> 445160b5bebb
Removing intermediate container 3ee402db540d
Step 4 : RUN mkdir -p /app
 ---> Running in 236bd3d78d9a
 ---> 9441393c6cc8
Removing intermediate container 236bd3d78d9a
Step 5 : WORKDIR /app
 ---> Running in d29b66063ee8
 ---> 4e17146f2c38
Removing intermediate container d29b66063ee8
Step 6 : COPY Gemfile Gemfile.lock /app/
 ---> fcdc11b05832
Removing intermediate container ffb8773a8acb
Step 7 : RUN gem install bundler && bundle install --jobs 20 --retry 5
 ---> Running in fbeaf1f023b2
Successfully installed bundler-1.11.2
1 gem installed
[...]
Removing intermediate container fbeaf1f023b2
Step 8 : ADD . /app
 ---> a53ec59090f6
Removing intermediate container e80472550747
Step 9 : ENV PORT 3000
 ---> Running in 8d68e64db143
 ---> bceed89e3c6e
Removing intermediate container 8d68e64db143
Step 10 : EXPOSE $PORT
 ---> Running in 1821fcb3e3d7
 ---> 8ea96dc0b9f8
Removing intermediate container 1821fcb3e3d7
Step 11 : CMD bundle exec rails -p $PORT
 ---> Running in f841cd21df61
 ---> bb540b0da5dc
Removing intermediate container f841cd21df61
Successfully built bb540b0da5dc

Docker compose

$ docker-compose
Define and run multi-container applications with Docker.

Usage:
  docker-compose [-f=<arg>...] [options] [COMMAND] [ARGS...]
  docker-compose -h|--help

Commands:
  build              Build or rebuild services
  help               Get help on a command
  kill               Kill containers
  logs               View output from containers
  pause              Pause services
  port               Print the public port for a port binding
  ps                 List containers
  pull               Pulls service images
  restart            Restart services
  rm                 Remove stopped containers
  run                Run a one-off command
  scale              Set number of containers for a service
  start              Start services
  stop               Stop services
  unpause            Unpause services
  up                 Create and start containers
  migrate-to-labels  Recreate containers to add labels
  version            Show the Docker-Compose version information

Docker compose

version: '2'

services:

  db:
    image: postgres

  web:
    build: .
    command: python manage.py runserver 0.0.0.0:8000
    volumes:
      - .:/code
    ports:
      - "8000:8000"
    depends_on:
      - db
app:
    build: docker/app
    volumes:
        - .:/srv
    net: app

db:
    image: postgres
    net: app
app:
    build: docker/app
    volumes:
        - .:/srv

db:
    image: postgres

OR

$ docker-compose --x-networking up

Docker compose

Docker compose

version: '2'

networks:
  proxy-tier:
    external:
      name: nginx-proxy

services:

  db:
    image: postgres

  web:
    build: .
    command: python manage.py runserver 0.0.0.0:8000
    volumes:
      - .:/code
    ports:
      - "8000:8000"
    depends_on:
      - db
  
  lb:
    image: tutum/haproxy
    links:
     - web
    ports:
     - "80:80"
    environment:
     - BACKEND_PORT=8000
     - BALANCE=roundrobin

compose: scale up

docker-compose up -d

docker-compose ps
docker-compose logs


# Scale up
docker-compose scale web=5
docker-compose up --force-recreate -d

with let's encrypt

version: '2'

networks:
  proxy-tier:
    external:
      name: nginx-proxy

services:

  db:
    image: postgres

  web:
    build: .
    command: python manage.py runserver 0.0.0.0:8000
    volumes:
      - .:/code
    ports:
      - "8000:8000"
    depends_on:
      - db
  
  nginx:
    image: nginx
    container_name: nginx
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - "/etc/nginx/conf.d"
      - "/etc/nginx/vhost.d"
      - "/usr/share/nginx/html"
      - "../../../volumes/proxy/certs:/etc/nginx/certs:ro"
    networks:
      - proxy-tier

  nginx-gen:
    image: jwilder/docker-gen
    container_name: nginx-gen
    volumes:
      - "/var/run/docker.sock:/tmp/docker.sock:ro"
      - "../../../volumes/proxy/templates/nginx-compose-v2.tmpl:/etc/docker-gen/templates/nginx.tmpl:ro"
    volumes_from:
      - nginx
    entrypoint: /usr/local/bin/docker-gen -notify-sighup nginx -watch -only-exposed -wait 5s:30s /etc/docker-gen/templates/nginx.tmpl /etc/nginx/conf.d/default.conf
  letsencrypt-nginx-proxy-companion:
    image: jrcs/letsencrypt-nginx-proxy-companion
    container_name: letsencrypt-nginx-proxy-companion
    volumes_from:
      - nginx
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "../../../volumes/proxy/certs:/etc/nginx/certs:rw"
    environment:
      - NGINX_DOCKER_GEN_CONTAINER=nginx-gen

Prerequisites check

Lab 0. 


# enable completition

https://github.com/docker/machine/\
  blob/master/contrib/completion/\
  bash/docker-machine.bash

Dont forget DevOps?

Cooperation

https://????.slack.com/messages/devops-workshop/

 

 

  • Login with your company email
  • Promote your progress, ask others, share code

Docker Lab I.

Break

(~ 5 min)

Docker practice

FIXME

Docker practice

Cross host dependencies

  • Ambassador pattern
    • link to local container, that act as forwarder
    • changing backend service = restarting ambasador
  • Environment pattern
    • pass the details of remote service as environment variable
    • docker -e DB=<IP ADDRESS>
    • --env-file <file>

Docker practice

Service Discovery

  • consul
  • registrator
  • etcd

Docker practice

Docker practice

Service Discovery

  • consul
  • registrator
  • etcd

Docker practice

Service Discovery

  • consul
  • registrator
  • etcd

Docker Orchestration

Orchestration

 

  • Kubernetes
  • Swarm
  • Messos framewok
  • ...

 

 

Docker at Watson

(Martin Cochner)

Docker Lab II.

Review


Questions


Feedback

Break

(~ 5 min)

Let's rock

Hack Session

Backup slides

ELK

Grafana & Sensu

Grafana dashboards

Made with Slides.com