LEVEL DEVil
il
Developer. IBMer. Vi(m) lover. DevOps kid. Performing cloud infrastructure and application architecture with passion for the edge thing.
I am not obsessed with emails and spreadsheet thing as an ordinary IBMers. Reach me on Twitter as @epcim.
Morning | LEVEL DEVil ~ about the 'event' DevOps, Docker intro Docker usage in Watson team (Martin Cochner) Learn core principles, volumes, networking, orchestration. Develop custom container to host micro-service. Consume community containers. |
Afternoon | Hack session * Leverage the skills and build something worth! * Drone.io/Jenkins CI pipeline, ... * Containers on Bluemix, ... * << bring your own ideas to work on >> |
T | Day 1 |
---|
Background | Practice | Expectations
Buzzword
DevOps is an enterprise capability for continuous software delivery that enables clients to seize market opportunities and reduce time to customer feedback
Pick #1
A cultural and professional movement, focused on how we build and operate high velocity organizations, born from the experiences of its practitioners.
Pick #2
The principles that all of the DevOps patterns can be derived from
Gee Kim, Co-Author of: “The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win.”
Pick #3
What did the Waterfall, Agile, Scrum done wrong?
retired buzzwords
From traditional IT to microservices
API loosely coupled architectures and micro services
Technology shift on frontends
Design for speed and simplicity
Cross platform requirements (Go, mobile platforms)
Measure for feedback, data analytics, visualization
Stream data
Dynamic configuration
Service discovery
...
Deploy and operate all infrastructures by code, as applications are code too.
As traditional IT becomes "truck" and cloud native is a "car". Practice DevOps and apply cloud architectures to acquire expertise, tools and operational skills.
Backup slides
(~ 5 min)
Client
Daemon
HTTP :2376
Filesystem
VM vs Container
Architecture
# Start daemon manually
$ docker daemon -d
$ docker ps
# Expose external host
$ docker -H tcp://192.168.10.1:2376 ps
$ export DOCKER_HOST="tcp://192.168.10.1:2376"
$ docker ps
# Win/OSX - Start docker-machine
$ docker-machine create --driver virtualbox demo
$ docker-machine ls
NAME ACTIVE DRIVER STATE URL SWARM
demo - virtualbox Running tcp://192.168.99.100:2376
$ docker-machine env demo
export DOCKER_TLS_VERIFY="1"
export DOCKER_HOST="tcp://192.168.99.100:2376"
export DOCKER_CERT_PATH="$HOME/.docker/machine/machines/demo"
export DOCKER_MACHINE_NAME="demo"
# Run this command to configure your shell:
eval "$(/usr/local/bin/docker-machine env demo)"
Usage: docker [OPTIONS] COMMAND [arg...]
docker daemon [ --help | ... ]
docker [ --help | -v | --version ]
A self-sufficient runtime for containers.
Commands:
attach Attach to a running container
build Build an image from a Dockerfile
commit Create a new image from a container's changes
cp Copy files/folders between a container and the local filesystem
create Create a new container
diff Inspect changes on a container's filesystem
events Get real time events from the server
exec Run a command in a running container
export Export a container's filesystem as a tar archive
history Show the history of an image
images List images
import Import the contents from a tarball to create a filesystem image
info Display system-wide information
inspect Return low-level information on a container or image
kill Kill a running container
load Load an image from a tar archive or STDIN
login Log in to a Docker registry
logout Log out from a Docker registry
logs Fetch the logs of a container
network Manage Docker networks
pause Pause all processes within a container
port List port mappings or a specific mapping for the CONTAINER
ps List containers
pull Pull an image or a repository from a registry
push Push an image or a repository to a registry
rename Rename a container
restart Restart a container
rm Remove one or more containers
rmi Remove one or more images
run Run a command in a new container
save Save one or more images to a tar archive
search Search the Docker Hub for images
start Start one or more stopped containers
stats Display a live stream of container(s) resource usage statistics
stop Stop a running container
tag Tag an image into a repository
top Display the running processes of a container
unpause Unpause all processes within a container
update Update configuration of one or more containers
version Show the Docker version information
volume Manage Docker volumes
wait Block until a container stops, then print its exit code
$ tree ~/.docker/
/home/userdir/.docker/
└── machine
├── certs
│ ├── ca-key.pem
│ ├── ca.pem
│ ├── cert.pem
│ └── key.pem
└── machines
├── demo
│ ├── boot2docker.iso
│ ├── ca.pem
│ ├── cert.pem
│ ├── config.json
│ ├── demo
│ │ ├── demo.vbox
│ │ ├── demo.vbox-prev
│ │ └── Logs
│ │ └── VBox.log
│ ├── disk.vmdk
│ ├── id_rsa
│ ├── id_rsa.pub
│ ├── key.pem
│ ├── server-key.pem
│ └── server.pem
├── demo2
$ docker volume create --name demo_volume
$ docker run -v demo_volume:/foo ubuntu touch /foo/test
# map to another image
$ docker run -v demo_volume:/bar ubuntu ls -al /bar
# STORAGE PLUGINS
# (flocker, nfs, convoy, glusterfs, rex-ray, ...)
$ docker volume create --driver nfs --name localhost/demo
$ docker run --rm -v localhost/demo:/foo ubuntu ls -al /foo
$ docker inspect -f {{.Volumes}} ecbada822b
map[/data:/var/lib/docker/vfs/dir/cde167197ccc3e138a14f1a4f7c0d65b32cecbada822b0db4cc92e79059437a9]
# touch it from the host
sudo touch /var/lib/docker/vfs/dir/cde167197...37a9/test-file
# Dockerfile
VOLUME /data
# Data only container + volumes from
$ docker run --name dbdata postgres echo "Data-only container for postgres"
$ docker run -it --volumes-from dbdata debian /bin/bash
# Perform a backup
$ docker run --rm --volumes-from dbdata -v $(pwd):/backup debian \
tar cvf /backup/backup.tar /var/lib/postgresql/data
$ docker network
docker: "network" requires a minimum of 1 argument.
See 'docker network --help'.
Usage: docker network [OPTIONS] COMMAND [OPTIONS]
Commands:
ls List all networks
rm Remove a network
create Create a network
connect Connect container to a network
disconnect Disconnect container from a network
inspect Display detailed network information
$ docker network create demo
$ docker network ls
NETWORK ID NAME DRIVER
b7c9b6d0c133 demo bridge
bfc56a57191f none null
0a07e021ffd5 host host
a69e42ce961a bridge bridge
$ docker run --rm -ti \
--name hostA \
debian bash
$ docker run --rm -ti \
--name hostB \
--net demo \
debian bash
$ docker run --rm -ti \
--name hostC \
--net demo \
debian bash
$ ping -c 1 hostB
ping: unknown host
$ ping -c 1 hostC
PING hostC (172.21.0.4): 56 data bytes
64 bytes from 172.21.0.4: icmp_seq=0 ttl=64 time=0.069 ms
--- hostC ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.069/0.069/0.069/0.000 ms
$ ping -c 1 hostA
ping: unknown host
$ docker network connect demo hostA
$ ping -c 1 hostB
PING hostB (172.21.0.3): 56 data bytes
64 bytes from 172.21.0.3: icmp_seq=0 ttl=64 time=0.069 ms
--- hostB ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.069/0.069/0.069/0.000 ms
$ ping -c 1 hostC
PING hostC (172.21.0.4): 56 data bytes
64 bytes from 172.21.0.4: icmp_seq=0 ttl=64 time=0.069 ms
--- hostC ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.069/0.069/0.069/0.000 ms
$ ping -c 1 hostA
PING hostA (172.21.0.2): 56 data bytes
64 bytes from 172.21.0.2: icmp_seq=0 ttl=64 time=0.069 ms
--- hostA ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.069/0.069/0.069/0.000 ms
$ docker run --rm -ti \
--name hostA \
debian bash
$ docker run --rm -ti \
--name hostB \
--net demo \
debian bash
$ docker run --rm -ti \
--name hostC \
--net demo \
debian bash
# [app]
# |
# FRONT |
# -----------------[rabbitmq]-----
# BACK |
# |
# [worker]
$ docker network create front
$ docker network create back
$ docker network connect front app
$ docker network connect front rabbitmq
$ docker network connect back rabbitmq
$ docker network connect back worker
FROM jpetazzo/dind
MAINTAINER Petr Michalec <epcim@apealive.net>
RUN apt-get update
RUN apt-get install -qqy curl \
sudo \
git \
mercurial \
subversion \
ca-certificates \
locales \
jq
RUN echo 'en_US.UTF-8 UTF-8'>>/etc/locale.gen
RUN locale-gen en_US.UTF-8
ENV LANG en_US.UTF-8
ENV DEBIAN_FRONTEND noninteractive
## CHEF DK ###########################
RUN curl -L https://www.opscode.com/chef/install.sh | sudo bash -s -- -P chefdk
ENV PATH /opt/chefdk/bin:/.chefdk/gem/ruby/2.1.0/bin:/opt/chefdk/embedded/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
# Make Chef DK the primary Ruby/Chef development environment.
RUN echo 'eval "$(chef shell-init bash)"' >> ~/.bash_profile
RUN eval "$(chef shell-init bash)"
RUN chef gem install kitchen-docker
RUN chef gem install kitchen-openstack
RUN chef gem install chef-sugar
RUN chef gem install chef-rewind
RUN chef gem install serverspec
RUN chef gem install infratester
# berks pre-fetch some common soup of cookbooks
RUN mkdir /tmp/fake_cookbook; cd $_
RUN echo "name 'fake_cookbook'\nmaintainer 'fake_cookbook'\nlicense 'fake_cookbook'\ndescription 'fake_cookbook'\nversion '0.0.1'" > metadata.rb
RUN echo "source 'https://supermarket.chef.io'\nmetadata\n\n" > Berksfile
RUN echo "cookbook '7-zip'\ncookbook 'apache2'\ncookbook 'apt'\ncookbook 'ark'\ncookbook 'bluepill'\ncookbook 'build-essential'\ncookbook 'certificate'\ncookbook 'chef-client'\ncookbook 'chef_handler'\ncookbook 'chef_ruby'\ncookbook 'chef-sugar'\ncookbook 'chef-vault'\ncookbook 'cron'\ncookbook 'database'\ncookbook 'device-mapper'\ncookbook 'git'\ncookbook 'minitest-handler'\ncookbook 'modules'\ncookbook 'ncurses'\ncookbook 'nginx'\ncookbook 'ntp'\ncookbook 'ohai'\ncookbook 'openssh'\ncookbook 'openssl'\ncookbook 'packagecloud'\ncookbook 'pacman'\ncookbook 'perl'\ncookbook 'rbenv'\ncookbook 'readline'\ncookbook 'resolver'\ncookbook 'resource-control'\ncookbook 'rsyslog'\ncookbook 'ruby'\ncookbook 'ruby_build'\ncookbook 'runit'\ncookbook 'subversion'\ncookbook 'sudo'\ncookbook 'sysctl'\ncookbook 'system'\ncookbook 'ulimit'\ncookbook 'users'\ncookbook 'windows'\ncookbook 'xml'\ncookbook 'yum'\ncookbook 'yum-epel'\ncookbook 'zlib'\n" >> Berksfile
RUN chef exec berks install
RUN cd -
## FIX UP'S ##########################
RUN chmod -R 0440 /etc/sudoers
RUN chmod -R 0440 /etc/sudoers.d
# workaround (drone.io has no way yet to modify this image before git clone happens)
RUN git config --global http.sslverify false
VOLUME /var/lib/docker
CMD ["wrapdocker"]
$ docker build -t myapp .
Sending build context to Docker daemon 92.67 kB
Step 1 : FROM ruby:2.3.0
---> 7ca70eb2dfea
Step 2 : MAINTAINER luka.dornhecker@gmail.com
---> Running in 61ffc257656d
---> 824c72ddfb87
Removing intermediate container 61ffc257656d
Step 3 : RUN apt-get update -qq && apt-get install -y build-essential nodejs && apt-get clean && rm -rf /var/lib/apt/lists/*
---> Running in 3ee402db540d
Reading package lists...
[...]
---> 445160b5bebb
Removing intermediate container 3ee402db540d
Step 4 : RUN mkdir -p /app
---> Running in 236bd3d78d9a
---> 9441393c6cc8
Removing intermediate container 236bd3d78d9a
Step 5 : WORKDIR /app
---> Running in d29b66063ee8
---> 4e17146f2c38
Removing intermediate container d29b66063ee8
Step 6 : COPY Gemfile Gemfile.lock /app/
---> fcdc11b05832
Removing intermediate container ffb8773a8acb
Step 7 : RUN gem install bundler && bundle install --jobs 20 --retry 5
---> Running in fbeaf1f023b2
Successfully installed bundler-1.11.2
1 gem installed
[...]
Removing intermediate container fbeaf1f023b2
Step 8 : ADD . /app
---> a53ec59090f6
Removing intermediate container e80472550747
Step 9 : ENV PORT 3000
---> Running in 8d68e64db143
---> bceed89e3c6e
Removing intermediate container 8d68e64db143
Step 10 : EXPOSE $PORT
---> Running in 1821fcb3e3d7
---> 8ea96dc0b9f8
Removing intermediate container 1821fcb3e3d7
Step 11 : CMD bundle exec rails -p $PORT
---> Running in f841cd21df61
---> bb540b0da5dc
Removing intermediate container f841cd21df61
Successfully built bb540b0da5dc
$ docker-compose
Define and run multi-container applications with Docker.
Usage:
docker-compose [-f=<arg>...] [options] [COMMAND] [ARGS...]
docker-compose -h|--help
Commands:
build Build or rebuild services
help Get help on a command
kill Kill containers
logs View output from containers
pause Pause services
port Print the public port for a port binding
ps List containers
pull Pulls service images
restart Restart services
rm Remove stopped containers
run Run a one-off command
scale Set number of containers for a service
start Start services
stop Stop services
unpause Unpause services
up Create and start containers
migrate-to-labels Recreate containers to add labels
version Show the Docker-Compose version information
version: '2'
services:
db:
image: postgres
web:
build: .
command: python manage.py runserver 0.0.0.0:8000
volumes:
- .:/code
ports:
- "8000:8000"
depends_on:
- db
app:
build: docker/app
volumes:
- .:/srv
net: app
db:
image: postgres
net: app
app:
build: docker/app
volumes:
- .:/srv
db:
image: postgres
OR
$ docker-compose --x-networking up
version: '2'
networks:
proxy-tier:
external:
name: nginx-proxy
services:
db:
image: postgres
web:
build: .
command: python manage.py runserver 0.0.0.0:8000
volumes:
- .:/code
ports:
- "8000:8000"
depends_on:
- db
lb:
image: tutum/haproxy
links:
- web
ports:
- "80:80"
environment:
- BACKEND_PORT=8000
- BALANCE=roundrobin
docker-compose up -d
docker-compose ps
docker-compose logs
# Scale up
docker-compose scale web=5
docker-compose up --force-recreate -d
version: '2'
networks:
proxy-tier:
external:
name: nginx-proxy
services:
db:
image: postgres
web:
build: .
command: python manage.py runserver 0.0.0.0:8000
volumes:
- .:/code
ports:
- "8000:8000"
depends_on:
- db
nginx:
image: nginx
container_name: nginx
ports:
- "80:80"
- "443:443"
volumes:
- "/etc/nginx/conf.d"
- "/etc/nginx/vhost.d"
- "/usr/share/nginx/html"
- "../../../volumes/proxy/certs:/etc/nginx/certs:ro"
networks:
- proxy-tier
nginx-gen:
image: jwilder/docker-gen
container_name: nginx-gen
volumes:
- "/var/run/docker.sock:/tmp/docker.sock:ro"
- "../../../volumes/proxy/templates/nginx-compose-v2.tmpl:/etc/docker-gen/templates/nginx.tmpl:ro"
volumes_from:
- nginx
entrypoint: /usr/local/bin/docker-gen -notify-sighup nginx -watch -only-exposed -wait 5s:30s /etc/docker-gen/templates/nginx.tmpl /etc/nginx/conf.d/default.conf
letsencrypt-nginx-proxy-companion:
image: jrcs/letsencrypt-nginx-proxy-companion
container_name: letsencrypt-nginx-proxy-companion
volumes_from:
- nginx
volumes:
- "/var/run/docker.sock:/var/run/docker.sock:ro"
- "../../../volumes/proxy/certs:/etc/nginx/certs:rw"
environment:
- NGINX_DOCKER_GEN_CONTAINER=nginx-gen
# enable completition
https://github.com/docker/machine/\
blob/master/contrib/completion/\
bash/docker-machine.bash
(~ 5 min)
FIXME
Cross host dependencies
Service Discovery
Service Discovery
Service Discovery
(~ 5 min)