CI/CD Workflow
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2939957/docker-jenkins.png)
Using
Jenkins & Docker
Teerapat Khunpech
ball@engineerball.com
Teerapat Khunpech
@engineerball
https://github.com/engineerball
https://engineerball.com
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2973182/Octocat.png)
ball@engineerball.com
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2990209/imageedit_2_9772924523.gif)
Outline |
---|
Basic CI/CD workflow |
Basic Docker |
Docker setup & config |
Jenkins setup & config |
CI/CD Workflow |
Basic CI/CD Workflow
Continuous Integration
http://www.mindtheproduct.com/2016/02/what-the-hell-are-ci-cd-and-devops-a-cheatsheet-for-the-rest-of-us/
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2990223/409-images-for-snap-blog-postedit_image1.png)
Continuous Delivery
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2990221/409-images-for-snap-blog-postedit_image3-auto.png)
http://www.mindtheproduct.com/2016/02/what-the-hell-are-ci-cd-and-devops-a-cheatsheet-for-the-rest-of-us/
Jenkins
- CI/CD tools
- Easy installation
- Rich plugins
- Distributed build
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2951781/jenkins.png)
CI/CD Workflow
Dev
Commit
Build
Test
Stage
Deploy
Production
What is it?
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2952056/pasted-from-clipboard.png)
Build, Ship and Run
Any App, Anywhere
Docker
VM v Docker
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2975967/pasted-from-clipboard.png)
Docker Basics
Docker Image
The basis of a Docker container
Docker Container
The standard unit in which the application service resides
Docker Engine
Creates, ships and runs Docker containers deployable on physical or virtual host locally, in a datacenter or cloud service provider
Docker Registry
On-premises registry for image storing and collaboration
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2976027/Image_2.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2976029/Image_3.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2976031/Image_4.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2976035/Image_5.png)
Docker Solutions
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2976093/docker-images.png)
Docker CLI
docker build | # Build an image from Dockerfile |
docker images | # List all images on a Docker host |
docker run | # Run an image |
docker ps | # List all running and stopping instances |
docker stop | # Stop running instances |
docker rm | # Remove an instance |
docker rmi | # Remove an image |
Docker Architecture
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2976104/docker-architechture.png)
Jenkins & Docker
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2976303/Image_37.png)
What is it?
It turns a pool of Docker host into a single
Native clustering for Docker
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2952087/docker-swarm-hero2.png)
Docker Swarm
Docker Architecture
Swarm Manager
Discovery Service
Swarm Node
Docker Engine
Swarm Node
Docker Engine
Swarm Node
Docker Engine
Docker Client
What is it?
Service discovery, Key/Value Storage
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2952116/logo_large-475cebb0.png)
Consul
What is it?
Tool for running multiple-container Docker application
Great with CI Workflows
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2952083/Compose.png)
Docker Compose
---
version: "2"
services:
redis:
image: redis
expose: ["6379"]
db:
image: postgres:9.4
voting-app:
image: engineerball/voting-app
ports:
- "5000:80"
result-app:
image: engineerball/result-app
ports:
- "5001:80"
worker:
image: engineerball/worker
$ docker-compose up
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2976205/Cont-icon-500-01.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2976205/Cont-icon-500-01.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2976205/Cont-icon-500-01.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2976205/Cont-icon-500-01.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2976205/Cont-icon-500-01.png)
Workshop
I : Docker Swarm Setup
II : Jenkins Setup
III : Repositories & Webhooks Setup
IV : Docker Registry
V : The Build Jobs
VI : The Deploy Job
VII : Deploying App
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2976770/Image_72.png)
CI/CD Workflow
LAB Architecture
node-1
node-2
node-3
Docker engine
Docker engine
Docker engine
Swarm Manager
Consul
Jenkins master
Swarm Node
Swarm Node
Jenkins slave
node-4
Docker engine
Swarm Node
(Git Server)
192.168.56.101
192.168.56.102
192.168.56.103
192.168.56.104
Vagrant
# Clone vagrant file
$ git clone https://gist.github.com/6b83a6d3ceecf6aa919efa40aa194f7f.git devops
# Enter directory
$ cd devops
# Start VM
$ vagrant up
# Acccess VM
$ vagrant ssh node-1
$ vagrant ssh node-2
$ vagrant ssh node-3
$ vagrant ssh node-4
# Stop VM
$ vagrant halt
# Delete VM
$ vagrant destroy
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2973487/Vagrant.png)
Task I
Docker Swarm Set up
Install Docker
all node
$ sudo apt-get install -y git
$ git clone https://gist.github.com/257fbfac796be900bf5171b307a062c1.git devops-cicd && cd devops-cicd
$ chmod +x install-docker.sh
$ sudo sh install-docker.sh
Edit Docker conf
all node
DOCKER_OPTS="-H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock \
--cluster-store=consul://192.168.56.101:8500/network \
--cluster-advertise=eth1:2375"
Edit file : /etc/default/docker
Restart service :
$ sudo /etc/init.d/docker restart
Start Consul
@node-1
$ sudo docker run -d -p 8500:8500 \ --name=consul progrium/consul \ -server -bootstrap
Create Swarm
@node-1
$ sudo docker run -d -p 4000:4000 swarm manage -H :4000 \
--replication --advertise 192.168.56.101:4000 \
consul://192.168.56.101:8500
Join Swarm node
@node-1
$ sudo docker run -d swarm join \ --advertise=192.168.56.102:2375 \ consul://192.168.56.101:8500
$ sudo docker run -d swarm join \ --advertise=192.168.56.103:2375 \ consul://192.168.56.101:8500
$ sudo docker run -d swarm join \ --advertise=192.168.56.104:2375 \ consul://192.168.56.101:8500
Confirm the cluster
@node-1
$ sudo docker -H :4000 info
Server Version: swarm/1.2.5
Role: primary
Nodes: 3
node-2: 192.168.56.102:2375
node-3: 192.168.56.103:2375
node-4: 192.168.56.104:2375
Confirm the cluster
@node-1
$ sudo docker -H :4000 version
Client:
Version: 1.12.1
API version: 1.24
Go version: go1.6.3
Git commit: 23cf638
Built: Thu Aug 18 05:22:43 2016
OS/Arch: linux/amd64
Server:
Version: swarm/1.2.5 <<<<<<<<<<<<<<<<<<<<<<<
API version: 1.22
Go version: go1.5.4
Git commit: 27968ed
Built: Thu Aug 18 23:10:29 UTC 2016
OS/Arch: linux/amd64
Install Docker Compose
@node-2
$ sudo curl -L https://github.com/docker/compose/releases/download/1.8.0/docker-compose-`uname -s`-`uname -m` | sudo tee /usr/local/bin/docker-compose > /dev/null
$ sudo chmod +x /usr/local/bin/docker-compose
Jenkins Setup
Task II
Start Jenkins Master
@node1
$ sudo docker run -d -p 8080:8080 \
--name jenkins-master jenkins
Jenkins credential
@node1
$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/vagrant/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/vagrant/.ssh/id_rsa.
Your public key has been saved in /home/vagrant/.ssh/id_rsa.pub.
The key fingerprint is:
bb:93:37:94:b7:80:41:bd:33:54:5f:7d:34:f6:e2:d7 vagrant@node-1
The key's randomart image is:
+--[ RSA 2048]----+
| . .. ++|
| . o ...=|
| . . . o o|
| . + . ..|
| So + . E|
| ..+ . . |
| .o o . |
| o.o . |
| .o . |
+-----------------+
Jenkins credential
@node1
$ ssh-copy-id vagrant@192.168.56.102
vagrant@192.168.56.102's password:
<type vagrant>
Jenkins Slave setup
@node2
$ sudo apt-get install -y default-jre
Jenkins 2.0 setup
@node1
Open browser and go to
http://192.168.56.101:8080
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2974310/Image_23.png)
Jenkins 2.0
@node-1
setup
@node1
$ sudo docker logs jenkins-master
*************************************************************
Jenkins initial setup is required. An admin user has been created and a password generated.
Please use the following password to proceed to installation:
9fc503eb2bf64126bb4705ae100f7e73
This may also be found at: /var/jenkins_home/secrets/initialAdminPassword
*************************************************************
--> setting agent port for jnlp
--> setting agent port for jnlp... done
Jenkins 2.0 setup
@node1
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2974319/Image_25.png)
Jenkins 2.0 setup
@node1
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2974336/Image_27.png)
Jenkins 2.0 install plugins
@node1
Install plugins
Go to menu
Manage Jenkins Manage Plugins
Available
- Cloudbees Docker build
and Publish plugin - Github plugin
- Gogs plugin
Jenkins 2.0
Add slave node
@node1
Set Node name
Go to menu
Manage Jenkins Manage Nodes New Node
"slave"
Jenkins 2.0
Add slave node
@node1
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2974362/Image_31.png)
Jenkins 2.0
Add slave node
@node1
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2975749/Image_33.png)
$ cat ~/.ssh/id_rsa
@node-1
Jenkins 2.0
Add slave node
@node1
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2975940/pasted-from-clipboard.png)
Repositories
&
Webhooks
Task III
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2976770/Image_72.png)
CI/CD with Jenkins
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2976770/Image_72.png)
CI/CD with Jenkins
Gogs
Repositories
&
Webhooks
Gogs setup
@node4
# Start MySQL container $ sudo docker run -d --name=mysql -p 3306:3306 -e MYSQL_PASS="password" -e ON_CREATE_DB="gogs_db" tutum/mysql # Create local directory for volume. $ sudo mkdir -p /var/gogs # Use `docker run` for the first time. $ sudo docker run -d --name=gogs -p 10022:22 \ -p 10080:3000 -v /var/gogs:/data gogs/gogs
Gogs setup
@node4
http://192.168.56.104:10080
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2976366/Image_39.png)
Gogs setup
@node4
http://192.168.56.104:10080
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2976369/Image_41.png)
Create Repo
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2976374/Image_43.png)
Create Repo
Create Repo
Commit
CODE
$ git clone https://github.com/engineerball/example-voting-app.git
$ cd example-voting-app
$ git remote set-url origin http://192.168.56.104:10080/devops/example-voting-app.git
$ git add .
$ git commit -m "First commit"
$ git push origin master
Create Repo
Commit
CODE
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2976415/Image_45.png)
Task IV
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2990320/Image_4.png)
The Build Jobs
Task V
The Voting App
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2990114/IGEM1OBUtW_rARFvGxZasrnTzqrqzN7Y9aBZTaXoQ8Q_.png)
▷ A Python webapp which lets you vote between two options
▷ A Redis queue which collects new votes
▷ A Java worker which consumes votes and stores them in…
▷ A Postgres database backed by a Docker volume
▷ A Node.js webapp which shows the results of the voting in real time
Voting App
# Using official python runtime base image
FROM python:2.7
# Set the application directory
WORKDIR /app
# Install our requirements.txt
ADD requirements.txt /app/requirements.txt
RUN pip install -r requirements.txt
# Copy our code from the current folder to /app inside the container
ADD . /app
# Make port 5000 available for links and/or publish
EXPOSE 80
# Define our command to be run when launching the container
CMD ["python", "app.py"]
Result App
FROM node:0.10
RUN mkdir /app
WORKDIR /app
ADD package.json /app/package.json
RUN npm install && npm ls
RUN mv /app/node_modules /node_modules
ADD . /app
ENV PORT 80
EXPOSE 80
CMD ["node", "server.js"]
Worker
FROM java:7
RUN apt-get update -qq && apt-get install -y maven && apt-get clean
WORKDIR /code
ADD pom.xml /code/pom.xml
RUN ["mvn", "dependency:resolve"]
RUN ["mvn", "verify"]
# Adding source, compile and package into a fat jar
ADD src /code/src
RUN ["mvn", "package"]
CMD ["/usr/lib/jvm/java-7-openjdk-amd64/bin/java", "-jar", "target/worker-jar-with-dependencies.jar"]
Worker
FROM java:7
RUN apt-get update -qq && apt-get install -y maven && apt-get clean
WORKDIR /code
ADD pom.xml /code/pom.xml
RUN ["mvn", "dependency:resolve"]
RUN ["mvn", "verify"]
# Adding source, compile and package into a fat jar
ADD src /code/src
RUN ["mvn", "package"]
CMD ["/usr/lib/jvm/java-7-openjdk-amd64/bin/java", "-jar", "target/worker-jar-with-dependencies.jar"]
docker-compose.yml
version: "2"
services:
redis:
image: redis
expose: ["6379"]
db:
image: postgres:9.4
voting-app:
image: $DTR/voting-app:$GIT_COMMIT
ports:
- "5000:80"
result-app:
image: $DTR/result-app:$GIT_COMMIT
ports:
- "5001:80"
worker:
image: $DTR/worker:$GIT_COMMIT
Create jobs (build-voting-app)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2976420/Image_47.png)
Select target to run
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2976438/Image_51.png)
Add SCM
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2976430/Image_49.png)
http://192.168.56.104:10080/devops/example-voting-app.git
Build Triggers
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2989466/Image_2.png)
App Build
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2976445/Image_53.png)
Docker hub credential
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2976453/Image_55.png)
App Build (Advanced)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2976471/Image_60.png)
Create jobs
- New Item
- Copy existing Item
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2976478/Image_62.png)
Create jobs
- Edit build
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2976482/Image_64.png)
Create jobs
- Edit build
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2976488/Image_66.png)
Create jobs
- Repeat step again
- Change
"build-result-app" to
"build-worker"
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2976478/Image_62.png)
Enable build triggers
- Go to Gogs http://192.168.56.104:10080
- Go to repository and Settings (http://192.168.56.104:10080/devops/example-voting-app/settings)
- At Webhooks click "Add Webhook"
- Select "Gogs"
- Fill information
-
Payload => "http://192.168.56.101:8080/gogs-webhook/?job=build-voting-app"
-
- Add another Webhooks and fill information
-
Payload => "http://192.168.56.101:8080/gogs-webhook/?job=build-result-app"
-
Payload => "http://192.168.56.101:8080/gogs-webhook/?job=build-worker"
-
Enable build triggers
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2976709/Image_68.png)
Enable build triggers
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2976716/Image_70.png)
The Deploy Job
Task VI
The Deploy Job
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2976848/Image_74.png)
Restrict run target
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2979937/Image_75.png)
Config SCM
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2976430/Image_49.png)
Config Build Triggers
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2979942/Image_76.png)
Config Build
export DOCKER_HOST=192.168.56.101:4000 # Docker swarm manager
export DTR=engineerball # DockerHub ID
docker-compose -f vote-apps/docker-compose.yml stop voting-app result-app worker
docker-compose -f vote-apps/docker-compose.yml rm -f
docker-compose -f vote-apps/docker-compose.yml pull voting-app result-app worker
docker-compose -f vote-apps/docker-compose.yml up -d
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2991159/Image_77.png)
vote-apps-deploy
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2979994/Image_79.png)
Deploying App
Task VII
Clone git
Make changes
Push git
Running App
@node-1
$ docker -H :4000 ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
edd35c4ad17d engineerball/result-app:99883f "node server.js" 18 minutes ago Up 18 minutes 192.168.56.103:5001->80/tcp node-3/voteapps_result-app_1
f4a6383f76ca engineerball/voting-app:99883f "python app.py" 18 minutes ago Up 18 minutes 192.168.56.104:5000->80/tcp node-4/voteapps_voting-app_1
9a885721e2cf engineerball/worker:99883f "/usr/lib/jvm/java-7-" 18 minutes ago Up 18 minutes node-3/voteapps_worker_1
fee6b1763e8c redis "docker-entrypoint.sh" 57 minutes ago Up 57 minutes 6379/tcp node-3/voteapps_redis_1
94388eb94323 postgres:9.4 "/docker-entrypoint.s" 57 minutes ago Up 57 minutes 5432/tcp node-3/voteapps_db_1
Running App
@node-1
result-app 192.168.56.103:5001->80/tcp voting-app 192.168.56.104:5000->80/tcp
Voting App
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2980047/Image_12.png)
Running App
@node-1
Result App
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2980041/Image_11.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/560276/images/2980059/1n7eoe.jpg)
CICD Workflow
By Teerapat Khunpech
CICD Workflow
- 2,344