CI/CD with Docker, Jenkins and Digital Ocean

Presented by

Space allocated by

http://majisti.com

Web Apps Experts

http://ctrlweb.ca

Welcome

Steven Rosato, B. Ing

President & Founder @ Solutions Majisti

Steven Rosato

http://majisti.com

Small company near Montreal (Boisbriand) focussed on development with Symfony and React/Redux.

/solutions.majisti

  • We are the hitmen developers of robust & maintainable Web platforms
  • We program with React & Redux
  • We program with Symfony
  • We automatically test all our code
  • We coach people

The developers you wish you had hired first!

Before we start...

New meetup name!

Before we start...

17h30 - 19h00 Networking 

19h00 - 20h00 Presentation

After... Pub Victoria

Today

Objectives

  • CI/CD what is it and why for?
  • Quick overview of Docker and Docker Compose for complete project isolation
  • Jenkins CI
  • Digital Ocean VPS Provider
  • Build that automatic machine!

The Mentra of the CI/CDer

??

a Boob

a Guru

Don't be a boob

Continous Integration

Hey the testing phase should be automated too by the way!!

Continous Integration

  • Integrate code to a shared environment several times a day
  • Each check-in is verified by an automated build
  • Allows to detect problems early

Continous Delivery

  • Short cycle releases of the current codebase
  • Software can be released at any time
  • Aims at testing and releasing software faster and more frequently

Fixing a bug in production

You should CI/CD when...

Our setup at Majisti

Vagrant local environment

VCS

CI Tool (and deploy tool for now)

VPS Cloud Provider / Docker Host

Hosted on

Pulls from

A small overview on costs

A small overview on costs

You should CI/CD when

Psst... Guru tip #1

  • Code maintainability is a concern
  • Client needs will evolve

Your project will need maintenance

Guru tip #2

Your client's needs will evolve

The Recipe

No trans fat I swear

Step 1

Isolate your project from global dependencies

https://github.com/majisti/boilerplate

Docker Compose

Step 1

Isolate your project from global dependencies

And if I don't??

You will have an easier (or harder) time with

Virtual Machines vs Docker

Ok what does that mean isolate dependencies?

$ sudo apt-get install npm
$ sudo apt-get install php5 apache2

$ npm install -g gulp
$ gulp watch
$ php app/console awesome:command

Wait and there is so much more!!

Behat Gulp NodeJs PHP Apache / Nginx
Ruby FPM XDebug PECL Python
Bzip2 g++ libicu-dev MySQL ...

From

To

$ docker-compose run --rm php php -v
$ docker-compose run --rm node npm install
$ docker-compose run --rm node gulp watch

Not only that, but you need to install that on all your servers...

dev.majisti.com

staging.majisti.com

majisti.com #1

Developer #1

Developer #2

Developer #n ...

majisti.com #2

majisti.com #n ...

And now multiple projects with different requirements

  • PHP5
  • Node 4.2
  • Python 2.3
  • Apache2
  • PHP5.3
  • Node 4.8
  • Python 3
  • nginx
  • PHP7
  • Node 5.5
  • RVM
  • ...

...

Ok so... isolating global dependencies lets you:

  • Run multiple projects with different versions of your tools
  • Deploy your solution on multiple environments, without the need to install them on each machine
  • Have a easier time upgrading in the future
  • Each developer have the exact same tools and can add/remove/upgrade them at any time
  • The upgrades bubbles up to the production!

But the most important

It lets you have the same environment no matter where you are and that unlocks the power to do CI/CD that is maintainable without hassle

Dockerfile Example

FROM php:7.1.0RC6-fpm

# Environment variable
ENV APCU_VERSION 5.1.2
ENV APCU_BC_VERSION 1.0.0

# Dependencies
RUN apt-get update \
    && apt-get install -y \
        libpq-dev \
        libicu-dev \
        zlib1g-dev \
        libfreetype6-dev \
        libjpeg62-turbo-dev \
        libpng12-dev \
        git \
    && docker-php-ext-configure gd --with-freetype-dir=/usr/include/ --with-jpeg-dir=/usr/include/ \
    && docker-php-ext-install intl mbstring pgsql pdo_pgsql zip gd \
    && apt-get clean \
    && rm -r /var/lib/apt/lists/*

# Xdebug
RUN pecl install -o -f xdebug-2.5.0 \
    && rm -rf /tmp/pear

# Configuration
COPY php.ini /usr/local/etc/php/php.ini
COPY xdebug.ini /usr/local/etc/php/conf.d/xdebug.ini
COPY entrypoint.sh /usr/local/bin/entrypoint.sh

ENTRYPOINT ["/usr/local/bin/entrypoint.sh"]

Docker Compose Example

nginx:
    image: 'nginx:1.9'
    volumes:
        - ./docker/nginx/app.conf:/etc/nginx/conf.d/default.conf:ro
    volumes_from:
        - data
    links:
        - fpm
    environment:
      VIRTUAL_HOST: majisti.boilerplate

fpm:
    build: 'docker/fpm'
    volumes_from:
        - data

node:
    build: 'docker/node'
    volumes_from:
        - data

php:
    build: 'docker/php'
    volumes_from:
        - data

data:
    image: busybox
    volumes:
    - .:/var/www/html
    - ~/.composer:/var/www/.composer

Step 2

Create a build file

  • Gulp file / Grunt file / Webpack for frontend
  • Phing / Makefile / Ant / Gradle / Maven / etc. for backend
DC=docker-compose
PHP=$(DC) run --rm php
NODE=$(DC) run --rm node

COMPOSER=$(PHP) php -n -d extension=zip.so /usr/local/bin/composer

ci: all code-fix test
all: configure start vendors-install node-install assets
restart: stop start
test: test-prepare test-integration test-acceptance

start:
	$(DC) up -d

stop:
	$(DC) kill
	$(DC) rm -vf

node-install:
	$(NODE) npm install

assets:
	$(NODE) bin/gulp

vendors-install:
	$(COMPOSER) install --no-interaction --prefer-dist

vendors-update:
	$(COMPOSER) update

code-fix:
	$(PHP) php -n bin/php-cs-fixer fix --no-interaction

Makefile example

Step 3

Install Jenkins

$ docker run -p 8080:8080 -p 50000:50000 -v /your/home:/var/jenkins_home jenkins

You will need to install the Digital Ocean Jenkins plugin. We have forked the repo and added support for volume mapping (more on that later)

Or however you want to install it.

Step 3

Install the Digital Ocean Jenkins Plugin

http://bit.ly/2m14UAe

Download Jenkins Plugin here:

https://github.com/majisti/digitalocean-plugin/tree/feature/volume-support

Fork available here:

Step 4

Configure Digital Ocean

  1. Create a Docker droplet
  2. Create a volume
  3. Attach that volume to the droplet
  4. Install Docker Compose
  5. Modify /etc/fstab to map to that volume
  6. Modify Docker home directory
  7. Make a named snapshot from that Droplet
  8. Destroy the droplet

Step 4

Configure Digital Ocean

Installing Docker Compose

sudo apt-get install -y curl
curl -L https://github.com/docker/compose/releases/download/1.8.0/docker-compose-`uname -s`-`uname -m` \
    > /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose
docker-compose -v #verify version

Step 4

Configure Digital Ocean

Attaching the volume

# list available volumes

ls -l /dev/disk/by-id/

# outputs

lrwxrwxrwx 1 root root  9 Feb 20 17:01 scsi-0DO_Volume_jenkins-data -> ../../sda


# modify /etc/fstab

/dev/disk/by-id/scsi-0DO_Volume_jenkins-data /mnt/jenkins-data ext4 defaults,nofail,discard 0 0

Step 4

Configure Digital Ocean

Modify Docker home directory

vi /etc/default/docker

#locate this line and change the directory

DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4 -g /mnt/jenkins-data/docker"

You can now create a snapshot and destroy the droplet

This way, docker image will not be built each time an instance of the VM is created

Step 5

Back to Jenkins

  1. Configure the Digital Ocean plugin to connect with Digital Ocean
  2. Create your build
  3. And then..
$ make ci

That's it for the CI aspect!

The plugin automatically destroys Droplets after the build, which is what saves costs

Step 6

Poor man's continuous deployment

  1. After build success, archive the build in a .tgz
  2. Use build promotion to start a deployment build
  3. Configure Jenkins to SSH into the VPS and upload the changes
  4. Make it restart docker
  5. And then..

That's it for the CD aspect!

$ make restart

Step 7

Commit some code

$ git commit -am"Some code!!"
  • Automatically triggers the build
  • When successful, automatically deploys to the dev server

Once satisfied we either have automatic staging/prod release or manually triggered by the CI build with one button click.

This is called parametrized builds

That's it!

Left to explore

  • Automatic deployment with a tool such as Capistrano
  • Automatic fallback to older version by using symlink switch in case of failure, instead of redeploying a build
  • Automatic scaling of ressources for your Docker Host.

Conclusion

Conclusion

http://booborguru.com

http://majisti.com

Read us ranting

  • Symfony & React Web apps dev
  • Consultation with video recording
  • Friendly expert advice

Come blog with us

Web Apps Experts

Made with Slides.com