CI/CD with Drupal

Problem

  • CI/CD tools exist for the popular Drupal hosting platforms
  • Difficult to piece together incomplete tutorials from off the Drupal island
  • Need a practical guide to self-hosted Drupal CI/CD

Prerequisites

  • Drupal
  • Composer
  • Git
  • Behat
  • Command Line Bash
  • Docker
  • SSH
  • Private-Public Keys

James

Candan /jənˈdən/

drupal.org/u/jcandan

github.com/jcandan

Software developer 10 yrs

From L.A., live in Chattanooga

twitter.com/jcandan82

  • Developer workflow rescues
  • Technology strategy
  • Data visualizations
  • CMS migrations

Agenda

  • 10,000 ft view
  • Tools
  • See it in action
  • Gotchas

CI/CD Process

<?php

/**
 * Say what's up to the world.
 */
class BlastIt {
  /**
   * Outputs a simple statement.
   */
  public function say_hello() {
    echo "hello, world";
  }
}

Code pushed to Gitlab

dev

stage

prod

Gitlab CI runs pipeline jobs on Gitlab runner

Gitlab runner spins up docker container to build, test, and deploy

Container connects to destination, fires deploy scripts

All jobs pass

CI/CD Process (cont...)

Pipelines

Jobs

# .gitlab-ci.yml

stages:
  - build
  - test
  - deploy

. . .

my_test:
  stage: test
  script:
    - echo "run tests"

Test

Build

Deploy

Build

Test

Deploy

What is an Artifact?

An artifact is one of many kinds of tangible by-products produced during the development of software.

- Wikipedia

Production is an Artifact of Development

Michelle Krejci

Drupal Con New Orleans session

Leave out dev dependencies, build files, config files, etc

Development

Artifact

section {
    height: 100px;
    width: 100px;

    .class-one {
        height: 50px;
        width: 50px;

        .button {
            color: #AF7AC5;
        }
    }
}
section {
    height: 100px;
    width: 100px;
}

section .class-one {
    height: 50px;
    width: 50px;
}

section .class-one .button {
    color: #AF7AC5;
}

CI jobs produce Artifacts

A downloadable archive of the files at the end of a Job.

This is whatever you have told the job to produce.

Jobs

Tools

Docker registry

Gitlab Registry

Docker hub

SSH Task Runner

Envoy

CI Services

Gitlab CI

Build Container

Object-Oriented Thinking
JULY 16, 2014 / RODRIGO ARAÚJO

http://www.universocomputacao.com/object-oriented-thinking/

Image

Container

# Dockerfile

# Set the base image for subsequent instructions
FROM php:7.1-apache-stretch

# Update packages
RUN apt-get update

# Install PHP and composer dependencies
RUN apt-get install -qq git curl libmcrypt-dev libjpeg-dev libpng-dev libfreetype6-dev libbz2-dev unzip mysql-client

# Clear out the local repository of retrieved package files
RUN apt-get clean

# Install needed extensions
RUN docker-php-ext-install mcrypt pdo_mysql zip gd

# Install Composer
RUN curl --silent --show-error https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer

# Install Laravel Envoy
RUN composer global require "laravel/envoy=~1.0"

Gitlab Container Registry

Docker hub

Push to Registry

# @file Envoy.blade.php

# Dockerfile

FROM php:7.1-apache-stretch

# Setup the base OS
RUN apt-get update -qq \
 && apt-get install -y --no-install-recommends build-essential  \
    apt-transport-https curl ca-certificates gnupg2 apt-utils

# Update packages
RUN apt-get update

# Install PHP and composer dependencies
RUN apt-get install -yqq git libmcrypt-dev mysql-client . . .

# Install needed extensions
RUN docker-php-ext-install mcrypt pdo_mysql . . .

# Install Composer
RUN curl --silent --show-error https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer

# Install Laravel Envoy
RUN composer global require "laravel/envoy=~1.0"

Define Docker Image, Container Build, & Push

$ 
$ docker build -t some-name .

$ 
$ docker build -t registry.gitlab.com/<USERNAME>/my-project .

$ docker login registry.gitlab.com

$ 
$ docker build -t registry.gitlab.com/<USERNAME>/my-project .

$ docker login registry.gitlab.com

$ docker push registry.gitlab.com/<USERNAME>/my-project
$ docker build -t registry.gitlab.com/<USERNAME>/my-project .

$

An SSH task runner

Envoy

Uses Blade syntax to define tasks to be run on remote servers, such as

  • cloning your project from the repository,
  • installing the Composer dependencies,
  • and running Drush commands.
$ 
$ envoy run my_first_task

# @file Envoy.blade.php

$ envoy run my_first_task
[localhost]: running first task . . .
[localhost]: hello, world
$
$ envoy run my_first_task
[localhost]: running first task . . .
[localhost]: hello, world
$ envoy run my_first_task
$ envoy run my_first_task
[localhost]: running first task . . .
[localhost]: hello, world
$ envoy run my_first_task
[localhost]: running first task . . .
[localhost]: hello, world
[localhost]: hello, world
# @file Envoy.blade.php

@servers()

@task()
@endtask
# @file Envoy.blade.php

@servers(['my_local' => 'localhost'])

@task('my_first_task', ['on' => 'my_local'])
@endtask

# @file Envoy.blade.php

@servers(['my_local' => 'localhost'])

@task('my_first_task', ['on' => 'my_local'])
    echo "Running first task . . ."
@endtask

# @file Envoy.blade.php

@servers(['my_local' => 'localhost'])

@task('my_first_task', ['on' => 'my_local'])
    echo "Running first task . . ."
    echo "hello, world" >> hello.txt
@endtask

# @file Envoy.blade.php

@servers(['my_local' => 'localhost'])

@task('my_first_task', ['on' => 'my_local'])
    echo "Running first task . . ."
    echo "hello, world" >> hello.txt
    cat hello.txt
@endtask

Envoy Basics

# Envoy.blade.php

@servers()

@task()
@endtask

$

Envoy Basics (cont...)

# Envoy.blade.php

@servers(['production' => 'myuser@192.168.1.1'])

@task('my_second_task', ['on' => 'production'])
@endtask

# Envoy.blade.php

@servers(['production' => 'myuser@192.168.1.1'])

@task('my_second_task', ['on' => 'production'])
    echo "hello, world, again."
@endtask

# Envoy.blade.php

@setup
    $name = isset($name) ? $name : 'James';
@endsetup

@servers(['production' => 'myuser@192.168.1.1'])

@task('my_second_task', ['on' => 'production'])
    echo "hello, {{ $name }}"
@endtask

$ envoy run my_second_task
$ envoy run my_second_task
[production]: hello, James
$ 
$ envoy run my_second_task
[production]: hello, James
$ envoy run my_second_task --name=Billy
$ envoy run my_second_task
[production]: hello, James
$ envoy run my_second_task --name=Billy
[production]: hello, Billy

Deployment Strategies

  1. Build and rsync artifact container
  2. Commit to "artifact repo"
  3. Pull and build at destination
  4. Pull and build at symlink

Option 1:

Build and rsync artifact

No Example. Sorry.

Option 2:

Commit to "Artifact Repo"

Option 2: Commit to artifact repo

$ ls

$ ls
my-project

$
$ ls
my-project

$ ls my-project

$ ls
my-project

$ ls my-project
LICENSE			composer.json	        vendor
README.md		load.environment.php	
composer.lock		web

$ 
$ ls
my-project

$ ls my-project
LICENSE			composer.json	        vendor
README.md		load.environment.php	
composer.lock		web

$ git clone git@gitlab.com:jcandan/my-project-artifact.git

$ ls
my-project

$ ls my-project
LICENSE			composer.json	        vendor
README.md		load.environment.php	
composer.lock		web

$ git clone --origin artifact git@gitlab.com:jcandan/my-project-artifact.git

$ 
$ ls
my-project

$ ls my-project
LICENSE			composer.json	        vendor
README.md		load.environment.php	
composer.lock		web

$ git clone --origin artifact git@gitlab.com:jcandan/my-project-artifact.git

$ rsync my-project/ my-project-artifact/

$ ls
my-project

$ ls my-project
LICENSE			composer.json	        vendor
README.md		load.environment.php	
composer.lock		web

$ git clone --origin artifact git@gitlab.com:jcandan/my-project-artifact.git

$ rsync --exclude='.git' my-project/ my-project-artifact/

$ ls
my-project

$ ls my-project
LICENSE			composer.json	        vendor
README.md		load.environment.php	
composer.lock		web

$ git clone --origin artifact git@gitlab.com:jcandan/my-project-artifact.git

$ rsync --exclude='.git' --archive my-project/ my-project-artifact/

$ ls
my-project

$ ls my-project
LICENSE			composer.json	        vendor
README.md		load.environment.php	
composer.lock		web

$ git clone --origin artifact git@gitlab.com:jcandan/my-project-artifact.git

$ rsync --exclude='.git' --archive --delete my-project/ my-project-artifact/

$ 
$ ls
my-project

$ ls my-project
LICENSE			composer.json	        vendor
README.md		load.environment.php	
composer.lock		web

$ git clone --origin artifact git@gitlab.com:jcandan/my-project-artifact.git

$ rsync --exclude='.git' --archive --delete my-project/ my-project-artifact/

$ cd my-project-artifact/

$
$ ls
my-project

$ ls my-project
LICENSE			composer.json	        vendor
README.md		load.environment.php	
composer.lock		web

$ git clone --origin artifact git@gitlab.com:jcandan/my-project-artifact.git

$ rsync --exclude='.git' --archive --delete my-project/ my-project-artifact/

$ cd my-project-artifact/

$ cat .gitignore-prod > .gitignore

$ 
$ ls
my-project

$ ls my-project
LICENSE			composer.json	        vendor
README.md		load.environment.php	
composer.lock		web

$ git clone --origin artifact git@gitlab.com:jcandan/my-project-artifact.git

$ rsync --exclude='.git' --archive --delete my-project/ my-project-artifact/

$ cd my-project-artifact/

$ cat .gitignore-prod > .gitignore

$ git add .

$ 
$ ls
my-project

$ ls my-project
LICENSE			composer.json	        vendor
README.md		load.environment.php	
composer.lock		web

$ git clone --origin artifact git@gitlab.com:jcandan/my-project-artifact.git

$ rsync --exclude='.git' --archive --delete my-project/ my-project-artifact/

$ cd my-project-artifact/

$ cat .gitignore-prod > .gitignore

$ git add .

$ git commit -m "Some artifact commit message."

$ ls
my-project

$ ls my-project
LICENSE			composer.json	        vendor
README.md		load.environment.php	
composer.lock		web

$ git clone --origin artifact git@gitlab.com:jcandan/my-project-artifact.git

$ rsync --exclude='.git' --archive --delete my-project/ my-project-artifact/

$ cd my-project-artifact/

$ cat .gitignore-prod > .gitignore

$ git add .

$ git commit -m "Some artifact commit message. Maybe the job id?"

$ 
$ ls
my-project

$ ls my-project
LICENSE			composer.json	        vendor
README.md		load.environment.php	
composer.lock		web

$ git clone --origin artifact git@gitlab.com:jcandan/my-project-artifact.git

$ rsync --exclude='.git' --archive --delete my-project/ my-project-artifact/

$ cd my-project-artifact/

$ cat .gitignore-prod > .gitignore

$ git add .

$ git commit -m "Some artifact commit message. Maybe the job id?"

$ git push artifact master
$ ls
my-project

$ ls my-project
LICENSE			composer.json	        vendor
README.md		load.environment.php	
composer.lock		web

$ git clone --origin artifact git@gitlab.com:jcandan/my-project-artifact.git

$ rsync --exclude='.git' --archive --delete my-project/ my-project-artifact/

$ cd my-project-artifact/

$ cat .gitignore-prod > .gitignore

$ git add .

$ git commit -m "Some artifact commit message. Maybe the job id?"

$ git push artifact $CI_COMMIT_REF_NAME
# Dockerfile

# Set the base image for subsequent instructions
FROM php:7.1-apache-stretch

# Add PostgreSQL 10.x Apt repository
RUN echo 'deb http://apt.postgresql.org/pub/repos/apt/ stretch-pgdg main' >> /etc/apt/sources.list.d/pgdg.list
RUN apt-get update && apt-get install -yqq wget gnupg
RUN wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add -

# Install dependencies, Postgres CLI, PHP extensions, and composer
RUN apt-get update
RUN apt-get install -yqq git curl . . . postgresql-client-10 libpq-dev unzip
RUN apt-get clean
RUN docker-php-ext-install mcrypt pdo_pgsql zip gd
RUN curl --silent --show-error https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer

# Install Laravel Envoy
RUN composer global require "laravel/envoy=~1.0"

Option 2: Commit to artifact repo

# .gitlab-ci.yml

# .gitlab-ci.yml

image: registry.gitlab.com/<USER>/my-project:latest

# .gitlab-ci.yml

image: registry.gitlab.com/<USER>/my-project:latest

services:
  - postgres:10

variables:
  POSTGRES_DB: drupal8
  POSTGRES_PASSWORD: drupal8


# .gitlab-ci.yml

image: registry.gitlab.com/<USER>/my-project:latest

services:
  - postgres:10

variables:
  POSTGRES_DB: drupal8
  POSTGRES_PASSWORD: drupal8

cache:
  paths:
    - drush/contrib/
    - vendor/
    # we purposely leave out web/modules/contrib, etc.
# .gitlab-ci.yml

image: registry.gitlab.com/<USER>/my-project:latest

# Postgres service and cache configs
# . . .

stages:
  - test
  - commit
  - deploy

my_test:
  stage: test

my_commit:
  stage: commit

my_deploy:
  stage: deploy
# .gitlab-ci.yml

image: registry.gitlab.com/<USER>/my-project:latest

stages:
  - test
  - commit
  - deploy

my_test:
  stage: test
  script:
    - echo "run tests"
  artifacts:
    expire_in: 1 week
    paths:
      - web/
      - config/

my_commit:
  stage: commit

my_deploy:
  stage: deploy

# .gitlab-ci.yml

image: registry.gitlab.com/<USER>/my-project:latest

stages:
  - test
  - commit
  - deploy

my_test:
  stage: test
  script:
    - echo "run tests"

my_commit:
  stage: commit
  script:
    # Some SSH key config magic and Git configurations
    - cd ..
    - git clone --origin artifact git@gitlab.com:<USER>/my-project-artifact.git
    - rsync --archive --delete --exclude='.git' my-project/ my-project-artifact/
    - cd my-project-artifact/
    - cat .gitignore-prod > .gitignore
    - git add .
    - git commit -m "gitlab job id: $CI_JOB_ID"
    - git push artifact master

my_deploy:
  stage: deploy
# .gitlab-ci.yml

image: registry.gitlab.com/<USER>/my-project:latest

stages:
  - test
  - commit
  - deploy

my_test:
  stage: test
  script:
    - echo "run tests"

my_commit:
  stage: commit
  script:
    # Some SSH key config magic and Git configurations
    # Commit changes to artifact repo
    - git push artifact master

my_deploy:
  stage: deploy
  script:
    - envoy run deploy  # SSH task runner runs git pull and config import on production
  when: manual
  only:
    - master

Option 2: Commit to artifact repo

Option 3:

Pull and build at destination

All on Production!

  1. Git pull
  2. Composer Install
  3. NPM Install
  4. Compile assets (Gulp)

Releases are pulled, built, and served.

Option 3: Pull and build at destination

# .gitlab-ci.yml

# .gitlab-ci.yml

image: registry.gitlab.com/<USER>/my-project:latest

# .gitlab-ci.yml

image: registry.gitlab.com/<USER>/my-project:latest

services:
  - postgres:10

variables:
  POSTGRES_DB: drupal8
  POSTGRES_PASSWORD: drupal8


# .gitlab-ci.yml

image: registry.gitlab.com/<USER>/my-project:latest

services:
  - postgres:10

variables:
  POSTGRES_DB: drupal8
  POSTGRES_PASSWORD: drupal8

cache:
  paths:
    - drush/contrib/
    - vendor/
    # we purposely leave out web/modules/contrib, etc.
# .gitlab-ci.yml

image: registry.gitlab.com/<USER>/my-project:latest

# Postgres service and cache configs
# . . .

stages:
  - test
  - commit
  - deploy

my_test:
  stage: test

my_deploy:
  stage: deploy
# .gitlab-ci.yml

image: registry.gitlab.com/<USER>/my-project:latest

stages:
  - test
  - commit
  - deploy

my_test:
  stage: test
  script:
    - echo "run tests"
  artifacts:
    expire_in: 1 week
    paths:
      - web/
      - config/

my_deploy:
  stage: deploy

# .gitlab-ci.yml

image: registry.gitlab.com/<USER>/my-project:latest

stages:
  - test
  - commit
  - deploy

my_test:
  stage: test
  script:
    - echo "run tests"

deploy_dev:
  stage: deploy
  script:
    - envoy run deploy --env=dev
  only:
    - develop

deploy_prod:
  stage: deploy
  script:
    - envoy run deploy --env=prod
  when: manual
  only:
    - master

Option 3: Pull and build at destination

# .gitlab-ci.yml

image: registry.gitlab.com/<USER>/my-project:latest

stages:
  - test
  - commit
  - deploy

my_test:
  stage: test
  script:
    - echo "run tests"

deploy_dev:
  stage: deploy
  script:
    - envoy run deploy --env=dev
  only:
    - develop

deploy_prod:
  stage: deploy
  script:
    - envoy run deploy --env=prod
  when: manual
  only:
    - master
@servers([ 'production' => 'my_user@192.168.0.11')

{{-- Assumes gitlab-runner ssh keys are on production, and composer and yarn are installed globally. --}}
@task('deploy_production', ['on' => 'production'])

@endtask

Option 3: Pull and build at destination

@servers([ 'production' => 'my_user@192.168.0.11')
{{-- Assumes gitlab-runner ssh keys are on production, and composer and yarn are installed globally. --}}
@task('deploy_production', ['on' => 'production'])

  cd /var/www/html
  echo 'Setting to maintenance mode.'
  vendor/bin/drush state:set system.maintenance_mode 1 -y

@endtask

@servers([ 'production' => 'my_user@192.168.0.11')
{{-- Assumes gitlab-runner ssh keys are on production, and composer and yarn are installed globally. --}}
@task('deploy_production', ['on' => 'production'])
  cd /var/www/html
  echo 'Setting to maintenance mode.'
  vendor/bin/drush state:set system.maintenance_mode 1 -y

  echo 'Fetching commit changes.'
  git fetch origin
  git reset --hard {{ $commit }}

@endtask

@servers([ 'production' => 'my_user@192.168.0.11')
{{-- Assumes gitlab-runner ssh keys are on production, and composer and yarn are installed globally. --}}
@task('deploy_production', ['on' => 'production'])
  cd /var/www/html
  echo 'Setting to maintenance mode.'
  vendor/bin/drush state:set system.maintenance_mode 1 -y

  echo 'Fetching commit changes.'
  git fetch origin
  git reset --hard {{ $commit }}

  echo 'Running composer install.'
  composer install --no-dev --optimize-autoloader

@endtask

@servers([ 'production' => 'my_user@192.168.0.11')
{{-- Assumes gitlab-runner ssh keys are on production, and composer and yarn are installed globally. --}}
@task('deploy_production', ['on' => 'production'])
  cd /var/www/html
  echo 'Setting to maintenance mode.'
  vendor/bin/drush state:set system.maintenance_mode 1 -y

  echo 'Fetching commit changes.'
  git fetch origin
  git reset --hard {{ $commit }}

  echo 'Running composer install.'
  composer install --no-dev --optimize-autoloader

  echo 'Running Drupal updates.'
  vendor/bin/drush cache:rebuild
  vendor/bin/drush updatedb -y
  vendor/bin/drush config-split:import -y
  vendor/bin/drush core:cron -y

@endtask

@servers([ 'production' => 'my_user@192.168.0.11')
{{-- Assumes gitlab-runner ssh keys are on production, and composer and yarn are installed globally. --}}
@task('deploy_production', ['on' => 'production'])
  cd /var/www/html
  echo 'Setting to maintenance mode.'
  vendor/bin/drush state:set system.maintenance_mode 1 -y

  echo 'Fetching commit changes.'
  git fetch origin
  git reset --hard {{ $commit }}

  echo 'Running composer install.'
  composer install --no-dev --optimize-autoloader

  echo 'Running Drupal updates.'
  vendor/bin/drush cache:rebuild
  vendor/bin/drush updatedb -y
  vendor/bin/drush config-split:import -y
  vendor/bin/drush core:cron -y

  echo "running yarn install and gulp"
  cd /var/www/html/web/themes/custom/my_awesome_theme
  npm install
  ./node_modules/gulp/bin/gulp.js
  cd /var/www/html

@endtask

@servers([ 'production' => 'my_user@192.168.0.11')
{{-- Assumes gitlab-runner ssh keys are on production, and composer and yarn are installed globally. --}}
@task('deploy_production', ['on' => 'production'])
  cd /var/www/html
  echo 'Setting to maintenance mode.'
  vendor/bin/drush state:set system.maintenance_mode 1 -y

  echo 'Fetching commit changes.'
  git fetch origin
  git reset --hard {{ $commit }}

  echo 'Running composer install.'
  composer install --no-dev --optimize-autoloader

  echo 'Running Drupal updates.'
  vendor/bin/drush cache:rebuild
  vendor/bin/drush updatedb -y
  vendor/bin/drush config-split:import -y
  vendor/bin/drush core:cron -y

  echo "running yarn install and gulp"
  cd /var/www/html/web/themes/custom/my_awesome_theme
  npm install
  ./node_modules/gulp/bin/gulp.js
  cd /var/www/html

  echo 'Turning off maintenance mode.'
  vendor/bin/drush state:set system.maintenance_mode 0 -y
  vendor/bin/drush cache:rebuild
@endtask

Releases are cloned to tagged release directory, built, and served via symlink

12 Factor App

Option 4: Pull and build at symlink (12 Factor)

/var/www/my-app/
  ├── files/
  ├── settings.php
  ├── releases/
  |  ├── 20181225013000/
  |  └── 20181225023030/
  |     ├── .git/
  |     ├── composer.json/
  |     ├── composer.lock/
  |     ├── vendor/
  |     └── web/
  |        ├── core/
  |        ├── sites/
  |        |    └── default/
  |        |       ├── files -> /var/www/my-app/files/
  |        |       └── settings.php -> /var/www/my-app/settings.php
  |        └── index.php
  └── current -> /var/www/my-app/releases/20181225023030/


# Nginx Directory served
root /var/www/my-app/current/web

/var/www/my-app/
  ├── files/
  ├── settings.php
  ├── releases/
  |  ├── 20181225013000/
  |     ├── .git/
  |     ├── composer.json/
  |     ├── composer.lock/
  |     ├── vendor/
  |     └── web/
  |        ├── core/
  |        ├── sites/
  |        └── index.php
  └── current -> /var/www/my-app/releases/20181225013000/


# Nginx Directory served
root /var/www/my-app/current/web

/var/www/my-app/
  ├── releases/
  |  ├── 20181225013000/
  |     ├── .git/
  |     ├── composer.json/
  |     ├── composer.lock/
  |     ├── vendor/
  |     └── web/
  |        ├── core/
  |        ├── sites/
  |        └── index.php
  └── current -> /var/www/my-app/releases/20181225013000/




# Nginx Directory served
root /var/www/my-app/current/web

/var/www/my-app/
  ├── releases/
  |  ├── 20181225013000/
  |     ├── .git/
  |     ├── composer.json/
  |     ├── composer.lock/
  |     ├── vendor/
  |     └── web/
  |        ├── core/
  |        ├── sites/
  |        └── index.php
  └── current -> /var/www/my-app/releases/20181225013000/


/var/www/my-app/
  ├── releases/
  |  ├── 20181225013000/
  |     ├── .git/
  |     ├── composer.json/
  |     ├── composer.lock/
  |     ├── vendor/
  |     └── web/
  |        ├── core/
  |        ├── sites/
  |        └── index.php

/var/www/my-app/
  ├── .git/
  ├── composer.json/
  ├── composer.lock/
  ├── vendor/
  └── web/
      ├── core/
      ├── sites/
      └── index.php


/var/www/my-app/
  ├── files/
  ├── settings.php
  ├── releases/
  |  ├── 20181225013000/
  |     ├── .git/
  |     ├── composer.json/
  |     ├── composer.lock/
  |     ├── vendor/
  |     └── web/
  |        ├── core/
  |        ├── sites/
  |        |    └── default/
  |        |       ├── files -> /var/www/my-app/files/
  |        |       └── settings.php -> /var/www/my-app/settings.php
  |        └── index.php
  └── current -> /var/www/my-app/releases/20181225013000/


# Nginx Directory served
root /var/www/my-app/current/web

Option 4: Pull and build at symlink (12 Factor)

@setup
    $production_host = 'deployer@104.128.175.222';
    $repository = 'git@gitlab.com:jcandan/laravel-gitlab-ci-sample.git';
    $app_dir = '/var/www/laravel-gitlab-ci-sample';
    $release_dir = $app_dir . '/releases';
    $release = date('YmdHis');
    $new_release_dir = $release_dir . '/' . $release;
@endsetup

@servers(['production' => $production_host])

@story('deploy')
    clone_repository
    run_composer
    update_symlinks
@endstory

@task('clone_repository', ['on' => 'production'])
    [ -d {{ $release_dir }} ] || mkdir {{ $release_dir }}
    git clone --depth 1 {{ $repository }} {{ $new_release_dir }}
@endtask

@task('run_composer', ['on' => 'production'])
    cd {{ $new_release_dir }}
    composer install --prefer-dist --no-scripts --quiet --optimize-autoloader
@endtask

@task('update_symlinks', ['on' => 'production'])
    rm -rf {{ $new_release_dir }}/sites/default/files
    ln -nfs {{ $app_dir }}/files {{ $new_release_dir }}/sites/default/files

    ln -nfs {{ $app_dir }}/settings.php {{ $new_release_dir }}/sites/default/settings.php

    ln -nfs {{ $new_release_dir }} {{ $app_dir }}/current
@endtask
@setup
    $production_host = 'deployer@104.128.175.222';
    $repository = 'git@gitlab.com:jcandan/laravel-gitlab-ci-sample.git';
    $app_dir = '/var/www/laravel-gitlab-ci-sample';
    $release_dir = $app_dir . '/releases';
    $release = date('YmdHis');
    $new_release_dir = $release_dir . '/' . $release;
@endsetup

@servers(['production' => $production_host])

@story('deploy')
    clone_repository
    run_composer
    update_symlinks
@endstory

@task('clone_repository', ['on' => 'production'])
    [ -d {{ $release_dir }} ] || mkdir {{ $release_dir }}
    git clone --depth 1 {{ $repository }} {{ $new_release_dir }}
@endtask

@task('run_composer', ['on' => 'production'])
    cd {{ $new_release_dir }}
    composer install --prefer-dist --no-scripts --quiet --optimize-autoloader
@endtask

@setup
    $production_host = 'deployer@104.128.175.222';
    $repository = 'git@gitlab.com:jcandan/laravel-gitlab-ci-sample.git';
    $app_dir = '/var/www/laravel-gitlab-ci-sample';
    $release_dir = $app_dir . '/releases';
    $release = date('YmdHis');
    $new_release_dir = $release_dir . '/' . $release;
@endsetup

@servers(['production' => $production_host])

@story('deploy')
    clone_repository
    run_composer
    update_symlinks
@endstory

@task('clone_repository', ['on' => 'production'])
    [ -d {{ $release_dir }} ] || mkdir {{ $release_dir }}
    git clone --depth 1 {{ $repository }} {{ $new_release_dir }}
@endtask


@setup
    $production_host = 'deployer@104.128.175.222';
    $repository = 'git@gitlab.com:jcandan/laravel-gitlab-ci-sample.git';
    $app_dir = '/var/www/laravel-gitlab-ci-sample';
    $release_dir = $app_dir . '/releases';
    $release = date('YmdHis');
    $new_release_dir = $release_dir . '/' . $release;
@endsetup

@servers(['production' => $production_host])

@story('deploy')
    clone_repository
    run_composer
    update_symlinks
@endstory


@setup
    $production_host = 'deployer@104.128.175.222';
    $repository = 'git@gitlab.com:jcandan/laravel-gitlab-ci-sample.git';
    $app_dir = '/var/www/laravel-gitlab-ci-sample';
    $release_dir = $app_dir . '/releases';
    $release = date('YmdHis');
    $new_release_dir = $release_dir . '/' . $release;
@endsetup

@servers(['production' => $production_host])


@setup
    $production_host = 'deployer@104.128.175.222';
    $repository = 'git@gitlab.com:jcandan/laravel-gitlab-ci-sample.git';
    $app_dir = '/var/www/laravel-gitlab-ci-sample';
    $release_dir = $app_dir . '/releases';
    $release = date('YmdHis');
    $new_release_dir = $release_dir . '/' . $release;
@endsetup

Provisioning Drupal

for Behat

Drupal CI Test Issues

# .gitlab-ci.yml

image: registry.gitlab.com/<USER>/my-project:latest

stages:
  - test

test:
  stage: test
  script:
    - phpunit  # note: drupal does not need to be provisioned
    - composer install
    # Provision Drupal via site-install or database backup
    - drush site:install  # or pg_restore . . .
    - drush config:import -y
    - php -S localhost:8080 -t ./web/ &
    - behat

Provisioning Options

  1. Restore a production database backup
  2. Run site install and config import

Option 1:

Restore a database backup

  • setup a cron job to pull a database backup to the runner server
  • setup a volume on the job's docker container

We need to get the backup to the runner job container

  • volumes are configurable at gitlab-runner
  • volumes are not accessible from jobs

Nope

gitlab-ci persist mysql data between stages

  1. Push a container with all data pre-loaded
  2. Connect to remote database from job container

Possible work-arounds

Option 2:

Site install and config import

New in 8.6, install with existing config

$ drupal site:install --existing-config

In order to import config, the site UUID database must match configuration

Gotchas

  • Standard install profile
  • Override with Minimal
  • Force UUID to match
hook_install()

Fails if install profile implements

$ drupal site:install --existing-config
$ drush site:install minimal
$ drush config:import

Originally Standard, try to override install profile

Fails, can't override existing install profile.

Force UUID from config

# replace UUID
$ NEW_UUID=cat config/sync/system.site.yml | grep uuid | awk '{print $2}'
$ drush config:set -y system.site uuid $NEW_UUID

Shortcut Links already set

# Delete them
$ drupal entity:delete shortcut_set default

Could not delete

field body does not exist on entity type node. 

  • Do not re-add field_body field via UI
  • Must restore original configs for body field

Removed Body field, to implement Paragraphs

my_test:
  stage: test
  script:
    - composer install
    - export PGPASSWORD=drupal8
    - vendor/bin/drush site:install standard --db-url=pgsql://postgres:drupal8@postgres:5432/drupal8 -y
    - vendor/bin/drush config:set -y system.site uuid $(cat config/sync/system.site.yml | grep uuid | awk '{print $2}')
    - vendor/bin/drupal entity:delete shortcut_set default
    - echo "\$config['system.mail']['interface']['default'] = 'devel_mail_log';" >> web/sites/default/settings.php
    - echo "\$config['system.file']['path']['temporary'] = '/tmp';" >> web/sites/default/settings.php
    - echo "\$config['config_split.config_split.development']['status'] = TRUE;" >> web/sites/default/settings.php
    - vendor/bin/drush en config_split -y
    - vendor/bin/drush cache:rebuild
    - vendor/bin/drush updatedb -y
    - vendor/bin/drush config-split:import -y
    - vendor/bin/drush config-split:import development -y
    - vendor/bin/drush core:cron -y
    - vendor/bin/drush cache:rebuild
    - cd $CI_PROJECT_DIR/web/themes/custom/myawesometheme
    - yarn install
    - node_modules/gulp/bin/gulp.js
    - cd $CI_PROJECT_DIR
    - php -S localhost:8080 -t ./web/ &
    - sleep 2
    - 'sed -e "s/drupal_root:.*/drupal_root: \/builds\/<GITLAB USER>\/<REPO SLUG>\/web/;s/base_url:.*/base_url: http:\/\/localhost:8080/" behat.example.yml > behat.yml'
    - vendor/bin/behat --colors
  artifacts:
    expire_in: 1 week
    paths:
      - web/
      - config/

Tips

  • Try it locally first
  • Iterate locally if you can
docker run -it -v "$(PWD)":/var/www/html . . .
docker exec -it . . .
sed -e "s/drupal_root:.*/drupal_root: '\/app\/web'/;s/base_url:.*/base_url: http:\/\/appserver" > behat.yml
$ cat build.sh
lando rebuild -y
lando drush site:install standard --db-url=pgsql://drupal8:drupal8@database:5432/drupal8 -y
lando drush config:set -y system.site uuid $(cat config/sync/system.site.yml | grep uuid | awk '{print $2}')
lando drupal entity:delete shortcut_set default
# committed settings.php has lando env configs
lando drush en config_split -y
lando drush cache:rebuild
lando drush updatedb -y
lando drush config-split:import -y
lando drush config-split:import development -y
lando drush core:cron -y
lando drush cache:rebuild
# lando takes care of theme gulp
sed -e "s/drupal_root:.*/drupal_root: '\/app\/web'/;s/base_url:.*/base_url: http:\/\/appserver/" behat.example.yml > behat.yml
lando behat --colors

Q & A

Drupal + Gitlab CI

By James Candan

Made with Slides.com

Drupal + Gitlab CI

  • 1,140

More from James Candan