Tips for running an efficient CI and CD system

Will Munn

works at: focusrite/novation

github: @willm

What is continuous integration?

  • Checking every commit can integrate with the current system
  • Alerting development teams of failures as quickly as possible

What is continuous delivery?

  • Regularly deploying small changes to production
  • Alerting development teams of failures as quickly as possible

Tools for CI / CD

Configuring your tool

Tip No1

keep your agents lean

Team A

Team B

Team C

4.x.x

7.10.0

Why is this bad

  • Hard to scale
  • Uses more resources

Solutions

  • install npm packages locally
  • Script the agent configuration process
  • Use docker

Tip No2

Keep configuration in source control

#! /env/sh
npm install
npm test
git push heroku master

Your build script 

git clone git@github.com:willm/speed-up-ci.git

Checkout source

Build step to build and deploy your code

#!/usr/bin/env/ sh
npm install
npm test
git push heroku master

Your build script 

git clone git@github.com:willm/speed-up-ci.git

Checkout source

Build step to build and deploy your code

0 9 * * *

Cron to run build at 9:00am daily

Plugin that configures canned test data

Plugin that reports awesome test output

Your build script 

Plugin that emails stake holders when a release happens

Plugin that zips your built source only and stores it somewhere

Super awesome plugin that is critical to making a release functional.

You can't deploy because the CI server that holds the of the necessary steps is down

Why this is bad

Solutions

  • Keep all the CI logic in a script or task runner
  • Resist the urge of using every feature / plugin of your CI tool.
  • Commit your build definitions (jenkins pipeline, circle.yaml, etc...)

Tip No 3

Avoid npm bloat

#!/usr/bin/env bash
npm init
npm install --save express request mocha typescript grunt
$ du -h -d 1
50M    ./node_modules
50M    .

$ ls -al node_modules | wc -l
199

$ find . -type f | wc -l
4086

Solutions

  • learn node core!
  • Install modules as you need them
  • Avoid massive frameworks (angular2, ionic)
  • Start with lightweight modules and solutions
const http = require('http');

http.createServer((req, res) => {
    res.writeHead(200, {'Content-Type': 'application/json'});
    res.end(JSON.stringify({greeting: 'Hello World'}));
}).listen(3000);

needle 808Kb vs request 6.3Mb

tape 2.1Mb vs ava 34Mb

router 1.1Mb vs express 1.7Mb

npm scripts vs $BUILD_TOOL

Tip No4

keep your docker images lean

Large Images

  • Take up unnecessary disk space on your instances
  • Take longer to build and deploy
  • More software can = more things to go wrong

Suggested workflow

2 dockerfiles

Development

FROM node:7.10.0-alpine
RUN mkdir /app
ADD package.json /app/package.json
WORKDIR /app
RUN npm install
ADD . /app
RUN npm run build
$ docker images
REPOSITORY        TAG                 IMAGE ID            CREATED             SIZE
node              6.9.2               178934e73268        6 days ago          651.2 MB
node              6.9.2-alpine        de529845c111        6 days ago          50.69 MB

Production

FROM 7.10.0-alpine
ADD . /app
WORKDIR /app
CMD npm start

'.' should be a directory containing your built application without dev dependencies, tests and anything else that isn't needed by your application at runtime.

Build steps

  1. Build development docker image
  2. Run tests inside development container
  3. Remove all non runtime specific files
  4. Build and publish your production docker image with your built artefact.
  5. Run smoke tests on the running image in a test environment.
  6. Repeat in production.

Multi stage pipelines

  • A new native docker (17.05+) solution for intermediate build and test containers in a single Dockerfile

https://codefresh.io/blog/node_docker_multistage/

Tip No5

Use docker-compose

version: '2'
services:
    openshift:
        build:
            context: './src/test/containers/openshift'
            dockerfile: 'Dockerfile'
    service:
        build:
            context: './src/test/containers/service'
            dockerfile: 'Dockerfile'
    mutual-ssl-service:
        build:
            context: './src/test/containers/mutual-ssl-service'
            dockerfile: 'Dockerfile'
    api:
        build:
            context: '.'
            dockerfile: dev.Dockerfile
        image: 'yaapi'
        expose:
            - '3000'
        links:
            - service
            - mutual-ssl-service
            - openshift
        environment: '...'
        command: sh -c 'npm start | node_modules/.bin/bunyan'
    tests:
        container_name: tests
        image: yaapi
        command: 'npm test'
        working_dir: /app
        links:
            - api
            - service

Other tips

  • Get a build monitor
  • Integrate with pull requests
  • Try yarn
  • Run your tests in parallel

Faster CI with node and docker

By Will Munn

Made with Slides.com

Faster CI with node and docker

  • 339

More from Will Munn