codes

Containment

data

efficacy

 

containerised component types

design Principles

• Continuous integration/deployment
  
• Deterministic pipeline
• Favour simplicity
• JSON everywhere
• Constant cycle-time optimisation
• Partial or whole stack deployable 
  on developers' machines
• Containers identical from dev
  through test to production

building containers - shared code

• Shared code repository
  • npm module
  • Present in every repository/image (even couchbase/redis)
  • Contains linting rules, extensible task runner, read-only credentials
  • Idempotent npm post-install task creates test directories and manipulates filesystem
  • Any change triggers cascading builds in dependent jobs

building containers - base image

FROM ubuntu:14.04
MAINTAINER Andrew Martin "sublimino@gmail.com"

ENV APT_PACKAGES curl build-essential wget git
ENV NPM_PACKAGES forever gulp http-server
ENV NODE_VERSION 0.11.16

# apt and npm installation
#
RUN apt-get update && apt-get install -y $APT_PACKAGES && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

RUN curl https://cdn.rawgit.com/isaacs/nave/master/nave.sh > /bin/nave && \   
    chmod a+x /bin/nave && \
    nave usemain $NODE_VERSION;

RUN npm install -g $NPM_PACKAGES;

# configure private npm
#
ADD ./node_modules/cb-shared/npmrc /root/.npmrc

BUILDING CONTAINERS -  Task runner

• Stream processing and parallel task execution
• Abstract invocation of language/library specificities
• Tight dev feedback loop, reduced cycle-time

BUILDING CONTAINERS -  TASK RUNNER

$ gulp <tab> 
browserify              fleet:restart          test:functional
build                   fleet:start            test:functional:remote
build-properties        help                   test:functional:watch
build:watch             less                   test:integration
default                 lint                   test:integration:remote
docker:build            livereload             test:integration:watch
...

• 4 different test runners
  
• Defers test execution to cross platform BASH scripts
  • BATS acceptance tested
    
  • https://github.com/sstephenson/bats
  • OSX vs Linux - some GNU-ish complexities
• Build server has no logic

building containers - a microservice

FROM harbour.example.com/cb-baseimage:latest
MAINTAINER Andrew Martin "sublimino@gmail.com"

# create installation directory
#
RUN mkdir -p /opt/d2
WORKDIR /opt/d2

# add package.json and npm install
#
ADD package.json /opt/d2/
RUN HOME=/root npm install

ADD ./bin /opt/d2/bin
ADD ./lib /opt/d2/lib
ADD ./test /opt/d2/test
ADD exemplary.crt /opt/d2/exemplary.crt

EXPOSE 4000

ENTRYPOINT ["bin/server.js"]

building containers - in summary

testing, testing, 123

testing a container

components - Datastores

COMPONENTS - Queues

COMPONENTS - Black box

COMPONENTS - Microservice

COMPONENTS - other

TESTING PROCESS - 

unit/integration

Testing process - functional/acceptance

• Functional/Component Integration: 
  "Limited component test using test doubles"
• Acceptance/E2E: "Verify system meets
  external requirements from end to end"
  
  
• Global setup/teardown and test override hooks
  
• External services brought up with fig
• Each repository's fig.yml configuration file:
  • Defines the component's invocation
  • ...and that of any immediate dependencies
• (database fixtures in a separate image)

TESTING PROCESS - fig.yml

• fig.yml  of c omponent under test is copied
  to  fig-test.yml
• Dependent components'  fig.yml  files 
  merged into  fig-test.yml
• Brings in transitive dependencies
• Environment variables are templated
  • NODE_ENV 
  • Database names
  • IPs and ports

fig.yml FOR "NHS" MICROSERVICE

redis:
  image: harbour.example.com/cb-nhs-redis:latest
  ports:
   - "6379:6379"
  ...

nhs:
  build: .
  # image: harbour.example.com/cb-nhs:latest
  environment:
   - CBNHS_LOGLEVEL=debug
   - CBNHS_DBBUCKET=cb-api
   - CBNHS_PRE_AUTHENTICATE=0
   ...
  ports:
   - "58000:8000"
  links:
   - redis
   - couchbase
   - api

FIG-test.YML FOR "NHS" MICROSERVICE

redis:
  image: harbour.example.com/cb-nhs-redis:latest
  ports:
   - "6379:6379"

nhs:
  build: .
  # image: harbour.example.com/cb-nhs:latest
  environment:
   ...
  ports:
   - "58000:8000"
  links:
   - redis
   - couchbase
   - api

couchbase:
  image: harbour.example.com/cb-couchbase-server:latest
  ...

api:
  image: harbour.example.com/cb-api:latest
  ...

in the test Setup script...

• fig --file=fig-test.yml \
  --project-name cb-nhs \ up &
  
• fig ps,  HTTP health checks and  /health   
  RESTful endpoints are used to determine availability
  
  
• Polling preferred to sleeping                 
• Any other necessary pre-test config
• API oAuth authentication
• Endpoints for exposing tunnels to Sauce labs 
• JS/CSS build chains

test demo

actual test demo

POST-TEST

• On build server test completion, image is tagged  
  
• :latest-dev                                   
  
• :20150106-42 (date-jenkinsBuild#)
• Tagged image pushed to private docker registry

system integration test

• Production-like environment 
• Test that the interfaces between
  components match or "hang together"
• Tests parts of the system not 
  covered by component tests
• infrastructure
• routing
• systemd units
• support containers
• On completion, non-functional tests 
  are run in an isolated environment

non-functional testing

• Same configuration  as system
  integration environment
  
• Tests
• Load/capacity
  
• Stress
• Resilience

container Deployment

• Current design informed by Kubernetes
• Automated by a Deployment Bot
• Journalling service manifest
• Service discovery via etcd
• HAProxy routing via confd
• Rolling updates
• Cluster/service health monitoring

a note on logging

Logstash Config - pt 1

input {
  lumberjack {
    port => 54321
    codec => "json"
    ssl_certificate => "server.crt"
    ssl_key => "server.key"
  }
}

filter {
  mutate {
    remove_field => [ "__CURSOR", "__MONOTONIC_TIMESTAMP" ]
  }
  if [type] == "stdin" {
    mutate {
      # microsecs to millisecs
      gsub => [ "__REALTIME_TIMESTAMP", "\d{3}$", "" ]
    }
    date {
      match => [ "__REALTIME_TIMESTAMP", "UNIX_MS" ]
    }
  }
  # (continued on next slide...)

LOGSTASH CONFIG - PT 2

  # (continued from last slide...)

  # if messages start and end with curly brace, interpret as json,
  # except messages from the logstash container, which do start and
  # end with curly brace and are NOT json!
  if [MESSAGE] =~ /^\{.*\}$/ and [_SYSTEMD_UNIT] !~ /^logstash/ {
    json {
      source => "MESSAGE"
      target => "json"
    }
  }
}
output {
  elasticsearch {
    host => "${ES_HOST}"
    port => ${ES_PORT}
    protocol => "http"
  }
}

BRING PAIN FORWARD: 

deliver continuously

Questions?