what this is *not* about

• Re-architecturing LCP/BGT
  • not about making LCP "reactive"
  • or Kafka, Redis, CouchDB2, etc
• Changing the deployment mechanism
  • not about the new CI/CD pipeline
  • or migrating to k8s

what this is about

• Codifying the lessons we learned deploying containers in production
  • "Better practices"
  • Re-eval Fabrika as our development and deployment protocol
• Paving the way for next-generation infrastructure
  • Making our containers conducive to being deployed to the cloud

"12-factor" containers

One imaGE to rule them all

1

Same image for app and deployment

(no more deployment images)

in fact, same image for managerial tasks as well

• analysis
• unit test
• integration test
• build image
• publish image
• run
• deploy

no more rebuilding image for dev/rc/prod

• Subject to vuln scan, integrity check, etc
• "Promoted" to a release candidate

configuration via environment vars

2

do not mount /config from host

into the container

• not cloud-friendly
  • hosts can come and go
  • should embrace ephemerality
  • make as few assumptions about the host as possible
• may get out of sync
• Rethink what is *actually* a config

How much of this file is *actually* configuration?

Use stdout as

log stream

3

corollary to the previous principle

Write everything to stdout

• syslog
• splunk

rethink base images

base image defines the capabilities 

• Is the image for a Python microservice?
• Does the image need nscd?

base image provides extension points 

• Impose contract so services can be built in a consistent way
• But, services can also extend its capability through its own Dockerfile
• ONBUILD directive
  • Defers the execution of the command until the build time of the child image

FROM python:2.7.11
MAINTAINER infrastructure-team@points.com
ENV GOSU_VERSION 1.9
ENV APP_GROUP=webgroup APP_USER=webuser
RUN DEBIAN_FRONTEND=noninteractive 
    # Install nscd
    ...
    # Install dumb-init
    ...
    # Install gosu
    ...
    # Add webapp user and group
    ...
WORKDIR /content
COPY ./timezone /etc/timezone
COPY ./entrypoint.sh /content/entrypoint.sh
ONBUILD COPY . /content
ONBUILD RUN chown -R $APP_USER:$APP_GROUP /content
ONBUILD RUN for r in $(ls requirements); do pip install -r requirements/$r; done
ONBUILD ARG service_name
ONBUILD ARG git_commit_hash
ONBUILD LABEL com.points.service_name=$service_name com.points.git_commit_hash=$git_commit_hash
ENTRYPOINT ["dumb-init", "/content/entrypoint.sh"]
CMD ["-l"]

FROM base-docker.points.com/python_service_onbuild

Dockerfile of Buy Service

Rethink

how services are run

we don't need apache mod_wsgi

• Causes dev and prod disparity
  • flask dev server during dev and apache in the container
• Only has sync worker
• We run a reverse proxy anyway (haproxy)
• Tightly coupled to Apache httpd

we want to have an app server

• Can run and debug locally the same way using the same server software as container
• Support sync and async worker types for different work loads
• Gunicorn, uwsgi

we don't need supervisord

• Necessary when we used to run apache and rsyslogd
  • Violates the container SRP
• Another layer of abstraction
  • supervisord has its own configuration and set of signals to respond to

we want to have a

container init system

• Lightweight solution to the "pid 1" problem
  • Normal processes do not reap zombies, only init does
• tini, dumb-init

an entrypoint

to tie these together

Entrypoint

library

Provides a consistent interface and reusable components for entrypoints of dockerized mciroservices at Points

entrypoint interface

• static_analysis
• test_unit
• test_integration
• build
• publish
• deploy
• run
• test_sanity

wait, is this another fabrika?

design tenets

Build Library,

Not framework

• Resist the urge to write classes, use inheritance or use template method pattern
• Favour composition over extension
• Limit the amount of options a dev is exposed to, while providing sane defaults
• Favour using CLI over API to reduce cognitive load

• It should be obvious what the tool is doing
• Every stage can be called independently as a task
  • As opposed to hiding functionality behind Fabrika API
• Do not hide necessary details
  • Fabrika generates Dockerfile as part of the build process
  • Fabrika builds app image during integration test

Transparency

• Entrypoint tasks should be invokable both inside and outside of a docker container
  • e.g., ``invoke test_unit`` should produce the same result as ``docker run IMAGE test_unit``

isomorphicity

• Builtin retry of idempotent operations
  • e.g., docker pull, docker run, setup load balancer

resiliency

implementation

python

invoke

• Written by the same author as Fabric
• Task invocation library for (in dev) Fabric 2.0
• Simple interface, doesn't try to do too much
  • e.g., no SSH remote execution
• Easy to write CLI with GNU calling convention

  • fab run_server:port=9090,check=1

  • inv run_server --port=9090 --check

why not fabric?

• Too many external dependencies
  • e.g., paramiko, pycrypto, cryptography for supporting SSH remote execution
• Remote execution is better done with direct SSH commands
  • More Straightforward
  • Less dependencies to install (paramiko, pycrypto and friends)
  • Can be reproduced by Copy/Paste the command and run it directly

why not bash/makefile?

• Adequate for small-scale scripts
• Becomes unmaintainable as the script grows
• Hard to share code between projects
• Harder to write unit tests
• Requires making a lot of mistakes, sweat and tears to become a bash expert
• Much so for Makefile

nothing is set in stone

• If there's compelling reason to use something else, we will consider it
• The advantage of invoke being a thin wrapper around pure python functions is that it's easily replaceable