> Why do we need gumref ?

2017 Team ratios

~ 7 Engineers

~ 20 Engineers

~ 3 Engineers

~ 12 Engineers

~ 4 Engineers

  > Why do we need gumref ?

2017 GumGum Infrastructure

  > Why do we need gumref ?

• Even with a DevOps team, developers are usually waiting for Ops to spin up environments or infrastructure components. Let's automate the last mile !
   
• Environment parity and standardization
  • Parity:
    • feature branch > dev > stage > prod
  • Standardization:
    • An application should follow a certain skeleton
       
• Development efficiency  (Image science builds -> ~1h )
   
• Better integration between applications
   
• Faster and safer deployments

  > What is gumref ?

• It is a Framework made of:
  • Ansible Container -> Docker
  • Jenkins pipeline -> CI / CD
  • AWS ECS -> Infrastructure running containers
     
• An application project implementing the Gumref framework must:
  • Contain the git submodule gumref (might change)
  • Copy the Gumref Jenkinsfile (CI / CD pipelines)
  • Copy and edit the Gumref Jenkinsfile.properties (pipeline strategy)
  • Create and implement an ansible-container project for the application

  > gumref - Dockerfile VS ansible-container

#!/bin/bash

  > GUMREF - DOCKERFILE VS ANSIBLE-CONTAINER

### Dockerfile
FROM nvidia/cuda:7.5-cudnn5-devel-ubuntu14.04
RUN apt-get update \
    && apt-get -y install \
        software-properties-common python-software-properties \
    && add-apt-repository -y 'ppa:ubuntu-toolchain-r/test' \
    && apt-get -y update \
    && apt-get -y install \
      gcc-4.9 \
      g++-4.9 \ 
      build-essential \
      cmake \
      git \
      wget \
      python-dev \
      python-pip \
      python-boto python-botocore \
      libssl-dev \     
      libcurl4-openssl-dev \
      libjsoncpp-dev \
      libprotobuf-dev \
      libleveldb-dev \
      protobuf-compiler \
      libgflags-dev \
      libgoogle-glog-dev \
      libsnappy-dev \
      liblmdb-dev \ 
      libhdf5-serial-dev \
      python$PYTHON_VERSION-dev \
      unzip \
      libatlas-base-dev \
      gfortran \
      libjasper-dev \
      libgtk2.0-dev \ 
      libavcodec-dev \
      libavformat-dev \
      libswscale-dev \ 
      libjpeg-dev

### container.yml
---
version: "2"

services:
  image-science-base:
    image: "nvidia/cuda:7.5-cudnn5-devel-ubuntu14.04"
    roles:
      - image-science-base

### roles/image-science-base/tasks/main.yml
---

- name: Install apt dependencies
  apt:
    state: present
    name: "{{ item }}"
    update_cache: yes
  with_items:
    - software-properties-common
    - python-software-properties

- name: Add toolchain PPA
  apt_repository:
     state: present
     repo: 'ppa:ubuntu-toolchain-r/test'

- name: Install apt packages
  apt:
    state: present
    name: "{{ item }}"
    update_cache: yes
  with_items:
    - gcc-4.9 
    - g++-4.9  
    - build-essential 
    - cmake 
    - git 
    - wget 
    - python-dev 
    - python-pip 
    - python-boto python-botocore 
    - libssl-dev      
    - libcurl4-openssl-dev 
    - libjsoncpp-dev 
    - libprotobuf-dev 
    - libleveldb-dev 
    - protobuf-compiler 
    - libgflags-dev 
    - libgoogle-glog-dev 
    - libsnappy-dev 
    - liblmdb-dev  
    - libhdf5-serial-dev 
    - python$PYTHON_VERSION-dev 
    - unzip 
    - libatlas-base-dev 
    - gfortran 
    - libjasper-dev 
    - libgtk2.0-dev  
    - libavcodec-dev 
    - libavformat-dev 
    - libswscale-dev  
    - libjpeg-dev

  > Gumref - Jenkins pipelines

#!groovy
pipeline {
    agent any

    stages {
        stage('PREPARE') {...}

        stage('BUILD') {...}

        stage('UNIT_TESTS') {...}

        stage('INTEGRATION_TESTS - Start up containers') {...}

        stage('INTEGRATION_TESTS - Run integration tests') {...}

        stage('INTEGRATION_TESTS - Shutdown containers') {...}

        stage('CREATE_RELEASE') {...}

        stage('DEPLOY_DEV') {...}

        stage('QA_DEV') {...}

        stage('DEPLOY_STAGING') {...}

        stage('QA_STAGING') {...}

        stage('OPS_SIGNING') {...}

        stage('DEPLOY_PROD') {...}
    }
}

• Pipeline definition is part of the application Git repository
   
• They define the way the application should be continuously integrated and deployed
   
• They can track and integrate multiple Git branches

  > Gumref - AWS ecs infrastructure

Image Science Team

Data Science Team

ds.ecs.va.ggops.com

image.ecs.va.ggops.com

web.ecs.va.ggops.com

Application Load Balancers (ALB)

ECS Clusters

/<application>/<env>

/vertex/stage

Route 53 DNS

target B

target A

target C

target D

target F

target E