DELIVERING QUALITY AUTOMATIONS

with Ansible - molecule - drone

@DambrineF

Florian Dambrine - Senior DevOps Engineer - @ GumGum

> Whoami

Florian Dambrine
DevOps Engineer @ GumGum
Joined GumGum 5 years ago
Ansible fan!

> Agenda

Test Driven Infrastructure Concepts
GumGum - Ansible Shop since 2014
- Ansible workflows past & present
- Ansible repository structure past & present
- Testing strategy past & present
Molecule
- Introduction
- Concepts
- Demo

> gumgum - Computer vision company

Invented In-Image

advertising in 2008

gumgum Advertising

> gumgum - Computer vision company

gumgum SPORTS

> gumgum - Computer vision company

gumgum DENTAL

> TEST-DRIVEN INFRASTRUCTURE

Make sure the automation worked as expected
Detect breaking changes earlier
Easily handle configuration management upgrades
Test automation with multiple software versions
Test complex infrastructure configurations:
- Clusters depending on other ones (Zookeeper dependency)
- Opscenter / Cassandra / Maintenance operations / Rolling restarts
Enforce Automation quality and best-practices

> GumGum - Ansible shop since 2014

 ~/Workspace/ops/ansible ● tree -L 2 -I '*.yaml'
.
├── ansible.cfg
├── <playbook>.yml
├── inventories/
│   ├── ireland
│   │   ├── ec2.ini
│   │   ├── ec2.py
│   │   └── group_vars
│   ├── japan
│   │   ├── ec2.ini
│   │   ├── ec2.py
│   │   └── group_vars
│   ├── oregon
│   │   ├── ec2.ini
│   │   ├── ec2.py
│   │   └── group_vars
│   └── virginia
│   │   ├── ec2.ini
│   │   ├── ec2.py
│   │   └── group_vars
├── library/
├── requirements.yml
└── roles/
    └── ...

+200 roles

> Ansible Workflows - Past & Present

Provision
Configure
Test
Cleanup
Snapshot
Deployment

Cluster Automation

Image creation

Previous Workflow

Current Workflow

Provision
Configure
Test
Cleanup
Snapshot
Deployment

DEPLOYMENT

> Ansible Repository structure - Past ...

...
### About 200 ansible roles
├── roles
│   ├── PHPCi-001
│   ├── activemq-001
│   ├── ad-server-001
│   ├── advertising-api-001
│   ├── ansible-001
│   ├── ansible-metadata-001
│   ├── apt-cacher-ng-001
│   ├── aws-api-server-002
│   ├── aws-cli-001
│   ├── aws-mon-linux-001
│   ├── bid-predictor-001
│   ├── bid-predictor-002
│   ├── bower-001
│   ├── caffe-on-spark-001
│   ├── cassandra-001
│   ├── cassandra-002
│   ├── cassandra-cleanup-001
│   ├── cassandra-clearsnapshot-001
│   ├── cassandra-compaction-001
│   ├── cassandra-firewall-001
│   ├── cassandra-repair-001
│   ├── cassandra-rolling-restart-001
│   ├── cloudfront-origin-001
│   ├── codedeploy-001
│   ├── collectd-001
│   ├── common-001
...

cons

Even though an effort was made a few years ago to CI/CD this monolith, no one was actually paying attention to it:
- Too complex
- Too heavy
No straight forward way to test locally (even though there was one)
More and more people writing Ansible roles
Still not good way to version roles

> Ansible Repository structure - ... And present

├── ansible-role-aws-cli

Pros

1 Ansible role = 1 Repository
- Easy to CI / CD
- Easy to contribute to
- Easy to version
- Easy to upgrade
Gain modularity and reusability
Open source openning

├── ansible-role-common

├── ansible-role-elasticsearch

├── ansible-role-gradle

├── ansible-role-groovy

├── ansible-role-java

...

ansible-ops

ansible-data-engineering

ansible-web-engineering

ansible-data-science

> Ansible Testing Strategy - Past ...

Ansible-Lint
Container provision
Ansible configure.yaml
Ansible test.yaml
ServerSpec tests

def getPlaybookList() {
    playbooks = [
        "queue-processor-001.yaml",
        "cron-001.yaml",
        "geo-server-003.yaml",
        "redis-001.yaml",
        "reporting-server-001.yaml",
        "taskrunner-001.yaml",
        "tomcat-001.yaml"
        "s3s3mirror-001.yaml",
        "spark-001.yaml"
    ]

    playbooks ? playbooks : null
}

for each $playbook

> Ansible Testing Strategy - ... And present

Ansible-Lint
Container provision
Ansible playbook.yml
Ansible side-effect.yml
Testinfra tests

for the $role

Introduction / Concepts / Demo

> Molecule - Introduction

Molecule provides a simple framework for easily and repeatedly testing your roles and playbooks against different environments and operating systems

Project originally created by retr0h
Recently moved to Ansible RedHat (October 2018)
Now integrated with Ansible Galaxy

$ pip install molecule

docker run --rm -it \
    -v '$(pwd)':/tmp/$(basename "${PWD}"):ro \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -w /tmp/$(basename "${PWD}") \
    quay.io/ansible/molecule:latest \
    sudo molecule test

Getting started

> Molecule - Concepts

Folder structure

<rolename>
├── README.md
├── defaults
│   └── main.yml
├── handlers
│   └── main.yml
├── meta
│   └── main.yml
├── molecule
│   └── default
│       ├── Dockerfile.j2
│       ├── INSTALL.rst
│       ├── create.yml
│       ├── destroy.yml
│       ├── molecule.yml
│       ├── playbook.yml
│       ├── prepare.yml
│       └── tests
│           └── test_default.py
├── tasks
│   └── main.yml
└── vars
    └── main.yml

Molecule internals

Developer files

Linters

Dependencies

Drivers

Verifiers

goss

> Molecule - Concepts

molecule.yml

lint:
  name: yamllint

dependency:
  name: galaxy
  options:
    role-file: ${PWD}/requirements.yml

driver:
  name: docker

platforms:
  - name: ubuntu_trusty
    image: ubuntu:trusty
    ...

  - name: ubuntu_xenial
    image: solita/ubuntu-systemd:xenial
    ...

provisioner:
  name: ansible
  config_options:
    defaults:
      callback_whitelist: timer
  inventory:
    links:
      group_vars: ../inventory/group_vars/
  lint:
    name: ansible-lint
    enabled: true

scenario:
  name: default
  converge_sequence:
    - dependency
    - create
    - prepare
    - converge
    - idempotence

verifier:
  name: testinfra
  lint:
    name: flake8

https://molecule.readthedocs.io

> Molecule - Concepts

A scenario is a self-contained directory containing everything necessary for testing the role in a particular way
The default scenario is named default, and every role should contain a default scenario
A Scenario defines a sequence of actions to execute for a command

Scenarios

scenario:
  name: default
  create_sequence:
    - create
    - prepare
  check_sequence:
    - destroy
    - dependency
    - create
    - prepare
    - converge
    - check
    - destroy
  converge_sequence:
    - dependency
    - create
    - prepare
    - converge
  destroy_sequence:
    - destroy
  test_sequence:
    - lint
    - destroy
    - dependency
    - syntax
    - create
    - prepare
    - converge
    - idempotence
    - side_effect
    - verify
    - destroy

command

sequence

> Molecule - Concepts

$ molecule init template --url \
    https://github.com/Lowess/ansible-role-cookiecutter

--> Initializing new role role_name...
role_name [role_name]: meetup-demo
role_description [TODO]: Cookiecutter is awesome !
role_author [Your Name]: Florian Dambrine
min_ansible_version [2.0]: 2.2
Initialized role in /tmp/tmp/role_name successfully.

$ tree ansible-role-meetup-demo -L 2
ansible-role-meetup-demo
├── .ansible-lint
├── .drone.yml
├── .gitignore
├── .pre-commit-config.yml
├── .yamllint
├── README.md
├── defaults
│   └── main.yml
├── handlers
│   └── main.yml
├── meta
│   └── main.yml
├── molecule
│   ├── default
│   ├── inventory
│   └── resources
├── tasks
│   ├── cleanup.yml
│   ├── configure.yml
│   ├── main.yml
│   └── test.yml
└── vars
    └── main.yml

Real time saver !
Enforce role convention / consistence
Defines linter rules, pipelines, pre-commit hooks

> Molecule - introducing Gumsible !

docker run --rm -it \
    -v '$(pwd)':/tmp/$(basename "${PWD}"):ro \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -w /tmp/$(basename "${PWD}") \
    quay.io/ansible/molecule:latest \
    sudo molecule test

gumsible test

Test containers managed by Molecule

Gumsible features

Bash and ZSH compatible
ZSH auto completion
Run useful sidecar containers
Finds the root folder of the role automatically
Easily configurable thanks to a .gumsible config file
Shortcut of molecule init with Cookiecutter

https://github.com/Lowess/gumsible-oh-my-zsh-plugin/

> Molecule - Custom Docker image

quay.io/ansible/molecule

lowess/drone-molecule

Vs

Missing git+ssh (PR #1604)

Latest Ansible version only

Drone CI plugin

Offers git+ssh (private repos)

Offers multiple Ansible versions

Extra ansible-lint rules

Mitogen plugin available

DEMO time !

Concepts / Demo

Drone.io

> Drone - concepts

# .drone.yml
---
kind: pipeline
name: ansible-ci
workspace:
  base: /drone
  path: src/${DRONE_REPO}

steps:
  - name: molecule
    image: lowess/drone-molecule:2.6.8
    environment:
      CI_UUID: _2.6_${DRONE_COMMIT_SHA:0:8}
      ANSIBLE_STRATEGY: mitogen_linear
    pull: true
    settings:
      task: test
    volumes:
      - name: dockersock
        path: /var/run/docker.sock

  - name: galaxy
    image: lowess/drone-molecule:2.6.8
    environment:
      GALAXY_TOKEN:
        from_secret: ansible_galaxy_token
    commands:
      - ansible-galaxy login --github-token $GALAXY_TOKEN
      - ansible-galaxy import \
          $DRONE_REPO_NAMESPACE $DRONE_REPO_NAME

volumes:
  - name: dockersock
    host:
      path: /var/run/docker.sock

/var/run/docker.sock

> Drone - Demo

> Thanks !

Let's start building a better galaxy, one molecule at a time...

🚀 We are hiring !

> Appendix - Mitogen for Ansible

A single network roundtrip is used to execute a step whose code already exists in RAM on the target (saves 4 ms runtime per 1 ms of network latency for every playbook step)
Processes are aggressively reused, avoiding the cost of invoking Python and recompiling imports, saving 300-800 ms for every playbook step.
Code is ephemerally cached in RAM, reducing bandwidth usage by an order of magnitude compared to SSH pipelining, with around 5x fewer frames traversing the network in a typical run.
The effect is most potent on playbooks that execute many short-lived actions, where Ansible’s overhead dominates the cost of the operation, for example when executing large with_items loops to run simple commands or write files.

Mitogen is a Python library for writing distributed self-replicating programs. [...] An extension to Ansible is included that implements connections over Mitogen, replacing embedded shell invocations with pure-Python equivalents invoked via highly efficient remote procedure calls to persistent interpreters tunnelled over SSH.

Expect a 1.25x - 7x speedup and a CPU usage reduction of at least 2x.