DELIVERING QUALITY AUTOMATIONS

with Ansible - molecule - drone

 

 

 

 

@DambrineF

Florian Dambrine - Senior DevOps Engineer - @ GumGum

 > Whoami

  • Florian Dambrine

  • DevOps Engineer @ GumGum

  • Joined GumGum 5 years ago

  • Ansible fan!

 > Agenda

  • Test Driven Infrastructure Concepts
  • GumGum - Ansible Shop since 2014
    • Ansible workflows past & present
    • Ansible repository structure past & present
    • Testing strategy past & present
  • Molecule
    • Introduction
    • Concepts
    • Demo

 > gumgum - Computer vision company

Invented In-Image 

advertising in 2008

gumgum Advertising

 > gumgum - Computer vision company

gumgum SPORTS

 > gumgum - Computer vision company

gumgum DENTAL

 > TEST-DRIVEN INFRASTRUCTURE

  • Make sure the automation worked as expected
  • Detect breaking changes earlier
  • Easily handle configuration management upgrades
  • Test automation with multiple software versions
  • Test complex infrastructure configurations:
    • Clusters depending on other ones (Zookeeper dependency)
    • Opscenter / Cassandra / Maintenance operations / Rolling restarts
  • Enforce Automation quality and best-practices

 > GumGum - Ansible shop since 2014

 ~/Workspace/ops/ansible ● tree -L 2 -I '*.yaml'
.
├── ansible.cfg
├── <playbook>.yml
├── inventories/
│   ├── ireland
│   │   ├── ec2.ini
│   │   ├── ec2.py
│   │   └── group_vars
│   ├── japan
│   │   ├── ec2.ini
│   │   ├── ec2.py
│   │   └── group_vars
│   ├── oregon
│   │   ├── ec2.ini
│   │   ├── ec2.py
│   │   └── group_vars
│   └── virginia
│   │   ├── ec2.ini
│   │   ├── ec2.py
│   │   └── group_vars
├── library/
├── requirements.yml
└── roles/
    └── ...

+200 roles

 > Ansible Workflows - Past & Present

  • Provision
  • Configure
  • Test
  • Cleanup
  • Snapshot
  • Deployment

 Cluster Automation

 Image creation

Previous Workflow

Current Workflow

  • Provision
  • Configure
  • Test
  • Cleanup
  • Snapshot
  • Deployment

 DEPLOYMENT

 > Ansible Repository structure - Past ...

...
### About 200 ansible roles
├── roles
│   ├── PHPCi-001
│   ├── activemq-001
│   ├── ad-server-001
│   ├── advertising-api-001
│   ├── ansible-001
│   ├── ansible-metadata-001
│   ├── apt-cacher-ng-001
│   ├── aws-api-server-002
│   ├── aws-cli-001
│   ├── aws-mon-linux-001
│   ├── bid-predictor-001
│   ├── bid-predictor-002
│   ├── bower-001
│   ├── caffe-on-spark-001
│   ├── cassandra-001
│   ├── cassandra-002
│   ├── cassandra-cleanup-001
│   ├── cassandra-clearsnapshot-001
│   ├── cassandra-compaction-001
│   ├── cassandra-firewall-001
│   ├── cassandra-repair-001
│   ├── cassandra-rolling-restart-001
│   ├── cloudfront-origin-001
│   ├── codedeploy-001
│   ├── collectd-001
│   ├── common-001
...

cons

  • Even though an effort was made a few years ago to CI/CD this monolith, no one was actually paying attention to it:
    • Too complex
    • Too heavy
  • No straight forward way to test locally (even though there was one)
  • More and more people writing Ansible roles 
  • Still not good way to version roles

 > Ansible Repository structure - ... And present

├── ansible-role-aws-cli

Pros

  • 1 Ansible role = 1 Repository
    • Easy to CI / CD
    • Easy to contribute to
    • Easy to version
    • Easy to upgrade
       
  • Gain modularity and reusability 
     
  • Open source openning
├── ansible-role-common
├── ansible-role-elasticsearch
├── ansible-role-gradle
├── ansible-role-groovy
├── ansible-role-java

...

ansible-ops

ansible-data-engineering

ansible-web-engineering

ansible-data-science

 > Ansible Testing Strategy - Past ...

  1. Ansible-Lint
  2. Container provision
  3. Ansible configure.yaml
  4. Ansible test.yaml
  5. ServerSpec tests
def getPlaybookList() {
    playbooks = [
        "queue-processor-001.yaml",
        "cron-001.yaml",
        "geo-server-003.yaml",
        "redis-001.yaml",
        "reporting-server-001.yaml",
        "taskrunner-001.yaml",
        "tomcat-001.yaml"
        "s3s3mirror-001.yaml",
        "spark-001.yaml"
    ]

    playbooks ? playbooks : null
}

for each $playbook

 > Ansible Testing Strategy - ... And present

  1. Ansible-Lint
  2. Container provision
  3. Ansible playbook.yml
  4. Ansible side-effect.yml
  5. Testinfra tests
for the $role

Introduction / Concepts / Demo

 > Molecule - Introduction

Molecule provides a simple framework for easily and repeatedly testing your roles and playbooks against different environments and operating systems

  • Project originally created by retr0h
  • Recently moved to Ansible RedHat (October 2018)
  • Now integrated with Ansible Galaxy
$ pip install molecule
docker run --rm -it \
    -v '$(pwd)':/tmp/$(basename "${PWD}"):ro \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -w /tmp/$(basename "${PWD}") \
    quay.io/ansible/molecule:latest \
    sudo molecule test

Getting started

 > Molecule - Concepts

Folder structure

<rolename>
├── README.md
├── defaults
│   └── main.yml
├── handlers
│   └── main.yml
├── meta
│   └── main.yml
├── molecule
│   └── default
│       ├── Dockerfile.j2
│       ├── INSTALL.rst
│       ├── create.yml
│       ├── destroy.yml
│       ├── molecule.yml
│       ├── playbook.yml
│       ├── prepare.yml
│       └── tests
│           └── test_default.py
├── tasks
│   └── main.yml
└── vars
    └── main.yml

Molecule internals

Developer files

Linters
Dependencies
Drivers
Verifiers

goss

 > Molecule - Concepts

molecule.yml

lint:
  name: yamllint

dependency:
  name: galaxy
  options:
    role-file: ${PWD}/requirements.yml

driver:
  name: docker

platforms:
  - name: ubuntu_trusty
    image: ubuntu:trusty
    ...

  - name: ubuntu_xenial
    image: solita/ubuntu-systemd:xenial
    ...
provisioner:
  name: ansible
  config_options:
    defaults:
      callback_whitelist: timer
  inventory:
    links:
      group_vars: ../inventory/group_vars/
  lint:
    name: ansible-lint
    enabled: true

scenario:
  name: default
  converge_sequence:
    - dependency
    - create
    - prepare
    - converge
    - idempotence

verifier:
  name: testinfra
  lint:
    name: flake8

https://molecule.readthedocs.io

 > Molecule - Concepts

  • A scenario is a self-contained directory containing everything necessary for testing the role in a particular way
     
  • The default scenario is named default, and every role should contain a default scenario
     
  • A Scenario defines a sequence of actions to execute for a command

  Scenarios

scenario:
  name: default
  create_sequence:
    - create
    - prepare
  check_sequence:
    - destroy
    - dependency
    - create
    - prepare
    - converge
    - check
    - destroy
  converge_sequence:
    - dependency
    - create
    - prepare
    - converge
  destroy_sequence:
    - destroy
  test_sequence:
    - lint
    - destroy
    - dependency
    - syntax
    - create
    - prepare
    - converge
    - idempotence
    - side_effect
    - verify
    - destroy
command
sequence

 > Molecule - Concepts

$ molecule init template --url \
    https://github.com/Lowess/ansible-role-cookiecutter
--> Initializing new role role_name...
role_name [role_name]: meetup-demo
role_description [TODO]: Cookiecutter is awesome !
role_author [Your Name]: Florian Dambrine
min_ansible_version [2.0]: 2.2
Initialized role in /tmp/tmp/role_name successfully.
$ tree ansible-role-meetup-demo -L 2
ansible-role-meetup-demo
├── .ansible-lint
├── .drone.yml
├── .gitignore
├── .pre-commit-config.yml
├── .yamllint
├── README.md
├── defaults
│   └── main.yml
├── handlers
│   └── main.yml
├── meta
│   └── main.yml
├── molecule
│   ├── default
│   ├── inventory
│   └── resources
├── tasks
│   ├── cleanup.yml
│   ├── configure.yml
│   ├── main.yml
│   └── test.yml
└── vars
    └── main.yml
  • Real time saver !
  • Enforce role convention / consistence
  • Defines linter rules, pipelines, pre-commit hooks

 > Molecule - introducing Gumsible !

docker run --rm -it \
    -v '$(pwd)':/tmp/$(basename "${PWD}"):ro \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -w /tmp/$(basename "${PWD}") \
    quay.io/ansible/molecule:latest \
    sudo molecule test
gumsible test

Test containers managed by Molecule

Gumsible features

  • Bash and ZSH compatible
  • ZSH auto completion
  • Run useful sidecar containers
  • Finds the root folder of the role automatically
  • Easily configurable thanks to  a .gumsible config file
  • Shortcut of molecule init with Cookiecutter

 

 

https://github.com/Lowess/gumsible-oh-my-zsh-plugin/

 > Molecule - Custom Docker image

quay.io/ansible/molecule

lowess/drone-molecule

Vs

 Missing git+ssh (PR #1604)

 Latest Ansible version only

 Drone CI plugin

 Offers git+ssh (private repos)

 Offers multiple Ansible versions

 Extra ansible-lint rules

 Mitogen plugin available

DEMO time !

Concepts / Demo

Drone.io

 > Drone - concepts

# .drone.yml
---
kind: pipeline
name: ansible-ci
workspace:
  base: /drone
  path: src/${DRONE_REPO}

steps:
  - name: molecule
    image: lowess/drone-molecule:2.6.8
    environment:
      CI_UUID: _2.6_${DRONE_COMMIT_SHA:0:8}
      ANSIBLE_STRATEGY: mitogen_linear
    pull: true
    settings:
      task: test
    volumes:
      - name: dockersock
        path: /var/run/docker.sock

  - name: galaxy
    image: lowess/drone-molecule:2.6.8
    environment:
      GALAXY_TOKEN:
        from_secret: ansible_galaxy_token
    commands:
      - ansible-galaxy login --github-token $GALAXY_TOKEN
      - ansible-galaxy import \
          $DRONE_REPO_NAMESPACE $DRONE_REPO_NAME

volumes:
  - name: dockersock
    host:
      path: /var/run/docker.sock

/var/run/docker.sock

 > Drone - Demo

 > Thanks !

Let's start building a better galaxy, one molecule at a time...

🚀 We are hiring !

> Appendix - Mitogen for Ansible

  • A single network roundtrip is used to execute a step whose code already exists in RAM on the target (saves 4 ms runtime per 1 ms of network latency for every playbook step)
     
  • Processes are aggressively reused, avoiding the cost of invoking Python and recompiling imports, saving 300-800 ms for every playbook step.
     
  • Code is ephemerally cached in RAM, reducing bandwidth usage by an order of magnitude compared to SSH pipelining, with around 5x fewer frames traversing the network in a typical run.
     
  • The effect is most potent on playbooks that execute many short-lived actions, where Ansible’s overhead dominates the cost of the operation, for example when executing large with_items loops to run simple commands or write files.

Mitogen is a Python library for writing distributed self-replicating programs. [...] An extension to Ansible is included that implements connections over Mitogen, replacing embedded shell invocations with pure-Python equivalents invoked via highly efficient remote procedure calls to persistent interpreters tunnelled over SSH.

Expect a 1.25x - 7x speedup and a CPU usage reduction of at least 2x.

Delivering quality automations with Ansible, Molecule and Drone

By Corey Gale

Source: Florian Dambrine

Delivering quality automations with Ansible, Molecule and Drone

  • 861

More from Corey Gale