DELIVERING QUALITY AUTOMATIONS
with Ansible - molecule - drone
@DambrineF
Florian Dambrine - Senior DevOps Engineer - @ GumGum
> Whoami
-
Florian Dambrine
-
DevOps Engineer @ GumGum
-
Joined GumGum 5 years ago
-
Ansible fan!
> Agenda
- Test Driven Infrastructure Concepts
- GumGum - Ansible Shop since 2014
- Ansible workflows past & present
- Ansible repository structure past & present
- Testing strategy past & present
- Molecule
- Introduction
- Concepts
- Demo
> gumgum - Computer vision company
Invented In-Image
advertising in 2008
gumgum Advertising
> gumgum - Computer vision company
gumgum SPORTS
> gumgum - Computer vision company
gumgum DENTAL
> TEST-DRIVEN INFRASTRUCTURE
- Make sure the automation worked as expected
- Detect breaking changes earlier
- Easily handle configuration management upgrades
- Test automation with multiple software versions
-
Test complex infrastructure configurations:
- Clusters depending on other ones (Zookeeper dependency)
- Opscenter / Cassandra / Maintenance operations / Rolling restarts
- Enforce Automation quality and best-practices
> GumGum - Ansible shop since 2014
~/Workspace/ops/ansible ● tree -L 2 -I '*.yaml'
.
├── ansible.cfg
├── <playbook>.yml
├── inventories/
│ ├── ireland
│ │ ├── ec2.ini
│ │ ├── ec2.py
│ │ └── group_vars
│ ├── japan
│ │ ├── ec2.ini
│ │ ├── ec2.py
│ │ └── group_vars
│ ├── oregon
│ │ ├── ec2.ini
│ │ ├── ec2.py
│ │ └── group_vars
│ └── virginia
│ │ ├── ec2.ini
│ │ ├── ec2.py
│ │ └── group_vars
├── library/
├── requirements.yml
└── roles/
└── ...
+200 roles
> Ansible Workflows - Past & Present
- Provision
- Configure
- Test
- Cleanup
- Snapshot
- Deployment
Cluster Automation
Image creation
Previous Workflow
Current Workflow
- Provision
- Configure
- Test
- Cleanup
- Snapshot
- Deployment
DEPLOYMENT
> Ansible Repository structure - Past ...
...
### About 200 ansible roles
├── roles
│ ├── PHPCi-001
│ ├── activemq-001
│ ├── ad-server-001
│ ├── advertising-api-001
│ ├── ansible-001
│ ├── ansible-metadata-001
│ ├── apt-cacher-ng-001
│ ├── aws-api-server-002
│ ├── aws-cli-001
│ ├── aws-mon-linux-001
│ ├── bid-predictor-001
│ ├── bid-predictor-002
│ ├── bower-001
│ ├── caffe-on-spark-001
│ ├── cassandra-001
│ ├── cassandra-002
│ ├── cassandra-cleanup-001
│ ├── cassandra-clearsnapshot-001
│ ├── cassandra-compaction-001
│ ├── cassandra-firewall-001
│ ├── cassandra-repair-001
│ ├── cassandra-rolling-restart-001
│ ├── cloudfront-origin-001
│ ├── codedeploy-001
│ ├── collectd-001
│ ├── common-001
...
cons
- Even though an effort was made a few years ago to CI/CD this monolith, no one was actually paying attention to it:
- Too complex
- Too heavy
- No straight forward way to test locally (even though there was one)
- More and more people writing Ansible roles
- Still not good way to version roles
> Ansible Repository structure - ... And present
├── ansible-role-aws-cli
Pros
- 1 Ansible role = 1 Repository
- Easy to CI / CD
- Easy to contribute to
- Easy to version
- Easy to upgrade
- Gain modularity and reusability
- Open source openning
├── ansible-role-common
├── ansible-role-elasticsearch
├── ansible-role-gradle
├── ansible-role-groovy
├── ansible-role-java
...
ansible-ops
ansible-data-engineering
ansible-web-engineering
ansible-data-science
> Ansible Testing Strategy - Past ...
- Ansible-Lint
- Container provision
- Ansible configure.yaml
- Ansible test.yaml
- ServerSpec tests
def getPlaybookList() {
playbooks = [
"queue-processor-001.yaml",
"cron-001.yaml",
"geo-server-003.yaml",
"redis-001.yaml",
"reporting-server-001.yaml",
"taskrunner-001.yaml",
"tomcat-001.yaml"
"s3s3mirror-001.yaml",
"spark-001.yaml"
]
playbooks ? playbooks : null
}
for each $playbook
> Ansible Testing Strategy - ... And present
- Ansible-Lint
- Container provision
- Ansible playbook.yml
- Ansible side-effect.yml
- Testinfra tests
for the $role
Introduction / Concepts / Demo
> Molecule - Introduction
Molecule provides a simple framework for easily and repeatedly testing your roles and playbooks against different environments and operating systems
- Project originally created by retr0h
- Recently moved to Ansible RedHat (October 2018)
- Now integrated with Ansible Galaxy
$ pip install molecule
docker run --rm -it \
-v '$(pwd)':/tmp/$(basename "${PWD}"):ro \
-v /var/run/docker.sock:/var/run/docker.sock \
-w /tmp/$(basename "${PWD}") \
quay.io/ansible/molecule:latest \
sudo molecule test
Getting started
> Molecule - Concepts
Folder structure
<rolename>
├── README.md
├── defaults
│ └── main.yml
├── handlers
│ └── main.yml
├── meta
│ └── main.yml
├── molecule
│ └── default
│ ├── Dockerfile.j2
│ ├── INSTALL.rst
│ ├── create.yml
│ ├── destroy.yml
│ ├── molecule.yml
│ ├── playbook.yml
│ ├── prepare.yml
│ └── tests
│ └── test_default.py
├── tasks
│ └── main.yml
└── vars
└── main.yml
Molecule internals
Developer files
Linters
Dependencies
Drivers
Verifiers
goss
> Molecule - Concepts
molecule.yml
lint:
name: yamllint
dependency:
name: galaxy
options:
role-file: ${PWD}/requirements.yml
driver:
name: docker
platforms:
- name: ubuntu_trusty
image: ubuntu:trusty
...
- name: ubuntu_xenial
image: solita/ubuntu-systemd:xenial
...
provisioner:
name: ansible
config_options:
defaults:
callback_whitelist: timer
inventory:
links:
group_vars: ../inventory/group_vars/
lint:
name: ansible-lint
enabled: true
scenario:
name: default
converge_sequence:
- dependency
- create
- prepare
- converge
- idempotence
verifier:
name: testinfra
lint:
name: flake8
> Molecule - Concepts
- A scenario is a self-contained directory containing everything necessary for testing the role in a particular way
- The default scenario is named default, and every role should contain a default scenario
- A Scenario defines a sequence of actions to execute for a command
Scenarios
scenario:
name: default
create_sequence:
- create
- prepare
check_sequence:
- destroy
- dependency
- create
- prepare
- converge
- check
- destroy
converge_sequence:
- dependency
- create
- prepare
- converge
destroy_sequence:
- destroy
test_sequence:
- lint
- destroy
- dependency
- syntax
- create
- prepare
- converge
- idempotence
- side_effect
- verify
- destroy
command
sequence
> Molecule - Concepts
$ molecule init template --url \
https://github.com/Lowess/ansible-role-cookiecutter
--> Initializing new role role_name...
role_name [role_name]: meetup-demo
role_description [TODO]: Cookiecutter is awesome !
role_author [Your Name]: Florian Dambrine
min_ansible_version [2.0]: 2.2
Initialized role in /tmp/tmp/role_name successfully.
$ tree ansible-role-meetup-demo -L 2
ansible-role-meetup-demo
├── .ansible-lint
├── .drone.yml
├── .gitignore
├── .pre-commit-config.yml
├── .yamllint
├── README.md
├── defaults
│ └── main.yml
├── handlers
│ └── main.yml
├── meta
│ └── main.yml
├── molecule
│ ├── default
│ ├── inventory
│ └── resources
├── tasks
│ ├── cleanup.yml
│ ├── configure.yml
│ ├── main.yml
│ └── test.yml
└── vars
└── main.yml
- Real time saver !
- Enforce role convention / consistence
- Defines linter rules, pipelines, pre-commit hooks
> Molecule - introducing Gumsible !
docker run --rm -it \
-v '$(pwd)':/tmp/$(basename "${PWD}"):ro \
-v /var/run/docker.sock:/var/run/docker.sock \
-w /tmp/$(basename "${PWD}") \
quay.io/ansible/molecule:latest \
sudo molecule test
gumsible test
Test containers managed by Molecule
Gumsible features
- Bash and ZSH compatible
- ZSH auto completion
- Run useful sidecar containers
- Finds the root folder of the role automatically
- Easily configurable thanks to a .gumsible config file
- Shortcut of molecule init with Cookiecutter
> Molecule - Custom Docker image
quay.io/ansible/molecule
lowess/drone-molecule
Vs
Missing git+ssh (PR #1604)
Latest Ansible version only
Drone CI plugin
Offers git+ssh (private repos)
Offers multiple Ansible versions
Extra ansible-lint rules
Mitogen plugin available
DEMO time !
Concepts / Demo
Drone.io
> Drone - concepts
# .drone.yml
---
kind: pipeline
name: ansible-ci
workspace:
base: /drone
path: src/${DRONE_REPO}
steps:
- name: molecule
image: lowess/drone-molecule:2.6.8
environment:
CI_UUID: _2.6_${DRONE_COMMIT_SHA:0:8}
ANSIBLE_STRATEGY: mitogen_linear
pull: true
settings:
task: test
volumes:
- name: dockersock
path: /var/run/docker.sock
- name: galaxy
image: lowess/drone-molecule:2.6.8
environment:
GALAXY_TOKEN:
from_secret: ansible_galaxy_token
commands:
- ansible-galaxy login --github-token $GALAXY_TOKEN
- ansible-galaxy import \
$DRONE_REPO_NAMESPACE $DRONE_REPO_NAME
volumes:
- name: dockersock
host:
path: /var/run/docker.sock
/var/run/docker.sock
> Drone - Demo
> Thanks !
Let's start building a better galaxy, one molecule at a time...
🚀 We are hiring !
> Appendix - Mitogen for Ansible
-
A single network roundtrip is used to execute a step whose code already exists in RAM on the target (saves 4 ms runtime per 1 ms of network latency for every playbook step)
-
Processes are aggressively reused, avoiding the cost of invoking Python and recompiling imports, saving 300-800 ms for every playbook step.
- Code is ephemerally cached in RAM, reducing bandwidth usage by an order of magnitude compared to SSH pipelining, with around 5x fewer frames traversing the network in a typical run.
- The effect is most potent on playbooks that execute many short-lived actions, where Ansible’s overhead dominates the cost of the operation, for example when executing large with_items loops to run simple commands or write files.
Mitogen is a Python library for writing distributed self-replicating programs. [...] An extension to Ansible is included that implements connections over Mitogen, replacing embedded shell invocations with pure-Python equivalents invoked via highly efficient remote procedure calls to persistent interpreters tunnelled over SSH.
Expect a 1.25x - 7x speedup and a CPU usage reduction of at least 2x.
Delivering quality automations with Ansible, Molecule and Drone
By Corey Gale
Delivering quality automations with Ansible, Molecule and Drone
- 842