EMBEDDING and EXTENDING Ansible with Python

Alejandro Guirao Rodríguez

@lekum

github.com/lekum

INTRODUCTION TO ANSIBLE

Source: Ansible Blog

WHAT ANSIBLE IS

• It is a Configuration Management tool

• Strong points
  • It works over SSH and is agentless
    • Python 2.X on remote hosts
  • Very readable (YAML)

Architecture overview

Source: sysadmincasts

Host file (inventory)

[web]
10.0.15.21
10.0.15.22

[db]
10.0.15.23

[db:vars]
db_user=guest

[infrastructure:children]
web
db

Ad Hoc Commands

• Throw-away, one-time actions

ansible group -m module -a options

• Execution of an Ansible module in each host of a group

Hello World! : Ping

(ansible)alex ~ $ ansible all -m ping
10.0.15.22 | success >> {
    "changed": false, 
    "ping": "pong"
}

10.0.15.21 | success >> {
    "changed": false, 
    "ping": "pong"
}

10.0.15.23 | success >> {
    "changed": false, 
    "ping": "pong"
}

ansible all -m ping

PLAYBOOKS

• The playbooks are YAML files that specify a list of plays

• Each play is a series of related tasks applied to a group of hosts

- name: Ensure that nginx is started
  service: name=nginx state=started

• Each task is the execution of a module with some parameters and a readable name:

A playbook to deploy nginx

---
- hosts: web
  sudo: yes
  vars:
      - external_port: 80
      - internal_port: 8000

  tasks:

      - name: Add the apt repository for nginx
        apt_repository: repo="ppa:nginx/stable" update_cache=yes

      - name: Install nginx
        apt: name=nginx state=present

      - name: Ensure that nginx is started
        service: name=nginx state=started

      - name: Remove default site
        file: path=/etc/nginx/sites-enabled/default state=absent
        notify:
            - Restart nginx

      - name: Configure a site
        template: src=templates/site.j2 dest=/etc/nginx/sites-available/site

      - name: Enable a site
        file: src=/etc/nginx/sites-available/site dest=/etc/nginx/sites-enabled/site state=link
        notify:
            - Restart nginx

  handlers:
      - name: Restart nginx
        service: name=nginx state=restarted

intro/deploy-nginx.yml

Let's run the playbook

$ ansible-playbook deploy-nginx.yml 

PLAY [web] ******************************************************************** 

GATHERING FACTS *************************************************************** 
ok: [10.0.15.21]
ok: [10.0.15.22]

TASK: [Add the apt repository for nginx] ************************************** 
changed: [10.0.15.21]
changed: [10.0.15.22]

TASK: [Install nginx] ********************************************************* 
changed: [10.0.15.22]
changed: [10.0.15.21]

TASK: [Ensure that nginx is started] ****************************************** 
ok: [10.0.15.21]
ok: [10.0.15.22]

TASK: [Remove default site] *************************************************** 
changed: [10.0.15.21]
changed: [10.0.15.22]

TASK: [Configure a site] ****************************************************** 
changed: [10.0.15.21]
changed: [10.0.15.22]

TASK: [Enable a site] ********************************************************* 
changed: [10.0.15.21]
changed: [10.0.15.22]

TASK: [Remove default site] *************************************************** 
ok: [10.0.15.21]
ok: [10.0.15.22]

NOTIFIED: [Restart nginx] ***************************************************** 
changed: [10.0.15.21]
changed: [10.0.15.22]

PLAY RECAP ******************************************************************** 
10.0.15.21                 : ok=9    changed=6    unreachable=0    failed=0   
10.0.15.22                 : ok=9    changed=6    unreachable=0    failed=0  

Encapsulation: roles

Roles pack:

• Variables
• Tasks
• Handlers
• Files and templates
• Dependencies to other roles

Embedding Ansible

EMBEDDING?

• Embedding: calling  Ansible modules and and executing playbooks from your Python code

• This is possible because Ansible has a Python API
  • Although not very well documented
  • ... but it is quite easy to use

BEFORE WE START...

• The sample code is valid for Ansible v 1.9.X.
  • ¿v2?

•  Python 2.X only :-(

• The source code of the examples are available at

github.com/lekum/ee-ansible-with-python

RUNNING AN ANSIBLE TASK

BASICS

• Import these classes:

from ansible.runner import Runner
from ansible.inventory import Inventory

• Build your inventory:

inventory = Inventory(["localhost"])

• Call the Runner.run():

runner = Runner(module_name='setup', module_args='',
                pattern='localhost', inventory=inventory,
                transport="local")
result = runner.run()

About facts

ansible all -m setup

• Hostnames
• IP addresses
• Hardware and devices attached
• Installed software versions...

You'll get:

EXAMPLE: FLASK FACTER

from ansible.runner import Runner
from ansible.inventory import Inventory
from flask import Flask
from flask_restful import Resource, Api

class AnsibleFactResource(Resource):
    """
    A read-only local ansible fact
    """

    def __init__(self):
        self.facts = None
        Resource.__init__(self)

    def get_local_facts(self):
        """
        Loads the Ansible local facts in self.facts
        Calls the Ansible Python API v1 'setup' module
        """
        inventory = Inventory(["localhost"])
        runner = Runner(module_name='setup', module_args='',
                               pattern='localhost', inventory=inventory,
                               transport="local")
        result = runner.run()
        self.facts = result["contacted"]["localhost"]["ansible_facts"]

    def get(self, fact_name):
        """
        Returns a top-level fact (not nested)
        """
        self.get_local_facts()
        try:
            return self.facts[fact_name]
        except KeyError:
            return "Not a valid fact: %s" % fact_name

app = Flask(__name__)
api = Api(app)
api.add_resource(AnsibleFactResource, '/facts/<string:fact_name>')

if __name__ == '__main__':
    app.run(debug=True, port=8000)

embedding/task/flask_facter.py

RUNNING AN ANSIBLE PLAYBOOK

BASICS

• Import these classes :

from ansible.playbook import PlayBook
from ansible.inventory import Host, Inventory
from ansible import callbacks
from ansible import utils

• Build your inventory

    # Create the inventory
    controller = Host(name = "localhost")
    controller.set_variable('users', user_list)
    controller.set_variable('apt_packages', package_list)
    local_inventory = Inventory([])
    local_inventory.get_group('all').add_host(controller)

BASICS

• Set the callbacks setup

    # Boilerplate for callbacks setup
    utils.VERBOSITY = 0
    # Output callbacks setup
    output_callbacks = callbacks.PlaybookCallbacks(verbose=utils.VERBOSITY)
    # API callbacks setup
    stats = callbacks.AggregateStats()
    api_callbacks = callbacks.PlaybookRunnerCallbacks(stats, verbose=utils.VERBOSITY)

• Call the PlayBook.run()

    provision_playbook = PlayBook(playbook = "installer.yml",
                                  stats = stats,
                                  callbacks = output_callbacks,
                                  runner_callbacks = api_callbacks,
                                  inventory = local_inventory,
                                  transport = "local",
                                  become_pass = sudo_password
                                 )
    playbook_result = provision_playbook.run()

EXAMPLE: INSTALLER

import sys
from getpass import getpass

from ansible.playbook import PlayBook
from ansible.inventory import Host, Inventory
from ansible import callbacks
from ansible import utils

def run_installer(user_list, package_list, sudo_password):
    """
    Runs the playbook `installer.yml` with the supplied parameters
    """

    # Create the inventory
    controller = Host(name = "localhost")
    controller.set_variable('users', user_list)
    controller.set_variable('apt_packages', package_list)
    local_inventory = Inventory([])
    local_inventory.get_group('all').add_host(controller)

    # Boilerplate for callbacks setup
    utils.VERBOSITY = 0
    # Output callbacks setup
    output_callbacks = callbacks.PlaybookCallbacks(verbose=utils.VERBOSITY)
    # API callbacks setup
    stats = callbacks.AggregateStats()
    api_callbacks = callbacks.PlaybookRunnerCallbacks(stats, verbose=utils.VERBOSITY)

    provision_playbook = PlayBook(playbook = "installer.yml",
                                  stats = stats,
                                  callbacks = output_callbacks,
                                  runner_callbacks = api_callbacks,
                                  inventory = local_inventory,
                                  transport = "local",
                                  become_pass = sudo_password
                                 )
    playbook_result = provision_playbook.run()
    return playbook_result

def get_selection_list(initial_prompt, input_prompt, continue_prompt):
    """
    Return a selection list chosen according to a flow described by:
        - initial_prompt: To enter the selection menu
        - input_prompt: To enter an item
        - continue_prompt: To continue entering another item
    """
    results = []
    enter_selection = raw_input(initial_prompt)
    enter_selection = True if enter_selection in ["y", "Y", "yes"] else False
    if enter_selection:
        while True:
            current_result = raw_input(input_prompt)
            if not current_result:
                break
            results.append(current_result)
            continue_selection = raw_input(continue_prompt)
            continue_selection = True if continue_selection in ["y","Y","yes"] else False
            if not continue_selection:
                break
    return results


if __name__ == '__main__':

    packages = []

    print("")
    print("Automated installation script")
    print("=============================")
    print("")

    sudo_password = getpass("Enter sudo password:")

    users = get_selection_list(
                               "Create users? (y/N)",
                               "Enter username:",
                               "Add more users? (y/N)"
                              )

    print("")

    packages = get_selection_list(
                                  "Install packages? (y/N)",
                                  "Enter package name:",
                                  "Add more packages? (y/N)"
                              )

    run_installer(user_list=users, package_list=packages, sudo_password=sudo_password)

embedding/playbook/installer.py

EXTENDING ANSIBLE

Source: Ansible Blog

EXTENDING?

• Extending: adding more functionality to Ansible or customising its behaviour with Python code

• Modules

• Plugins

• Dynamic inventory scripts

CREATING AN ANSIBLE MODULE

AN ANSIBLE MODULE ...

• ... is an executable file
  • In ./library or ANSIBLE_LIBRARY_PATH

• ... has a JSON interface, programming language agnostic
  • Although it's easier with Python

#!/usr/bin/python
# -*- coding: utf-8 -*-

DOCUMENTATION = '''
'''

EXAMPLES = '''
'''
# Your custom Python code / external libraries
from mypackage import mypythonicfunction  

def mypythonicfunction(...):
    """
    Here you do the hard work. Use external libraries or your own code
    And return a meaningful value and message to main()
    """ 
    return status, msg

def main():
    module = AnsibleModule(
        argument_spec=dict(
        # Here you parse the arguments
        )
        # and set the options and behaviour of Ansible
    )

    # Here you assign your arguments to Python variables
    variable_a = module.params['var_a']

    # and call your mypythonicfunction() with the variables
    status, msg = mypythonicfunction(variable_a, ...)

    # Parse the results and return valid JSONs and meaningful messages
    if status == "ok":
        module.exit_json(changed=changed, msg=msg)
    else:
        module.fail_json(msg=msg)

# Boilerplate but IMPORTANT code
from ansible.module_utils.basic import *
main()

STRUCTURE OF THE MODULE

DOCUMENTATION AND EXAMPLES

• Used by make webdocs to generate HTML

Keypoints:

• Keep the options part in sync with the argument_spec

• Specify the requirements

• Use the notes section

• Test your examples :-)

the pythonic part

• However, there are helper funcions
  • run_command

• Try to use pure Python
  • Business logic

• Tips:
  • Return a value and a msg
  • Don't print to stdout or stderr!

THE MAIN() FUNCTION

• AnsibleModule instance

• Don't forget:
  • Mutual exclusion
  • Support for check mode

• The argument_spec dict defines module arguments
  • Required vs optional
  • Default values
  • Possible choices
  • Aliases

THE MAIN() FUNCTION

• Life is easy: the params dict

• Call the pythonic part and parse the results

• Super-handy helper functions:

exit_json(changed, ...)

fail_json(msg, ...)

BOILERPLATE CODE

• Put these two lines at the end of the file:

• Resist the temptation to make explicit imports!

from ansible.module_utils.basic import *
main()

MODULE CREATION TIPS

• Make a module you would love to use:
  • Idempotent
  • Check mode

• Test your module with the test-module script:

ansible/hacking/test-module
-m /path/to/module
-a module_args

• Follow the module creation checklist

MODULE EXAMPLE: TAIGA_ISSUE

• It uses python-taiga to deal with the Taiga REST API

extending/module/library/taiga_issue.py

CREATING A DYNAMIC INVENTORY SCRIPT

DYNAMIC INVENTORY SCRIPTS

• An executable file that supports this command line switches:
  • --host=<hostname> for showing host details
  • --list for listing groups

• The output is JSON

EXAMPLE: SHELVE_INVENTORY

#!/usr/bin/env python

import argparse
import json
import sys
import shelve
import ConfigParser

def parse_args():
    parser = argparse.ArgumentParser(description="Shelve inventory script")
    group = parser.add_mutually_exclusive_group(required=True)
    group.add_argument('--list', action='store_true')
    group.add_argument('--host')
    return parser.parse_args()

def get_shelve_groups(shelvefile):
    d = shelve.open(shelvefile)
    g = d["groups"]
    d.close()
    return g

def get_shelve_host_details(shelvefile, host):
    d = shelve.open(shelvefile)
    h = d["hostvars"][host]
    d.close()
    return h

def main():

    config = ConfigParser.RawConfigParser()
    config.read('shelve_inventory.ini')
    shelvefile = config.get('defaults', 'shelvefile')

    args = parse_args()
    if args.list:
        groups = get_shelve_groups(shelvefile)
        json.dump(groups, sys.stdout)
    else:
        details = get_shelve_host_details(shelvefile, args.host)
        json.dump(details, sys.stdout)

if __name__ == '__main__':
    main()

extending/dynamic_inventory/shelve_inventory.py

CREATING PLUGINS

PLUGINS

• They run in the Controller node!

• Different kinds of plugins:
  • Callback plugins
  • Connection plugins
  • Lookup plugins
  • Filter plugins
  • Action plugins
  • Vars plugins
  • Cache plugins

How To add PLUGINS

• Or you can tweak ansible.cfg:

[defaults]
callback_plugins = ./plugins

• Default location: /usr/share/ansible/plugins/

action_plugins/
lookup_plugins/
callback_plugins/
connection_plugins/
filter_plugins/
vars_plugins/

CREATING A CALLBAck PLUGIN

CALLBACK PLUGINS

• React to Playbook and Runner events

• Define the CallbackModule class and override the methods you want to

• You can find an example with an extensive list of available methods

EXAMPLE: NOTIFY_SEND

from subprocess import call

def notify_send(title, message, icon):
    call(["notify-send", title, message,"-i", icon])

class CallbackModule(object):

    def runner_on_failed(self, host, res, ignore_errors=False):
        template = "Host: {}\nModule: {}\nMessage: {}"
        notification = template.format(
                                       host,
                                       res.get('invocation').get('module_name'),
                                       res.get('msg')
                                      )
        notify_send("ANSIBLE FAILURE", notification, "dialog-warning")

extending/plugin/callbacks/plugins/notify_send.py

CREATING A CONNECTION PLUGIN

CONNECTION PLUGINS

• A connection plugin allows the controller to connect to the hosts using different protocols:
  • SSH
  • paramiko_ssh
  • local
  • winrm (for Win$ hosts)
  • chroot
  • jails (for FreeBSD)
  • zones (Solaris)
  • libvirt_lxc
  • ...

• You can find examples the list of official connection plugins here

CONNECTION PLUGIN BASICS

• Define a Connection class and override the following methods:

class Connection(object):

    def __init__(self, runner, host, port, user,
                 password, private_key_file, *args, **kwargs):
        [...]

     def connect(self):
        [...]

     def exec_command(self, cmd, tmp_path, become_user=None,
                      sudoable=False, executable='/bin/sh', in_data=None):
        [...]

     def put_file(self, in_path, out_path):
        [...]

     def fetch_file(self, in_path, out_path):
        [...]

     def close(self):
        [...]

CREATING A LOOKUP PLUGIN

LOOKUP PLUGINS

• Data access
  • Filesystem
  • Databases
  • External services

• Syntax:

contents: "{{ lookup('file', '/etc/foo.txt') }}"

• Base of the with_ expression

LOOKUP PLUGIN BASICS

from ansible import utils, errors

class LookupModule(object):

    def __init__(self, basedir=None, **kwargs):
        self.basedir = basedir

    def run(self, terms, inject=None, **kwargs):
        # terms holds the parameters of the lookup call

• Implement a run() method of the LookupModule class that returns a list of results
  • Each one corresponding to each element of the terms list

EXAMPLE: SHELVEFILE

import shelve
import os
from ansible import utils, errors

class LookupModule(object):

    def __init__(self, basedir=None, **kwargs):
        self.basedir = basedir

    def read_shelve(self, shelve_filename, key):
        """
        Read the value of "key" from a shelve file
        """
        d = shelve.open(shelve_filename)
        res = d.get(key, None)
        d.close()
        return res

    def run(self, terms, inject=None, **kwargs):

        terms = utils.listify_lookup_plugin_terms(terms, self.basedir, inject)
        ret = []

        if not isinstance(terms, list):
            terms = [ terms ]

        for term in terms:
            playbook_path = None
            relative_path = None
            paramvals = {"file": None, "key": None}
            params = term.split()

            try:
                for param in params:
                    name, value = param.split('=')
                    assert(name in paramvals)
                    paramvals[name] = value

            except (ValueError, AssertionError), e:
                # In case "file" or "key" are not present
                raise errors.AnsibleError(e)

            file = paramvals['file']
            key = paramvals['key']
            basedir_path  = utils.path_dwim(self.basedir, file)

            # Search also in the role/files directory and in the playbook directory
            if '_original_file' in inject:
                relative_path = utils.path_dwim_relative(inject['_original_file'], 'files', file, self.basedir, check=False)
            if 'playbook_dir' in inject:
                playbook_path = os.path.join(inject['playbook_dir'], file)

            for path in (basedir_path, relative_path, playbook_path):
                if path and os.path.exists(path):
                    res = self.read_shelve(path, key)
                    if res is None:
                        raise errors.AnsibleError("Key %s not found in shelve file %s" % (key, file))
                    # Convert the value read to string
                    ret.append(str(res))
                    break
            else:
                raise errors.AnsibleError("Could not locate shelve file in lookup: %s" % file)

        return ret

extending/plugin/lookup/plugins/shelvefile.py

Usage:

{{ lookup('shelvefile', 'file=book.db key=current_book_author') }}

CREATING A FILTER PLUGIN

FILTER PLUGINS

• Jinja2 filters

• Ansible ships with some useful ones

{{ some_variable | default(5) }}
{{ some_list | min }}
{{ some_register_variable | changed }}

• Syntax: variable | filter

FILTER PLUGIN BASICS

class FilterModule(object):

    def filters(self):
        return {
                'filter1': filter1functionname,
                'filter2': filter2functionname,
                [...]
               }

• Create a FilterModule class with a filters method that returns a dictionary mapping the filters to functions

EXAMPLE: ROT13

def rot13(s):
    def lookup(v):
        o, c = ord(v), v.lower()
        if 'a' <= c <= 'm':
            return chr(o + 13)
        if 'n' <= c <= 'z':
            return chr(o - 13)
        return v
    return ''.join(map(lookup, s))

class FilterModule(object):

    def filters(self):
        return {
                'rot13': rot13
               }

extending/plugin/filter/plugins/rot13.py

Usage:

{{ variable | rot13 }}

OTHER PLUGINS

ACTION PLUGINS

• Controller actions for the module

• To develop one, implement a class ActionModule with a method run()
  •  self.runner._execute_module() to call the real module

VARS PLUGINS

• Playbook constructs like host_vars and group_vars work via vars plugins

• It is quite an obscure and undocumented feature, I have only found this code sample in the Ansible repository

VARS PLUGIN TEMPLATE CODE

class VarsModule(object):

    """
    Loads variables for groups and/or hosts
    """

    def __init__(self, inventory):

        """ constructor """

        self.inventory = inventory
        self.inventory_basedir = inventory.basedir()


    def run(self, host, vault_password=None):
        """ For backwards compatibility, when only vars per host were retrieved
            This method should return both host specific vars as well as vars
            calculated from groups it is a member of """
        return {}


    def get_host_vars(self, host, vault_password=None):
        """ Get host specific variables. """
        return {}


    def get_group_vars(self, group, vault_password=None):
        """ Get group specific variables. """
        return {}

CACHE PLUGINS

Fact caching:

• hostvars of hosts not contacted in the playbook execution

• v1.8+

• Tweak the ansible.cfg as follows: 

[defaults]
gathering = smart
fact_caching = redis # Name of the plugin
fact_caching_timeout = 86400

CACHE PLUGINS

• Currently supported cache plugins:
  • Memory
  • JSONfile
  • Memcached
  • Redis

CACHE PLUGIN TEMPLATE CODE

import exceptions

class BaseCacheModule(object):

    def get(self, key):
        raise exceptions.NotImplementedError

    def set(self, key, value):
        raise exceptions.NotImplementedError

    def keys(self):
        raise exceptions.NotImplementedError

    def contains(self, key):
        raise exceptions.NotImplementedError

    def delete(self, key):
        raise exceptions.NotImplementedError

    def flush(self):
        raise exceptions.NotImplementedError

    def copy(self):
        raise exceptions.NotImplementedError

REFERENCES

• Ansible documentation

• Ansible Up & Running by Lorin Hochstein

• Learning Ansible by M.Mohan and R.Raithatha
  

Books:

Articles:

• Ansible for DevOps by Jeff Geerling

• Using Ansible like library programming in Python by Oriol Rius

• Running Ansible Programatically by Servers for Hackers

Happy hacking!

github.com/lekum