EMBEDDING and EXTENDING Ansible with Python

Alejandro Guirao Rodríguez

@lekum

github.com/lekum

INTRODUCTION TO ANSIBLE

Source: Ansible Blog

WHAT ANSIBLE IS

  • It is a Configuration Management tool
  • Strong points
    • It works over SSH and is agentless
      • Python 2.X on remote hosts
    • Very readable (YAML)

Architecture overview

Source: sysadmincasts

Host file (inventory)

[web]
10.0.15.21
10.0.15.22

[db]
10.0.15.23

[db:vars]
db_user=guest

[infrastructure:children]
web
db

Ad Hoc Commands

  • Throw-away, one-time actions
ansible group -m module -a options
  • Execution of an Ansible module in each host of a group

Hello World! : Ping

(ansible)alex ~ $ ansible all -m ping
10.0.15.22 | success >> {
    "changed": false, 
    "ping": "pong"
}

10.0.15.21 | success >> {
    "changed": false, 
    "ping": "pong"
}

10.0.15.23 | success >> {
    "changed": false, 
    "ping": "pong"
}
ansible all -m ping

PLAYBOOKS

  • The playbooks are YAML files that specify a list of plays
  • Each play is a series of related tasks applied to a group of hosts
- name: Ensure that nginx is started
  service: name=nginx state=started
  • Each task is the execution of a module with some parameters and a readable name:

A playbook to deploy nginx

---
- hosts: web
  sudo: yes
  vars:
      - external_port: 80
      - internal_port: 8000

  tasks:

      - name: Add the apt repository for nginx
        apt_repository: repo="ppa:nginx/stable" update_cache=yes

      - name: Install nginx
        apt: name=nginx state=present

      - name: Ensure that nginx is started
        service: name=nginx state=started

      - name: Remove default site
        file: path=/etc/nginx/sites-enabled/default state=absent
        notify:
            - Restart nginx

      - name: Configure a site
        template: src=templates/site.j2 dest=/etc/nginx/sites-available/site

      - name: Enable a site
        file: src=/etc/nginx/sites-available/site dest=/etc/nginx/sites-enabled/site state=link
        notify:
            - Restart nginx

  handlers:
      - name: Restart nginx
        service: name=nginx state=restarted
intro/deploy-nginx.yml

Let's run the playbook

$ ansible-playbook deploy-nginx.yml 

PLAY [web] ******************************************************************** 

GATHERING FACTS *************************************************************** 
ok: [10.0.15.21]
ok: [10.0.15.22]

TASK: [Add the apt repository for nginx] ************************************** 
changed: [10.0.15.21]
changed: [10.0.15.22]

TASK: [Install nginx] ********************************************************* 
changed: [10.0.15.22]
changed: [10.0.15.21]

TASK: [Ensure that nginx is started] ****************************************** 
ok: [10.0.15.21]
ok: [10.0.15.22]

TASK: [Remove default site] *************************************************** 
changed: [10.0.15.21]
changed: [10.0.15.22]

TASK: [Configure a site] ****************************************************** 
changed: [10.0.15.21]
changed: [10.0.15.22]

TASK: [Enable a site] ********************************************************* 
changed: [10.0.15.21]
changed: [10.0.15.22]

TASK: [Remove default site] *************************************************** 
ok: [10.0.15.21]
ok: [10.0.15.22]

NOTIFIED: [Restart nginx] ***************************************************** 
changed: [10.0.15.21]
changed: [10.0.15.22]

PLAY RECAP ******************************************************************** 
10.0.15.21                 : ok=9    changed=6    unreachable=0    failed=0   
10.0.15.22                 : ok=9    changed=6    unreachable=0    failed=0  

Encapsulation: roles

Roles pack:

  • Variables
  • Tasks
  • Handlers
  • Files and templates
  • Dependencies to other roles

Embedding Ansible

EMBEDDING?

  • Embedding: calling  Ansible modules and and executing playbooks from your Python code

 

BEFORE WE START...

  • The sample code is valid for Ansible v 1.9.X.
    • ¿v2?
  •  Python 2.X only :-(
  • The source code of the examples are available at

RUNNING AN ANSIBLE TASK

BASICS

  • Import these classes:
from ansible.runner import Runner
from ansible.inventory import Inventory
  • Build your inventory:
inventory = Inventory(["localhost"])
  • Call the Runner.run():
runner = Runner(module_name='setup', module_args='',
                pattern='localhost', inventory=inventory,
                transport="local")
result = runner.run()

About facts

ansible all -m setup
  • Hostnames
  • IP addresses
  • Hardware and devices attached
  • Installed software versions...

You'll get:

EXAMPLE: FLASK FACTER

from ansible.runner import Runner
from ansible.inventory import Inventory
from flask import Flask
from flask_restful import Resource, Api

class AnsibleFactResource(Resource):
    """
    A read-only local ansible fact
    """

    def __init__(self):
        self.facts = None
        Resource.__init__(self)

    def get_local_facts(self):
        """
        Loads the Ansible local facts in self.facts
        Calls the Ansible Python API v1 'setup' module
        """
        inventory = Inventory(["localhost"])
        runner = Runner(module_name='setup', module_args='',
                               pattern='localhost', inventory=inventory,
                               transport="local")
        result = runner.run()
        self.facts = result["contacted"]["localhost"]["ansible_facts"]

    def get(self, fact_name):
        """
        Returns a top-level fact (not nested)
        """
        self.get_local_facts()
        try:
            return self.facts[fact_name]
        except KeyError:
            return "Not a valid fact: %s" % fact_name

app = Flask(__name__)
api = Api(app)
api.add_resource(AnsibleFactResource, '/facts/<string:fact_name>')

if __name__ == '__main__':
    app.run(debug=True, port=8000)
embedding/task/flask_facter.py

RUNNING AN ANSIBLE PLAYBOOK

BASICS

  • Import these classes :
from ansible.playbook import PlayBook
from ansible.inventory import Host, Inventory
from ansible import callbacks
from ansible import utils
  • Build your inventory
    # Create the inventory
    controller = Host(name = "localhost")
    controller.set_variable('users', user_list)
    controller.set_variable('apt_packages', package_list)
    local_inventory = Inventory([])
    local_inventory.get_group('all').add_host(controller)

BASICS

  • Set the callbacks setup
    # Boilerplate for callbacks setup
    utils.VERBOSITY = 0
    # Output callbacks setup
    output_callbacks = callbacks.PlaybookCallbacks(verbose=utils.VERBOSITY)
    # API callbacks setup
    stats = callbacks.AggregateStats()
    api_callbacks = callbacks.PlaybookRunnerCallbacks(stats, verbose=utils.VERBOSITY)
  • Call the PlayBook.run()
    provision_playbook = PlayBook(playbook = "installer.yml",
                                  stats = stats,
                                  callbacks = output_callbacks,
                                  runner_callbacks = api_callbacks,
                                  inventory = local_inventory,
                                  transport = "local",
                                  become_pass = sudo_password
                                 )
    playbook_result = provision_playbook.run()

EXAMPLE: INSTALLER

import sys
from getpass import getpass

from ansible.playbook import PlayBook
from ansible.inventory import Host, Inventory
from ansible import callbacks
from ansible import utils

def run_installer(user_list, package_list, sudo_password):
    """
    Runs the playbook `installer.yml` with the supplied parameters
    """

    # Create the inventory
    controller = Host(name = "localhost")
    controller.set_variable('users', user_list)
    controller.set_variable('apt_packages', package_list)
    local_inventory = Inventory([])
    local_inventory.get_group('all').add_host(controller)

    # Boilerplate for callbacks setup
    utils.VERBOSITY = 0
    # Output callbacks setup
    output_callbacks = callbacks.PlaybookCallbacks(verbose=utils.VERBOSITY)
    # API callbacks setup
    stats = callbacks.AggregateStats()
    api_callbacks = callbacks.PlaybookRunnerCallbacks(stats, verbose=utils.VERBOSITY)

    provision_playbook = PlayBook(playbook = "installer.yml",
                                  stats = stats,
                                  callbacks = output_callbacks,
                                  runner_callbacks = api_callbacks,
                                  inventory = local_inventory,
                                  transport = "local",
                                  become_pass = sudo_password
                                 )
    playbook_result = provision_playbook.run()
    return playbook_result

def get_selection_list(initial_prompt, input_prompt, continue_prompt):
    """
    Return a selection list chosen according to a flow described by:
        - initial_prompt: To enter the selection menu
        - input_prompt: To enter an item
        - continue_prompt: To continue entering another item
    """
    results = []
    enter_selection = raw_input(initial_prompt)
    enter_selection = True if enter_selection in ["y", "Y", "yes"] else False
    if enter_selection:
        while True:
            current_result = raw_input(input_prompt)
            if not current_result:
                break
            results.append(current_result)
            continue_selection = raw_input(continue_prompt)
            continue_selection = True if continue_selection in ["y","Y","yes"] else False
            if not continue_selection:
                break
    return results


if __name__ == '__main__':

    packages = []

    print("")
    print("Automated installation script")
    print("=============================")
    print("")

    sudo_password = getpass("Enter sudo password:")

    users = get_selection_list(
                               "Create users? (y/N)",
                               "Enter username:",
                               "Add more users? (y/N)"
                              )

    print("")

    packages = get_selection_list(
                                  "Install packages? (y/N)",
                                  "Enter package name:",
                                  "Add more packages? (y/N)"
                              )

    run_installer(user_list=users, package_list=packages, sudo_password=sudo_password)
embedding/playbook/installer.py

EXTENDING ANSIBLE

Source: Ansible Blog

EXTENDING?

  • Extending: adding more functionality to Ansible or customising its behaviour with Python code
  • Modules
  • Plugins
  • Dynamic inventory scripts

CREATING AN ANSIBLE MODULE

AN ANSIBLE MODULE ...

  • ... is an executable file
    • In ./library or ANSIBLE_LIBRARY_PATH
  • ... has a JSON interface, programming language agnostic
    • Although it's easier with Python
#!/usr/bin/python
# -*- coding: utf-8 -*-

DOCUMENTATION = '''
'''

EXAMPLES = '''
'''
# Your custom Python code / external libraries
from mypackage import mypythonicfunction  

def mypythonicfunction(...):
    """
    Here you do the hard work. Use external libraries or your own code
    And return a meaningful value and message to main()
    """ 
    return status, msg

def main():
    module = AnsibleModule(
        argument_spec=dict(
        # Here you parse the arguments
        )
        # and set the options and behaviour of Ansible
    )

    # Here you assign your arguments to Python variables
    variable_a = module.params['var_a']

    # and call your mypythonicfunction() with the variables
    status, msg = mypythonicfunction(variable_a, ...)

    # Parse the results and return valid JSONs and meaningful messages
    if status == "ok":
        module.exit_json(changed=changed, msg=msg)
    else:
        module.fail_json(msg=msg)

# Boilerplate but IMPORTANT code
from ansible.module_utils.basic import *
main()

STRUCTURE OF THE MODULE

DOCUMENTATION AND EXAMPLES

  • Used by make webdocs to generate HTML

Keypoints:

  • Keep the options part in sync with the argument_spec
  • Specify the requirements
  • Use the notes section
  • Test your examples :-)

the pythonic part

  • However, there are helper funcions
    • run_command
  • Try to use pure Python
    • Business logic
  • Tips:
    • Return a value and a msg
    • Don't print to stdout or stderr!

THE MAIN() FUNCTION

  • AnsibleModule instance
  • Don't forget:
    • Mutual exclusion
    • Support for check mode
  • The argument_spec dict defines module arguments
    • Required vs optional
    • Default values
    • Possible choices
    • Aliases

THE MAIN() FUNCTION

  • Life is easy: the params dict
  • Call the pythonic part and parse the results
  • Super-handy helper functions:
exit_json(changed, ...)

fail_json(msg, ...)

BOILERPLATE CODE

  • Put these two lines at the end of the file:
  • Resist the temptation to make explicit imports!
from ansible.module_utils.basic import *
main()

MODULE CREATION TIPS

  • Make a module you would love to use:
    • Idempotent
    • Check mode
  • Test your module with the test-module script:
ansible/hacking/test-module
-m /path/to/module
-a module_args

MODULE EXAMPLE: TAIGA_ISSUE

  • It uses python-taiga to deal with the Taiga REST API
extending/module/library/taiga_issue.py

CREATING A DYNAMIC INVENTORY SCRIPT

DYNAMIC INVENTORY SCRIPTS

  • An executable file that supports this command line switches:
    • --host=<hostname> for showing host details
    • --list for listing groups
  • The output is JSON

EXAMPLE: SHELVE_INVENTORY

#!/usr/bin/env python

import argparse
import json
import sys
import shelve
import ConfigParser

def parse_args():
    parser = argparse.ArgumentParser(description="Shelve inventory script")
    group = parser.add_mutually_exclusive_group(required=True)
    group.add_argument('--list', action='store_true')
    group.add_argument('--host')
    return parser.parse_args()

def get_shelve_groups(shelvefile):
    d = shelve.open(shelvefile)
    g = d["groups"]
    d.close()
    return g

def get_shelve_host_details(shelvefile, host):
    d = shelve.open(shelvefile)
    h = d["hostvars"][host]
    d.close()
    return h

def main():

    config = ConfigParser.RawConfigParser()
    config.read('shelve_inventory.ini')
    shelvefile = config.get('defaults', 'shelvefile')

    args = parse_args()
    if args.list:
        groups = get_shelve_groups(shelvefile)
        json.dump(groups, sys.stdout)
    else:
        details = get_shelve_host_details(shelvefile, args.host)
        json.dump(details, sys.stdout)

if __name__ == '__main__':
    main()
extending/dynamic_inventory/shelve_inventory.py

CREATING PLUGINS

PLUGINS

  • They run in the Controller node!
  • Different kinds of plugins:
    • Callback plugins
    • Connection plugins
    • Lookup plugins
    • Filter plugins
    • Action plugins
    • Vars plugins
    • Cache plugins

How To add PLUGINS

  • Or you can tweak ansible.cfg:
[defaults]
callback_plugins = ./plugins
  • Default location: /usr/share/ansible/plugins/
action_plugins/
lookup_plugins/
callback_plugins/
connection_plugins/
filter_plugins/
vars_plugins/

CREATING A CALLBAck PLUGIN

CALLBACK PLUGINS

  • React to Playbook and Runner events
  • Define the CallbackModule class and override the methods you want to
  • You can find an example with an extensive list of available methods

EXAMPLE: NOTIFY_SEND

from subprocess import call

def notify_send(title, message, icon):
    call(["notify-send", title, message,"-i", icon])

class CallbackModule(object):

    def runner_on_failed(self, host, res, ignore_errors=False):
        template = "Host: {}\nModule: {}\nMessage: {}"
        notification = template.format(
                                       host,
                                       res.get('invocation').get('module_name'),
                                       res.get('msg')
                                      )
        notify_send("ANSIBLE FAILURE", notification, "dialog-warning")
extending/plugin/callbacks/plugins/notify_send.py

CREATING A CONNECTION PLUGIN

CONNECTION PLUGINS

  • A connection plugin allows the controller to connect to the hosts using different protocols:
    • SSH
    • paramiko_ssh
    • local
    • winrm (for Win$ hosts)
    • chroot
    • jails (for FreeBSD)
    • zones (Solaris)
    • libvirt_lxc
    • ...
  • You can find examples the list of official connection plugins here

CONNECTION PLUGIN BASICS

  • Define a Connection class and override the following methods:
class Connection(object):

    def __init__(self, runner, host, port, user,
                 password, private_key_file, *args, **kwargs):
        [...]

     def connect(self):
        [...]

     def exec_command(self, cmd, tmp_path, become_user=None,
                      sudoable=False, executable='/bin/sh', in_data=None):
        [...]

     def put_file(self, in_path, out_path):
        [...]

     def fetch_file(self, in_path, out_path):
        [...]

     def close(self):
        [...]

CREATING A LOOKUP PLUGIN

LOOKUP PLUGINS

  • Data access
    • Filesystem
    • Databases
    • External services
  • Syntax:
contents: "{{ lookup('file', '/etc/foo.txt') }}"
  • Base of the with_ expression

LOOKUP PLUGIN BASICS

from ansible import utils, errors

class LookupModule(object):

    def __init__(self, basedir=None, **kwargs):
        self.basedir = basedir

    def run(self, terms, inject=None, **kwargs):
        # terms holds the parameters of the lookup call
  • Implement a run() method of the LookupModule class that returns a list of results
    • Each one corresponding to each element of the terms list

EXAMPLE: SHELVEFILE

import shelve
import os
from ansible import utils, errors

class LookupModule(object):

    def __init__(self, basedir=None, **kwargs):
        self.basedir = basedir

    def read_shelve(self, shelve_filename, key):
        """
        Read the value of "key" from a shelve file
        """
        d = shelve.open(shelve_filename)
        res = d.get(key, None)
        d.close()
        return res

    def run(self, terms, inject=None, **kwargs):

        terms = utils.listify_lookup_plugin_terms(terms, self.basedir, inject)
        ret = []

        if not isinstance(terms, list):
            terms = [ terms ]

        for term in terms:
            playbook_path = None
            relative_path = None
            paramvals = {"file": None, "key": None}
            params = term.split()

            try:
                for param in params:
                    name, value = param.split('=')
                    assert(name in paramvals)
                    paramvals[name] = value

            except (ValueError, AssertionError), e:
                # In case "file" or "key" are not present
                raise errors.AnsibleError(e)

            file = paramvals['file']
            key = paramvals['key']
            basedir_path  = utils.path_dwim(self.basedir, file)

            # Search also in the role/files directory and in the playbook directory
            if '_original_file' in inject:
                relative_path = utils.path_dwim_relative(inject['_original_file'], 'files', file, self.basedir, check=False)
            if 'playbook_dir' in inject:
                playbook_path = os.path.join(inject['playbook_dir'], file)

            for path in (basedir_path, relative_path, playbook_path):
                if path and os.path.exists(path):
                    res = self.read_shelve(path, key)
                    if res is None:
                        raise errors.AnsibleError("Key %s not found in shelve file %s" % (key, file))
                    # Convert the value read to string
                    ret.append(str(res))
                    break
            else:
                raise errors.AnsibleError("Could not locate shelve file in lookup: %s" % file)

        return ret
extending/plugin/lookup/plugins/shelvefile.py

Usage:

{{ lookup('shelvefile', 'file=book.db key=current_book_author') }}

CREATING A FILTER PLUGIN

FILTER PLUGINS

{{ some_variable | default(5) }}
{{ some_list | min }}
{{ some_register_variable | changed }}
  • Syntax: variable | filter

FILTER PLUGIN BASICS

class FilterModule(object):

    def filters(self):
        return {
                'filter1': filter1functionname,
                'filter2': filter2functionname,
                [...]
               }
  • Create a FilterModule class with a filters method that returns a dictionary mapping the filters to functions

EXAMPLE: ROT13

def rot13(s):
    def lookup(v):
        o, c = ord(v), v.lower()
        if 'a' <= c <= 'm':
            return chr(o + 13)
        if 'n' <= c <= 'z':
            return chr(o - 13)
        return v
    return ''.join(map(lookup, s))

class FilterModule(object):

    def filters(self):
        return {
                'rot13': rot13
               }
extending/plugin/filter/plugins/rot13.py

Usage:

{{ variable | rot13 }}

OTHER PLUGINS

ACTION PLUGINS

  • Controller actions for the module
  • To develop one, implement a class ActionModule with a method run()
    •  self.runner._execute_module() to call the real module

VARS PLUGINS

  • Playbook constructs like host_vars and group_vars work via vars plugins
  • It is quite an obscure and undocumented feature, I have only found this code sample in the Ansible repository

VARS PLUGIN TEMPLATE CODE

class VarsModule(object):

    """
    Loads variables for groups and/or hosts
    """

    def __init__(self, inventory):

        """ constructor """

        self.inventory = inventory
        self.inventory_basedir = inventory.basedir()


    def run(self, host, vault_password=None):
        """ For backwards compatibility, when only vars per host were retrieved
            This method should return both host specific vars as well as vars
            calculated from groups it is a member of """
        return {}


    def get_host_vars(self, host, vault_password=None):
        """ Get host specific variables. """
        return {}


    def get_group_vars(self, group, vault_password=None):
        """ Get group specific variables. """
        return {}

CACHE PLUGINS

Fact caching:

  • hostvars of hosts not contacted in the playbook execution

  • v1.8+

  • Tweak the ansible.cfg as follows: 

[defaults]
gathering = smart
fact_caching = redis # Name of the plugin
fact_caching_timeout = 86400

CACHE PLUGINS

  • Currently supported cache plugins:
    • Memory
    • JSONfile
    • Memcached
    • Redis

CACHE PLUGIN TEMPLATE CODE


import exceptions

class BaseCacheModule(object):

    def get(self, key):
        raise exceptions.NotImplementedError

    def set(self, key, value):
        raise exceptions.NotImplementedError

    def keys(self):
        raise exceptions.NotImplementedError

    def contains(self, key):
        raise exceptions.NotImplementedError

    def delete(self, key):
        raise exceptions.NotImplementedError

    def flush(self):
        raise exceptions.NotImplementedError

    def copy(self):
        raise exceptions.NotImplementedError

REFERENCES

Books:

Articles:

Happy hacking!

Embedding and extending Ansible with Python

By Alejandro Guirao Rodríguez

Embedding and extending Ansible with Python

  • 12,952