EMBEDDING and EXTENDING Ansible with Python
INTRODUCTION TO ANSIBLE
Source: Ansible Blog
WHAT ANSIBLE IS
- It is a Configuration Management tool
- Strong points
- It works over SSH and is agentless
- Python 2.X on remote hosts
- Very readable (YAML)
- It works over SSH and is agentless
Architecture overview
Source: sysadmincasts
Host file (inventory)
[web]
10.0.15.21
10.0.15.22
[db]
10.0.15.23
[db:vars]
db_user=guest
[infrastructure:children]
web
db
Ad Hoc Commands
- Throw-away, one-time actions
ansible group -m module -a options
- Execution of an Ansible module in each host of a group
Hello World! : Ping
(ansible)alex ~ $ ansible all -m ping
10.0.15.22 | success >> {
"changed": false,
"ping": "pong"
}
10.0.15.21 | success >> {
"changed": false,
"ping": "pong"
}
10.0.15.23 | success >> {
"changed": false,
"ping": "pong"
}
ansible all -m ping
PLAYBOOKS
- The playbooks are YAML files that specify a list of plays
- Each play is a series of related tasks applied to a group of hosts
- name: Ensure that nginx is started
service: name=nginx state=started
- Each task is the execution of a module with some parameters and a readable name:
A playbook to deploy nginx
---
- hosts: web
sudo: yes
vars:
- external_port: 80
- internal_port: 8000
tasks:
- name: Add the apt repository for nginx
apt_repository: repo="ppa:nginx/stable" update_cache=yes
- name: Install nginx
apt: name=nginx state=present
- name: Ensure that nginx is started
service: name=nginx state=started
- name: Remove default site
file: path=/etc/nginx/sites-enabled/default state=absent
notify:
- Restart nginx
- name: Configure a site
template: src=templates/site.j2 dest=/etc/nginx/sites-available/site
- name: Enable a site
file: src=/etc/nginx/sites-available/site dest=/etc/nginx/sites-enabled/site state=link
notify:
- Restart nginx
handlers:
- name: Restart nginx
service: name=nginx state=restarted
intro/deploy-nginx.yml
Let's run the playbook
$ ansible-playbook deploy-nginx.yml
PLAY [web] ********************************************************************
GATHERING FACTS ***************************************************************
ok: [10.0.15.21]
ok: [10.0.15.22]
TASK: [Add the apt repository for nginx] **************************************
changed: [10.0.15.21]
changed: [10.0.15.22]
TASK: [Install nginx] *********************************************************
changed: [10.0.15.22]
changed: [10.0.15.21]
TASK: [Ensure that nginx is started] ******************************************
ok: [10.0.15.21]
ok: [10.0.15.22]
TASK: [Remove default site] ***************************************************
changed: [10.0.15.21]
changed: [10.0.15.22]
TASK: [Configure a site] ******************************************************
changed: [10.0.15.21]
changed: [10.0.15.22]
TASK: [Enable a site] *********************************************************
changed: [10.0.15.21]
changed: [10.0.15.22]
TASK: [Remove default site] ***************************************************
ok: [10.0.15.21]
ok: [10.0.15.22]
NOTIFIED: [Restart nginx] *****************************************************
changed: [10.0.15.21]
changed: [10.0.15.22]
PLAY RECAP ********************************************************************
10.0.15.21 : ok=9 changed=6 unreachable=0 failed=0
10.0.15.22 : ok=9 changed=6 unreachable=0 failed=0
Encapsulation: roles
Roles pack:
- Variables
- Tasks
- Handlers
- Files and templates
- Dependencies to other roles
Embedding Ansible
EMBEDDING?
- Embedding: calling Ansible modules and and executing playbooks from your Python code
- This is possible because Ansible has a Python API
- Although not very well documented
- ... but it is quite easy to use
BEFORE WE START...
- The sample code is valid for Ansible v 1.9.X.
- ¿v2?
- Python 2.X only :-(
- The source code of the examples are available at
RUNNING AN ANSIBLE TASK
BASICS
- Import these classes:
from ansible.runner import Runner
from ansible.inventory import Inventory
- Build your inventory:
inventory = Inventory(["localhost"])
- Call the Runner.run():
runner = Runner(module_name='setup', module_args='',
pattern='localhost', inventory=inventory,
transport="local")
result = runner.run()
About facts
ansible all -m setup
- Hostnames
- IP addresses
- Hardware and devices attached
- Installed software versions...
You'll get:
EXAMPLE: FLASK FACTER
from ansible.runner import Runner
from ansible.inventory import Inventory
from flask import Flask
from flask_restful import Resource, Api
class AnsibleFactResource(Resource):
"""
A read-only local ansible fact
"""
def __init__(self):
self.facts = None
Resource.__init__(self)
def get_local_facts(self):
"""
Loads the Ansible local facts in self.facts
Calls the Ansible Python API v1 'setup' module
"""
inventory = Inventory(["localhost"])
runner = Runner(module_name='setup', module_args='',
pattern='localhost', inventory=inventory,
transport="local")
result = runner.run()
self.facts = result["contacted"]["localhost"]["ansible_facts"]
def get(self, fact_name):
"""
Returns a top-level fact (not nested)
"""
self.get_local_facts()
try:
return self.facts[fact_name]
except KeyError:
return "Not a valid fact: %s" % fact_name
app = Flask(__name__)
api = Api(app)
api.add_resource(AnsibleFactResource, '/facts/<string:fact_name>')
if __name__ == '__main__':
app.run(debug=True, port=8000)
embedding/task/flask_facter.py
RUNNING AN ANSIBLE PLAYBOOK
BASICS
- Import these classes :
from ansible.playbook import PlayBook
from ansible.inventory import Host, Inventory
from ansible import callbacks
from ansible import utils
- Build your inventory
# Create the inventory
controller = Host(name = "localhost")
controller.set_variable('users', user_list)
controller.set_variable('apt_packages', package_list)
local_inventory = Inventory([])
local_inventory.get_group('all').add_host(controller)
BASICS
- Set the callbacks setup
# Boilerplate for callbacks setup
utils.VERBOSITY = 0
# Output callbacks setup
output_callbacks = callbacks.PlaybookCallbacks(verbose=utils.VERBOSITY)
# API callbacks setup
stats = callbacks.AggregateStats()
api_callbacks = callbacks.PlaybookRunnerCallbacks(stats, verbose=utils.VERBOSITY)
- Call the PlayBook.run()
provision_playbook = PlayBook(playbook = "installer.yml",
stats = stats,
callbacks = output_callbacks,
runner_callbacks = api_callbacks,
inventory = local_inventory,
transport = "local",
become_pass = sudo_password
)
playbook_result = provision_playbook.run()
EXAMPLE: INSTALLER
import sys
from getpass import getpass
from ansible.playbook import PlayBook
from ansible.inventory import Host, Inventory
from ansible import callbacks
from ansible import utils
def run_installer(user_list, package_list, sudo_password):
"""
Runs the playbook `installer.yml` with the supplied parameters
"""
# Create the inventory
controller = Host(name = "localhost")
controller.set_variable('users', user_list)
controller.set_variable('apt_packages', package_list)
local_inventory = Inventory([])
local_inventory.get_group('all').add_host(controller)
# Boilerplate for callbacks setup
utils.VERBOSITY = 0
# Output callbacks setup
output_callbacks = callbacks.PlaybookCallbacks(verbose=utils.VERBOSITY)
# API callbacks setup
stats = callbacks.AggregateStats()
api_callbacks = callbacks.PlaybookRunnerCallbacks(stats, verbose=utils.VERBOSITY)
provision_playbook = PlayBook(playbook = "installer.yml",
stats = stats,
callbacks = output_callbacks,
runner_callbacks = api_callbacks,
inventory = local_inventory,
transport = "local",
become_pass = sudo_password
)
playbook_result = provision_playbook.run()
return playbook_result
def get_selection_list(initial_prompt, input_prompt, continue_prompt):
"""
Return a selection list chosen according to a flow described by:
- initial_prompt: To enter the selection menu
- input_prompt: To enter an item
- continue_prompt: To continue entering another item
"""
results = []
enter_selection = raw_input(initial_prompt)
enter_selection = True if enter_selection in ["y", "Y", "yes"] else False
if enter_selection:
while True:
current_result = raw_input(input_prompt)
if not current_result:
break
results.append(current_result)
continue_selection = raw_input(continue_prompt)
continue_selection = True if continue_selection in ["y","Y","yes"] else False
if not continue_selection:
break
return results
if __name__ == '__main__':
packages = []
print("")
print("Automated installation script")
print("=============================")
print("")
sudo_password = getpass("Enter sudo password:")
users = get_selection_list(
"Create users? (y/N)",
"Enter username:",
"Add more users? (y/N)"
)
print("")
packages = get_selection_list(
"Install packages? (y/N)",
"Enter package name:",
"Add more packages? (y/N)"
)
run_installer(user_list=users, package_list=packages, sudo_password=sudo_password)
embedding/playbook/installer.py
EXTENDING ANSIBLE
Source: Ansible Blog
EXTENDING?
- Extending: adding more functionality to Ansible or customising its behaviour with Python code
- Modules
- Plugins
- Dynamic inventory scripts
CREATING AN ANSIBLE MODULE
AN ANSIBLE MODULE ...
- ... is an executable file
- In ./library or ANSIBLE_LIBRARY_PATH
- ... has a JSON interface, programming language agnostic
- Although it's easier with Python
#!/usr/bin/python
# -*- coding: utf-8 -*-
DOCUMENTATION = '''
'''
EXAMPLES = '''
'''
# Your custom Python code / external libraries
from mypackage import mypythonicfunction
def mypythonicfunction(...):
"""
Here you do the hard work. Use external libraries or your own code
And return a meaningful value and message to main()
"""
return status, msg
def main():
module = AnsibleModule(
argument_spec=dict(
# Here you parse the arguments
)
# and set the options and behaviour of Ansible
)
# Here you assign your arguments to Python variables
variable_a = module.params['var_a']
# and call your mypythonicfunction() with the variables
status, msg = mypythonicfunction(variable_a, ...)
# Parse the results and return valid JSONs and meaningful messages
if status == "ok":
module.exit_json(changed=changed, msg=msg)
else:
module.fail_json(msg=msg)
# Boilerplate but IMPORTANT code
from ansible.module_utils.basic import *
main()
STRUCTURE OF THE MODULE
DOCUMENTATION AND EXAMPLES
- Used by make webdocs to generate HTML
Keypoints:
- Keep the options part in sync with the argument_spec
- Specify the requirements
- Use the notes section
- Test your examples :-)
the pythonic part
- However, there are helper funcions
- run_command
- Try to use pure Python
- Business logic
- Tips:
- Return a value and a msg
- Don't print to stdout or stderr!
THE MAIN() FUNCTION
- AnsibleModule instance
- Don't forget:
- Mutual exclusion
- Support for check mode
- The argument_spec dict defines module arguments
- Required vs optional
- Default values
- Possible choices
- Aliases
THE MAIN() FUNCTION
- Life is easy: the params dict
- Call the pythonic part and parse the results
- Super-handy helper functions:
exit_json(changed, ...) fail_json(msg, ...)
BOILERPLATE CODE
- Put these two lines at the end of the file:
- Resist the temptation to make explicit imports!
from ansible.module_utils.basic import *
main()
MODULE CREATION TIPS
- Make a module you would love to use:
- Idempotent
- Check mode
- Test your module with the test-module script:
ansible/hacking/test-module
-m /path/to/module
-a module_args
- Follow the module creation checklist
MODULE EXAMPLE: TAIGA_ISSUE
- It uses python-taiga to deal with the Taiga REST API
extending/module/library/taiga_issue.py
CREATING A DYNAMIC INVENTORY SCRIPT
DYNAMIC INVENTORY SCRIPTS
- An executable file that supports this command line switches:
- --host=<hostname> for showing host details
- --list for listing groups
- The output is JSON
EXAMPLE: SHELVE_INVENTORY
#!/usr/bin/env python
import argparse
import json
import sys
import shelve
import ConfigParser
def parse_args():
parser = argparse.ArgumentParser(description="Shelve inventory script")
group = parser.add_mutually_exclusive_group(required=True)
group.add_argument('--list', action='store_true')
group.add_argument('--host')
return parser.parse_args()
def get_shelve_groups(shelvefile):
d = shelve.open(shelvefile)
g = d["groups"]
d.close()
return g
def get_shelve_host_details(shelvefile, host):
d = shelve.open(shelvefile)
h = d["hostvars"][host]
d.close()
return h
def main():
config = ConfigParser.RawConfigParser()
config.read('shelve_inventory.ini')
shelvefile = config.get('defaults', 'shelvefile')
args = parse_args()
if args.list:
groups = get_shelve_groups(shelvefile)
json.dump(groups, sys.stdout)
else:
details = get_shelve_host_details(shelvefile, args.host)
json.dump(details, sys.stdout)
if __name__ == '__main__':
main()
extending/dynamic_inventory/shelve_inventory.py
CREATING PLUGINS
PLUGINS
- They run in the Controller node!
- Different kinds of plugins:
- Callback plugins
- Connection plugins
- Lookup plugins
- Filter plugins
- Action plugins
- Vars plugins
- Cache plugins
How To add PLUGINS
- Or you can tweak ansible.cfg:
[defaults]
callback_plugins = ./plugins
- Default location: /usr/share/ansible/plugins/
action_plugins/
lookup_plugins/
callback_plugins/
connection_plugins/
filter_plugins/
vars_plugins/
CREATING A CALLBAck PLUGIN
CALLBACK PLUGINS
- React to Playbook and Runner events
- Define the CallbackModule class and override the methods you want to
- You can find an example with an extensive list of available methods
EXAMPLE: NOTIFY_SEND
from subprocess import call
def notify_send(title, message, icon):
call(["notify-send", title, message,"-i", icon])
class CallbackModule(object):
def runner_on_failed(self, host, res, ignore_errors=False):
template = "Host: {}\nModule: {}\nMessage: {}"
notification = template.format(
host,
res.get('invocation').get('module_name'),
res.get('msg')
)
notify_send("ANSIBLE FAILURE", notification, "dialog-warning")
extending/plugin/callbacks/plugins/notify_send.py
CREATING A CONNECTION PLUGIN
CONNECTION PLUGINS
- A connection plugin allows the controller to connect to the hosts using different protocols:
- SSH
- paramiko_ssh
- local
- winrm (for Win$ hosts)
- chroot
- jails (for FreeBSD)
- zones (Solaris)
- libvirt_lxc
- ...
- You can find examples the list of official connection plugins here
CONNECTION PLUGIN BASICS
- Define a Connection class and override the following methods:
class Connection(object):
def __init__(self, runner, host, port, user,
password, private_key_file, *args, **kwargs):
[...]
def connect(self):
[...]
def exec_command(self, cmd, tmp_path, become_user=None,
sudoable=False, executable='/bin/sh', in_data=None):
[...]
def put_file(self, in_path, out_path):
[...]
def fetch_file(self, in_path, out_path):
[...]
def close(self):
[...]
CREATING A LOOKUP PLUGIN
LOOKUP PLUGINS
- Data access
- Filesystem
- Databases
- External services
- Syntax:
contents: "{{ lookup('file', '/etc/foo.txt') }}"
- Base of the with_ expression
LOOKUP PLUGIN BASICS
from ansible import utils, errors
class LookupModule(object):
def __init__(self, basedir=None, **kwargs):
self.basedir = basedir
def run(self, terms, inject=None, **kwargs):
# terms holds the parameters of the lookup call
- Implement a run() method of the LookupModule class that returns a list of results
- Each one corresponding to each element of the terms list
EXAMPLE: SHELVEFILE
import shelve
import os
from ansible import utils, errors
class LookupModule(object):
def __init__(self, basedir=None, **kwargs):
self.basedir = basedir
def read_shelve(self, shelve_filename, key):
"""
Read the value of "key" from a shelve file
"""
d = shelve.open(shelve_filename)
res = d.get(key, None)
d.close()
return res
def run(self, terms, inject=None, **kwargs):
terms = utils.listify_lookup_plugin_terms(terms, self.basedir, inject)
ret = []
if not isinstance(terms, list):
terms = [ terms ]
for term in terms:
playbook_path = None
relative_path = None
paramvals = {"file": None, "key": None}
params = term.split()
try:
for param in params:
name, value = param.split('=')
assert(name in paramvals)
paramvals[name] = value
except (ValueError, AssertionError), e:
# In case "file" or "key" are not present
raise errors.AnsibleError(e)
file = paramvals['file']
key = paramvals['key']
basedir_path = utils.path_dwim(self.basedir, file)
# Search also in the role/files directory and in the playbook directory
if '_original_file' in inject:
relative_path = utils.path_dwim_relative(inject['_original_file'], 'files', file, self.basedir, check=False)
if 'playbook_dir' in inject:
playbook_path = os.path.join(inject['playbook_dir'], file)
for path in (basedir_path, relative_path, playbook_path):
if path and os.path.exists(path):
res = self.read_shelve(path, key)
if res is None:
raise errors.AnsibleError("Key %s not found in shelve file %s" % (key, file))
# Convert the value read to string
ret.append(str(res))
break
else:
raise errors.AnsibleError("Could not locate shelve file in lookup: %s" % file)
return ret
extending/plugin/lookup/plugins/shelvefile.py
Usage:
{{ lookup('shelvefile', 'file=book.db key=current_book_author') }}
CREATING A FILTER PLUGIN
FILTER PLUGINS
- Jinja2 filters
- Ansible ships with some useful ones
{{ some_variable | default(5) }}
{{ some_list | min }}
{{ some_register_variable | changed }}
- Syntax: variable | filter
FILTER PLUGIN BASICS
class FilterModule(object):
def filters(self):
return {
'filter1': filter1functionname,
'filter2': filter2functionname,
[...]
}
- Create a FilterModule class with a filters method that returns a dictionary mapping the filters to functions
EXAMPLE: ROT13
def rot13(s):
def lookup(v):
o, c = ord(v), v.lower()
if 'a' <= c <= 'm':
return chr(o + 13)
if 'n' <= c <= 'z':
return chr(o - 13)
return v
return ''.join(map(lookup, s))
class FilterModule(object):
def filters(self):
return {
'rot13': rot13
}
extending/plugin/filter/plugins/rot13.py
Usage:
{{ variable | rot13 }}
OTHER PLUGINS
ACTION PLUGINS
- Controller actions for the module
- To develop one, implement a class ActionModule with a method run()
- self.runner._execute_module() to call the real module
VARS PLUGINS
- Playbook constructs like host_vars and group_vars work via vars plugins
- It is quite an obscure and undocumented feature, I have only found this code sample in the Ansible repository
VARS PLUGIN TEMPLATE CODE
class VarsModule(object):
"""
Loads variables for groups and/or hosts
"""
def __init__(self, inventory):
""" constructor """
self.inventory = inventory
self.inventory_basedir = inventory.basedir()
def run(self, host, vault_password=None):
""" For backwards compatibility, when only vars per host were retrieved
This method should return both host specific vars as well as vars
calculated from groups it is a member of """
return {}
def get_host_vars(self, host, vault_password=None):
""" Get host specific variables. """
return {}
def get_group_vars(self, group, vault_password=None):
""" Get group specific variables. """
return {}
CACHE PLUGINS
Fact caching:
-
hostvars of hosts not contacted in the playbook execution
-
v1.8+
-
Tweak the ansible.cfg as follows:
[defaults]
gathering = smart
fact_caching = redis # Name of the plugin
fact_caching_timeout = 86400
CACHE PLUGINS
- Currently supported cache plugins:
- Memory
- JSONfile
- Memcached
- Redis
CACHE PLUGIN TEMPLATE CODE
import exceptions
class BaseCacheModule(object):
def get(self, key):
raise exceptions.NotImplementedError
def set(self, key, value):
raise exceptions.NotImplementedError
def keys(self):
raise exceptions.NotImplementedError
def contains(self, key):
raise exceptions.NotImplementedError
def delete(self, key):
raise exceptions.NotImplementedError
def flush(self):
raise exceptions.NotImplementedError
def copy(self):
raise exceptions.NotImplementedError
REFERENCES
- Ansible Up & Running by Lorin Hochstein
-
Learning Ansible by M.Mohan and R.Raithatha
Books:
Articles:
- Ansible for DevOps by Jeff Geerling
- Using Ansible like library programming in Python by Oriol Rius
- Running Ansible Programatically by Servers for Hackers
Happy hacking!
Embedding and extending Ansible with Python
By Alejandro Guirao Rodríguez
Embedding and extending Ansible with Python
- 12,952