Deploying
With
Fabric

Handling multiple servers and environments, and handling common deployment tasks.

By Brandon Konkle (@bkonkle)
For the Dallas Django User's Group

Fabric

http://fabfile.org

A tool written in Python to help you run tasks on your systems
Helps you build automated tasks on top of SSH connections
Gives you tools for common tasks and multiple servers and roles

This Talk

Will cover a way to manage multiple environments and roles.

Will introduce a few real-world deployment and convenience tasks.

Managing
Multiple
Environments

The Goal

To select your environment as easily as this:

$ fab dev deploy

And to target specific server roles like this:

$ fab -R app dev mytask

Defining your environments

First, define the details of your environments in the fabfile:

ENVIRONMENTS = {
    'prod': {
        'roledefs': {
            'web': ['webserver1.myapp.com'],
            'app': ['appserver1.myapp.com', 'appserver2.myapp.com'],
            'util': ['utilserver1.myapp.com'],
            'db': ['dbserver1.myapp.com'],
        },
    },
    'dev': {
        'roledefs': {
            'web': ['dev.myapp.com'],
            'app': ['dev.myapp.com'],
            'util': ['dev.myapp.com'],
            'db': ['dev.myapp.com'],
        },
        'proj_rev': 'develop',
    }
}

Set some defaults

To handle things that are the same in most of your environments, define some defaults:

DEFAULTS = {
    'user': 'myappuser',
    'root': '/opt/webapps/myapp',
    'proj_rev': 'master',
}

As you can see, the dev environment above overrides the default "proj_rev" to deploy the "develop" branch instead of "master".

selecting an environment

To put these definitions to work, create a task to select the desired environment:

def environment(name):
    """Environment selector"""
    env.update(dict(DEFAULTS, **ENVIRONMENTS[name]))
    env.name = name

    # Set dependent attributes
    env.proj_root = env.root + '/src/myapp'
    env.pip_file = env.proj_root + '/requirements.pip'

This updates the "env" object with details of the environment, and sets a couple of dynamic attributes.

Minor Details

Fabric won't allow you to use the '-R' option to select a role unless you have roledefs before any tasks are run. To work around that, add some placeholders:

env.roledefs = {'web': [], 'app': [], 'db': [], 'util': []}

Next, default to all roles if no specific roles or hosts were specific on the command line:

if not env.get('roles') and not env.get('hosts'):
    env.roles = env.roledefs.keys()

Aliases

At this point, you can begin selecting your environment like this:

$ fab environment:dev deploy

This is too verbose for me, however. I like to set up quick aliases:

def prod():
    """Shortcut for environment:prod"""
    return environment("prod")

def dev():
    """Shortcut for environment:dev"""
    return environment("dev")

Now you can just use "dev" and "prod" to select your environment.

Specific roles

To target specific roles, use fabric.decorators.roles and decorate your tasks with the roles they should run on.

@roles('app', 'util')
def deploy(rev=None):
    # Your deploy code here

Requiring an Environment

Some tasks absolutely need to have an environment selected before they can run.

from functools import wraps
from fabric.api import require

def requires_env(task_func):
    """Decorator for tasks that require an environment"""
    @wraps(task_func)
    def wrapper(*args, **kwargs):
        require('name', provided_by=['prod', 'dev'])
        return task_func(*args, **kwargs)
    return wrapper

Now you can decorate your tasks like this:

@roles('app', 'util')
@requires_env
def deploy(rev=None):
    # Your deploy code here

Handling
Deploy
Tasks

Getting started

You may want to add a quick helper function to run a command using your virtualenv environment:

@roles('app', 'util')
@requires_env
def ve_run(cmd):
    return run('source {}/bin/activate; {}'.format(env.root, cmd))

Update your code

Start out with a task to update the code on your servers:

@roles('app', 'util')
@requires_env
def checkout(branch=None):
    """Updates project source"""
    if branch is None:
        branch = env.proj_rev
    with cd(env.proj_root):
        # Clean .pyc files
        run('find . -name *.pyc -delete')

        run('git fetch --all')
        run('git checkout origin/{}'.format(branch))

This code also takes the extra step of cleaning *.pyc files, which you may or may not need.

Refresh your requirements

Next, create a task to refresh your pip requirements:

@roles('app', 'util')
@requires_env
def requirements():
    """Update pip requirements"""
    ve_run('pip install --exists-action=w -r {}'.format(env.pip_file))

    # Check for environment-specific requirements
    env_reqs = '{}/{}_requirements.pip'.format(env.proj_root, env.name)
    if exists(env_reqs):
        ve_run('pip install --exists-action=w -r {}'.format(env_reqs))

This also checks for environment specific requirements with names like "dev_requirements.pip".

Run migrations

Next you'll want to run any necessary migrations:

@roles('app', 'util')
@requires_env
def migrate():
    """Run south migrations"""
    ve_run('manage.py migrate --noinput --no-initial-data')

collect Static files

You'll want to collect your staticfiles and take any other necessary actions for your static media, such as compiling Compass:

@roles('app')
@requires_env
def collectstatic():
    with cd(env.proj_root):
        ve_run("compass compile")
        ve_run("manage.py collectstatic --noinput")

Restart Processes

Finally, you'll need to restart your processes:

@roles('app')
@requires_env
def restart():
    """Restart using upstart"""
    sudo('restart myapp')

Deploy!

Put it all together, and what have you got?

@roles('app', 'util')
@requires_env
def deploy(branch=None):
    """
    Run all the tasks for a normal deploy.

    Branches and individual commits can be deployed with::

        $ fab envname deploy:refname
    """
    checkout(branch)
    requirements()
    migrate()
    collectstatic()
    restart()

Pick and choose

If you want to run tasks independently, specify them by name:

$ fab dev requirements restart

Convenience
Tasks

What version?

If you want to see what revision is currently on your servers, you can write a task like this:

@roles('app', 'util')
@requires_env
def version():
    """Show last commit to repo on server"""
    with cd(env.proj_root):
        run('git log -1')

cut your losses

In times of great peril, you may need to restart your database server:

from fabric.contrib.console import confirm

@roles('db')
@requires_env
def restart_db():
    if confirm("Are you sure you want to restart the database process?"):
        sudo('/etc/init.d/postgresql restart')

Check the logs

Maybe you want a quick look at a log file?

@requires_env
@roles('app', 'util')
def django_log(lines=30):
    """Tail the Django Log"""
    log_file = "{env.root}/var/log/django.log".format(env=env)
    run("tail -n{lines} {file}".format(lines=lines, file=log_file))

Just the tip of the iceberg

This is just a taste of what you can do with Fabric.

How are you using it?

Deploying With Fabric