Simple Validation, Testing and Deployment

Toolbox for Python Web Apps

by Kevin Stone
CTO/Founder Subblime

GH: kevinastone | TW: @kevinastone | LI: kevinastone


Simple validation, testing and deployment practices for small teams

(You could add all this in a weekend)

Subblime's Tech Stack

Web Stack

  • Python Django
  • MySQL + Redis
  • AngularJS + Coffeescript
  • Haml + LessCSS + Bootstrap

Support Services

  • GitHub
  • Jenkins (Continuous Integration)
  • Sentry (Exception Tracking)
  • SendGrid (SMTP)
  • MixPanel + Google Analytics (User Tracking)

Software Quality Assurance

Tiered Approach

Validation (as we code)

  • Coding Standards (PEP8)
  • Static Analysis (Pyflakes)

Testing (before we deploy)

  • Unit Tests
  • Functional Tests (APIs and External Services)
  • Acceptance/E2E Tests (Browser based - Lettuce)
  • Continuous Integration (Jenkins)

Monitoring (after its live)

  • Exception Reporting (Sentry)
  • Service Health (Cacti or Nagios)

Validation and Testing as of 6/13


  • 153 Source Files (99% coverage - means basically nothing)
  • 4569 Lines (68% coverage)


  • 277 Unit Tests (52s runtime)
  • 15 Functional Tests (25s runtime)
  • 8 Features, 30 Scenarios (312s runtime)

Code Validation

  • PEP8-derived Coding Standard
  • PyFlakes for unused or missing symbols and other coding errors 
  • Leverage IDE/Editor plugins to validate during development
  • Validate standards during continuous integration (fail builds on violation)

Our .pep8 Configuration

# List of PEP8 Errors and Warnings to Ignore
# E501  line too long (82 > 79 characters)
# W191  indentation contains tabs
# W293	blank line contains whitespace
# E302	expected 2 blank lines, found 0

# Most of the Indentation continuation rules are ignored (except mixed spaces and tabs)
# E121	continuation line indentation is not a multiple of four
# E122	continuation line missing indentation or outdented
# E123	closing bracket does not match indentation of opening bracket’s line
# E124	closing bracket does not match visual indentation
# E125	continuation line does not distinguish itself from next logical line
# E126	continuation line over-indented for hanging indent
# E127	continuation line over-indented for visual indent
# E128	continuation line under-indented for visual indent

# Whitespace
# E261	at least two spaces before inline comment

ignore = E501,W191,W293,E302,E12,E261
exclude = migrations


# F403	unable to detect undefined names (from whatever import *)

ignore = E501,W191,W293,E302,E12,E261,F403
exclude = migrations,*

IDE/Text Editor Integration

SublimeLinter for Sublime Text


Continuous Integration

Think of CI as your tripwire for potential problems 

Potential Issues

  • Regression Testing (did that change break anything?)
  • Smoke Testing dependency installation and external APIs
  • Dev Environment Leakage (it worked on my machine)

Other Opportunities

  • Process Reinforcement (mandatory code coverage, etc)
  • Annotating Builds (tagging for release)
  • Packaging for Deployment

Continuous Integration Process

  1. Code update pushed to GitHub
  2. GitHub Commit-Hook triggers build on Jenkins server
  3. Jenkins spawns a Worker for the build job
  4. Build job checks out the committed version and runs the tests
  5. Jenkins captures the results of the tests
  6. Stakeholders are notified of build results

Continuous Integration with Jenkins

Jenkins hosted on EC2

  1. Install Jenkins on t1.micro as a Master (
  2. Create AMI of Worker instance for running tests on demand (using large instances, e.g. c1.medium)
  3. Setup Jenkins Jobs for each test variant
    1. Unit Tests (with code coverage and pep8 violations)
    2. Functional Tests
    3. Lettuce Tests
  4. Trigger Jenkins Builds with GitHub Post-Commit Hook

    PaaS Alternative:
    Shining Panda (

    Helpful Jenkins Plugins

    • GitHub OAuth for Credentials
    • Jenkins Build Timeout for unreliable builds (watchdog timer)
    • Naginator for inconsistent/unreliable tests (like external API outages)
    • Jenkins Cobertura for code coverage results
    • Jenkins Violations for PEP8 + Pyflakes results
    • Amazon EC2 for Worker Management (if AWS based)
    • Throttle Concurrent Builds to cap workers

    Jenkins Configuration

    *Yes, you're stuck entering your GH password if you want auto-managed hook URLs

    Jenkins EC2 Configuration

    GitHub OAuth Login

    Job Configuration

    Unit Tests

    Execute Shell

    cd ~/$WORKSPACE
    virtualenv ~/envs/${JOB_NAME}
    source ~/envs/${JOB_NAME}/bin/activate
    pip install -r requirements.txt
    pip install -r testing_requirements.txt
    python develop
    # run tests using django-nose
    ./ test
    pep8 [your_package] >

    Publish Cobertura Coverage Report (setup.cfg)


    Test Results and Violations

    Functional Tests

    cd ~/$WORKSPACE
    virtualenv ~/envs/${JOB_NAME}
    source ~/envs/${JOB_NAME}/bin/activate
    pip install -r requirements.txt
    pip install -r testing_requirements.txt
    python develop
    nosetests functional/*.py

    Functional Tests are Unreliable (by definition)
    Set retry build after failure to isolate outages from bugs

    Lettuce Tests

    cd ~/$WORKSPACE
    virtualenv ~/envs/${JOB_NAME}
    source ~/envs/${JOB_NAME}/bin/activate
    pip install -r requirements.txt
    pip install -r testing_requirements.txt
    pip install -r lettuce_requirements.txt
    python develop
    ./ harvest  --verbosity=3 --with-xunit

    Exception Tracking with Sentry

    Sentry provides remote exception logging for identifying and debugging issues in Production

    Sentry Installation


    > virtualenv $ENVS/sentry && source $ENVS/sentry/bin/activate
    > pip install -U sentry mysql-python # (or psychopg, etc...)


    DATABASES = {...}
    # Set this to false to require authentication
    SENTRY_WEB_PORT = 9000
        'workers': 3,  # the number of gunicorn workers
        # 'worker_class': 'gevent',
    # Mail server configuration
    EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'
    # TODO: Configure your SMTP Credentials
    EMAIL_SUBJECT_PREFIX = '[Sentry] '

    Install Sentry (continued)

    Manage Sentry with SupervisorD

    command=$ENVS/sentry/bin/sentry --config=/etc/ start http

    Proxy through NGINX

    server {
        listen 80;
        listen 443 ssl;
    	location / {
    		proxy_pass http://localhost:9000;
    		proxy_redirect off;
    		proxy_set_header	Host	$host;
    		proxy_set_header	X-Real-IP	$remote_addr;
    		proxy_set_header	X-Forwarded-For	$proxy_add_x_forwarded_for;

    Sentry Installation (continued)

    *Or Use Hosted Solution

    Configure (Django) App for Sentry

    Standard Settings

    • SENTRY_DSN = 'http://<token>:<key><id>'
    • MIDDLEWARE_CLASSES +=('raven.contrib.django.middleware.Sentry404CatchMiddleware',)
    • LOGGING = {..., 'handlers': { 'sentry: {'level': 'DEBUG', 'class': 'raven.contrib.django.handlers.SentryHandler'}}}

    Extra Settings

    • PUBLIC_SENTRY_DSN = 'http://<token><id>' (for raven-js)
    • LOGGING = {..., 'loggers': { 'celery': {'level': 'WARNING', 'handlers': ['sentry'], 'propagate': True'}}}

    Deployment Process

    • Spawn Servers
      • AWS Console or Boto
      • (Other steps like DNS config)
    • Configure Services and Dependencies
      • Salt-Stack
    • Deploy Application(s)
      • Fabric

    Production Environment

    Spawn Instances

    [Manually Create and Manage via AWS Console]

    Configure Services

    1. Install Packages and Dependencies
    2. Configure Firewall, Network, Timezone, etc
    3. Setup/Configure Services
      1. NGINX
      2. SupervisorD
      3. Redis
      4. MySQL
      5. Cron scripts

    Install Packages and Dependencies

    (The Manual Way)

    Core Services

    sudo DEBIAN_FRONTEND=noninteractive apt-get -y install percona-server-server nginx openssh-server supervisor redis-server ntp

    Base Libraries

    sudo apt-get -y install build-essential git linux-headers-generic htop tmux pv

    Python Dependencies

    sudo apt-get -y install python-virtualenv
    sudo apt-get -y install libmysqlclient-dev libxml2-dev libxslt-dev python-dev libjpeg-dev zlib1g-dev

    NodeJS (for static asset management)

    sudo apt-get -y install nodejs

    Install Packages via Salt Stack


    base:    '*':
            - core
            - ssh
            - ssh.deploy
            - match: grain
            - ssl.certs
            - nginx
            - app.web
            - match: grain
            - supervisor
            - redis
            - node
            - match: grain
            - mysql.server
            - mysql.backup
            - mysql.utils
            - app.db

    Fabric Based App Deployment

    Deploy Task Sequence

    1. enter_maintenance (nginx 503 message)
    2. code_checkout
    3. clean (remove .pyc's)
    4. install_dependencies
    5. migrate_db
    6. static_assets
    7. flush_cache
    8. restart_services
    9. exit_maintenance

    Document your Deployment

    All the way from bare metal*

    At a minimum:
    • List of commands to install and configure services
    • Version control any config or settings files
      • Even better, link config files to version control checkout
      • sudo ln -sf /srv/ /etc/my.cnf

    Your 4am or on-vacation self** will thank you

    *Like smoke alarms, test routinely
    **or accessible substitute

    Deployment Tips

    • NGINX + µWSGi Config
    • Maintenance Modes
    • Fabric Environmental Config Tasks

    NGINX + µWSGi

    location @uwsgi {
        include		uwsgi_params;
        uwsgi_param	UWSGI_SCHEME	$scheme;    # scheme for redirects under SSL

    command=$VIRTUAL_ENV/bin/uwsgi --socket --processes 5 --master --home $VIRTUAL_ENV --vacuum --harakiri 200  --wsgi "<your_project>.wsgi" --buffer-size 16384

    Manual Maintenance Window


    geo $maintenance {
        # Change default to 1 to enable maintenance (and 0 to disable)
    	default 0;
  0; # Local Allowed 0; # Private Addressing Allowed; 0; # Amazon Local Allowed
  0; # Your Office IP
    location / {
    	if ($maintenance) {
    		return 503;


    1. Edit /etc/nginx/sites-available/<your_site>.conf and change default 0 to default 1
    2. > sudo service nginx reload

    Deployment Based Maintenance Window


    location / {
        try_files / @uwsgi


    def enter_maintenance():
        with cd(CODE_DIR):
            run('ln -sf maintenance.html')
    def exit_maintenance():
        with cd(CODE_DIR):
    		with settings(warn_only=True):

    Fabric Environmental Config Tasks

    def production(branch='master'):
        env.hosts = ['']
        env.branch = branch
    def sandbox(branch='master'):
    	env.hosts = ['']
    	env.branch = branch
    def dev(branch='develop'):
    	env.hosts = ['']
    	env.branch = branch
    def beta(branch='develop'):
    	env.hosts = ['']
    	env.branch = branch
    	if not confirm("Do NOT run this with any migrations pending, continue?", default=False):


    > fab production deploy
    > fab dev:branch=new-feature-branch deploy
    > fab beta:new-admin-feature deploy


    The End

    About the Author:

    Kevin Stone is the CTO and Founder of Subblime

    Interested in working on these challenges?  Subblime is hiring

    Python Integration and Deployment

    By Kevin Stone

    Python Integration and Deployment

    Validation, testing and deployment practices for Python.

    • 18,793