Simple Validation, Testing and Deployment


Toolbox for Python Web Apps



by Kevin Stone
CTO/Founder Subblime

GH: kevinastone | TW: @kevinastone | LI: kevinastone

Objective




Simple validation, testing and deployment practices for small teams


(You could add all this in a weekend)

Subblime's Tech Stack


Web Stack

  • Python Django
  • MySQL + Redis
  • AngularJS + Coffeescript
  • Haml + LessCSS + Bootstrap
  • NGINX + AWS


Support Services

  • GitHub
  • Jenkins (Continuous Integration)
  • Sentry (Exception Tracking)
  • SendGrid (SMTP)
  • MixPanel + Google Analytics (User Tracking)

Software Quality Assurance

Tiered Approach

Validation (as we code)

  • Coding Standards (PEP8)
  • Static Analysis (Pyflakes)

Testing (before we deploy)

  • Unit Tests
  • Functional Tests (APIs and External Services)
  • Acceptance/E2E Tests (Browser based - Lettuce)
  • Continuous Integration (Jenkins)

Monitoring (after its live)

  • Exception Reporting (Sentry)
  • Service Health (Cacti or Nagios)

Validation and Testing as of 6/13


Validation

  • 153 Source Files (99% coverage - means basically nothing)
  • 4569 Lines (68% coverage)

Testing

  • 277 Unit Tests (52s runtime)
  • 15 Functional Tests (25s runtime)
  • 8 Features, 30 Scenarios (312s runtime)

Code Validation


  • PEP8-derived Coding Standard
  • PyFlakes for unused or missing symbols and other coding errors 
  • Leverage IDE/Editor plugins to validate during development
  • Validate standards during continuous integration (fail builds on violation)

Our .pep8 Configuration

# List of PEP8 Errors and Warnings to Ignore
# E501  line too long (82 > 79 characters)
# W191  indentation contains tabs
# W293	blank line contains whitespace
# E302	expected 2 blank lines, found 0

# Most of the Indentation continuation rules are ignored (except mixed spaces and tabs)
# E121	continuation line indentation is not a multiple of four
# E122	continuation line missing indentation or outdented
# E123	closing bracket does not match indentation of opening bracket’s line
# E124	closing bracket does not match visual indentation
# E125	continuation line does not distinguish itself from next logical line
# E126	continuation line over-indented for hanging indent
# E127	continuation line over-indented for visual indent
# E128	continuation line under-indented for visual indent

# Whitespace
# E261	at least two spaces before inline comment
[pep8]

ignore = E501,W191,W293,E302,E12,E261
exclude = migrations

[flake8]

# F403	unable to detect undefined names (from whatever import *)

ignore = E501,W191,W293,E302,E12,E261,F403
exclude = migrations,*-steps.py

IDE/Text Editor Integration


SublimeLinter for Sublime Text


"settings":
{
    "SublimeLinter":
	{
		"pep8_ignore":
		[
			"E501",
			"W191",
			"W293",
			"E302",
			"E12",
			"E261"
		]
	}
}

Continuous Integration

Think of CI as your tripwire for potential problems 


Potential Issues

  • Regression Testing (did that change break anything?)
  • Smoke Testing dependency installation and external APIs
  • Dev Environment Leakage (it worked on my machine)

Other Opportunities

  • Process Reinforcement (mandatory code coverage, etc)
  • Annotating Builds (tagging for release)
  • Packaging for Deployment

Continuous Integration Process


  1. Code update pushed to GitHub
  2. GitHub Commit-Hook triggers build on Jenkins server
  3. Jenkins spawns a Worker for the build job
  4. Build job checks out the committed version and runs the tests
  5. Jenkins captures the results of the tests
  6. Stakeholders are notified of build results

Continuous Integration with Jenkins


Jenkins hosted on EC2


  1. Install Jenkins on t1.micro as a Master (ci.example.com)
  2. Create AMI of Worker instance for running tests on demand (using large instances, e.g. c1.medium)
  3. Setup Jenkins Jobs for each test variant
    1. Unit Tests (with code coverage and pep8 violations)
    2. Functional Tests
    3. Lettuce Tests
  4. Trigger Jenkins Builds with GitHub Post-Commit Hook

    PaaS Alternative:
    Shining Panda (https://www.shiningpanda-ci.com)

    Helpful Jenkins Plugins


    • GitHub OAuth for Credentials
    • Jenkins Build Timeout for unreliable builds (watchdog timer)
    • Naginator for inconsistent/unreliable tests (like external API outages)
    • Jenkins Cobertura for code coverage results
    • Jenkins Violations for PEP8 + Pyflakes results
    • Amazon EC2 for Worker Management (if AWS based)
    • Throttle Concurrent Builds to cap workers

    Jenkins Configuration




    *Yes, you're stuck entering your GH password if you want auto-managed hook URLs

    Jenkins EC2 Configuration

    GitHub OAuth Login


    Job Configuration


    Unit Tests

    Execute Shell

    #!/bin/bash
    cd ~/$WORKSPACE
    virtualenv ~/envs/${JOB_NAME}
    source ~/envs/${JOB_NAME}/bin/activate
    pip install -r requirements.txt
    pip install -r testing_requirements.txt
    python setup.py develop
    # run tests using django-nose
    ./manage.py test
    pep8 [your_package] > pep8.report
    

    Publish Cobertura Coverage Report (setup.cfg)

    [nosetests]
    with-doctest=1
    with-xcoverage=1
    cover-erase=
    cover-package=[your_package]
    with-xunit=1
    

    Test Results and Violations

    Functional Tests

    
    #!/bin/bash
    cd ~/$WORKSPACE
    virtualenv ~/envs/${JOB_NAME}
    source ~/envs/${JOB_NAME}/bin/activate
    pip install -r requirements.txt
    pip install -r testing_requirements.txt
    python setup.py develop
    nosetests functional/*.py
    

    Functional Tests are Unreliable (by definition)
    Set retry build after failure to isolate outages from bugs

    Lettuce Tests


    
    #!/bin/bash
    cd ~/$WORKSPACE
    virtualenv ~/envs/${JOB_NAME}
    source ~/envs/${JOB_NAME}/bin/activate
    pip install -r requirements.txt
    pip install -r testing_requirements.txt
    pip install -r lettuce_requirements.txt
    python setup.py develop
    ./manage.py harvest  --verbosity=3 --with-xunit
    

    Exception Tracking with Sentry


    Sentry provides remote exception logging for identifying and debugging issues in Production

    Sentry Installation

    Installation

    
    > virtualenv $ENVS/sentry && source $ENVS/sentry/bin/activate
    > pip install -U sentry mysql-python # (or psychopg, etc...)
    

    /etc/sentry.conf.py

    
    DATABASES = {...}
    
    # Set this to false to require authentication
    SENTRY_PUBLIC = True
    
    SENTRY_WEB_HOST = '0.0.0.0'
    SENTRY_WEB_PORT = 9000
    SENTRY_WEB_OPTIONS = {
        'workers': 3,  # the number of gunicorn workers
        # 'worker_class': 'gevent',
    }
    
    # Mail server configuration
    EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'
    
    # TODO: Configure your SMTP Credentials
    
    EMAIL_SUBJECT_PREFIX = '[Sentry] '
    SERVER_EMAIL = 'sentry@example.com'
    

    Install Sentry (continued)


    Manage Sentry with SupervisorD

    [program:sentry]
    directory=/tmp
    command=$ENVS/sentry/bin/sentry --config=/etc/sentry.conf.py start http


    Proxy through NGINX

    server {
        listen 80;
        listen 443 ssl;
    	server_name sentry.example.com;
    	
    	location / {
    		proxy_pass http://localhost:9000;
    		proxy_redirect off;
    
    		proxy_set_header	Host	$host;
    		proxy_set_header	X-Real-IP	$remote_addr;
    		proxy_set_header	X-Forwarded-For	$proxy_add_x_forwarded_for;
    	}
    }
    

    Sentry Installation (continued)




    *Or Use Hosted Solution

    getsentry.com

    Configure (Django) App for Sentry


    Standard Settings

    • SENTRY_DSN = 'http://<token>:<key>@sentry.example.com/<id>'
    • MIDDLEWARE_CLASSES +=('raven.contrib.django.middleware.Sentry404CatchMiddleware',)
    • LOGGING = {..., 'handlers': { 'sentry: {'level': 'DEBUG', 'class': 'raven.contrib.django.handlers.SentryHandler'}}}


    Extra Settings

    • PUBLIC_SENTRY_DSN = 'http://<token>@sentry.example.com/<id>' (for raven-js)
    • LOGGING = {..., 'loggers': { 'celery': {'level': 'WARNING', 'handlers': ['sentry'], 'propagate': True'}}}

    Deployment Process

    • Spawn Servers
      • AWS Console or Boto
      • (Other steps like DNS config)
    • Configure Services and Dependencies
      • Salt-Stack
    • Deploy Application(s)
      • Fabric

    Production Environment


    Spawn Instances




    [Manually Create and Manage via AWS Console]

    Configure Services


    1. Install Packages and Dependencies
    2. Configure Firewall, Network, Timezone, etc
    3. Setup/Configure Services
      1. NGINX
      2. SupervisorD
      3. Redis
      4. MySQL
      5. Cron scripts

    Install Packages and Dependencies

    (The Manual Way)


    Core Services

    sudo DEBIAN_FRONTEND=noninteractive apt-get -y install percona-server-server nginx openssh-server supervisor redis-server ntp

    Base Libraries

    sudo apt-get -y install build-essential git linux-headers-generic htop tmux pv

    Python Dependencies

    sudo apt-get -y install python-virtualenv
    sudo apt-get -y install libmysqlclient-dev libxml2-dev libxslt-dev python-dev libjpeg-dev zlib1g-dev

    NodeJS (for static asset management)

    sudo apt-get -y install nodejs

    Install Packages via Salt Stack

    salt/top.sls

    base:    '*':
            - core
            - ssh
            - ssh.deploy
        'roles:web':
            - match: grain
            - ssl.certs
            - nginx
            - app.web
        'roles:app':
            - match: grain
            - supervisor
            - redis
            - node
            - app.app
        'roles:db':
            - match: grain
            - mysql.server
            - mysql.backup
            - mysql.utils
            - app.db
    
    

    Fabric Based App Deployment


    Deploy Task Sequence

    1. enter_maintenance (nginx 503 message)
    2. code_checkout
    3. clean (remove .pyc's)
    4. install_dependencies
    5. migrate_db
    6. static_assets
    7. flush_cache
    8. restart_services
    9. exit_maintenance

    Document your Deployment

    All the way from bare metal*


    At a minimum:
    • List of commands to install and configure services
    • Version control any config or settings files
      • Even better, link config files to version control checkout
      • sudo ln -sf /srv/example.com/config/etc/my.cnf /etc/my.cnf



    Your 4am or on-vacation self** will thank you



    *Like smoke alarms, test routinely
    **or accessible substitute

    Deployment Tips


    • NGINX + µWSGi Config
    • Maintenance Modes
    • Fabric Environmental Config Tasks

    NGINX + µWSGi

    /etc/nginx/sites-available/<your_site>.conf
    
    location @uwsgi {
        uwsgi_pass	127.0.0.1:3031;
        include		uwsgi_params;
        uwsgi_param	UWSGI_SCHEME	$scheme;    # scheme for redirects under SSL
    }
    

    /etc/supervisor/conf.d/uwsgi.conf
    
    [program:uwsgi]
    command=$VIRTUAL_ENV/bin/uwsgi --socket 0.0.0.0:3031 --processes 5 --master --home $VIRTUAL_ENV --vacuum --harakiri 200  --wsgi "<your_project>.wsgi" --buffer-size 16384
    
    directory=$PROJECT_HOME
    user=ubuntu
    numprocs=1
    stdout_logfile=/var/log/uwsgi.log
    stderr_logfile=/var/log/uwsgi.log
    autostart=true
    autorestart=true
    startsecs=1
    stopwaitsecs=10
    stopsignal=INT
    

    Manual Maintenance Window

    Configuration

    geo $maintenance {
        # Change default to 1 to enable maintenance (and 0 to disable)
    	default 0;
    	
    	127.0.0.0/8 0; # Local Allowed
    	192.168.0.0/16 0; # Private Addressing Allowed;
    	10.0.0.0/7 0; # Amazon Local Allowed
    	
    	12.34.56.78/32 0; # Your Office IP
    }
    
    location / {
    	if ($maintenance) {
    		return 503;
    	}
    ...
    }
    

    Execution

    1. Edit /etc/nginx/sites-available/<your_site>.conf and change default 0 to default 1
    2. > sudo service nginx reload

    Deployment Based Maintenance Window


    Configuration

    
    location / {
        ...
        try_files /maintenance.active.html @uwsgi
    }
    

    Execution

    
    @task
    def enter_maintenance():
        with cd(CODE_DIR):
            run('ln -sf maintenance.html maintenance.active.html')
    
    
    @task
    def exit_maintenance():
        with cd(CODE_DIR):
    		with settings(warn_only=True):
    			run('unlink maintenance.active.html')
    

    Fabric Environmental Config Tasks

    @task
    def production(branch='master'):
        env.hosts = ['example.com']
        env.branch = branch
    
    @task
    def sandbox(branch='master'):
    	env.hosts = ['sandbox.example.com']
    	env.branch = branch
    
    @task
    def dev(branch='develop'):
    	env.hosts = ['dev.example.com']
    	env.branch = branch
    
    @task
    def beta(branch='develop'):
    	env.hosts = ['beta.example.com']
    	env.branch = branch
    	if not confirm("Do NOT run this with any migrations pending, continue?", default=False):
    		abort("Aborting...")
    

    Execution

    > fab production deploy
    > fab dev:branch=new-feature-branch deploy
    > fab beta:new-admin-feature deploy
    



    Questions?



    The End





    About the Author:

    Kevin Stone is the CTO and Founder of Subblime

    Interested in working on these challenges?  Subblime is hiring

    Python Integration and Deployment

    By Kevin Stone

    Python Integration and Deployment

    Validation, testing and deployment practices for Python.

    • 9,616

    More from Kevin Stone