tox

(not poison)

Nir Cohen@Gigaspaces

nir0s@github

@thinkops

Agenda

current problems with testing
tox!
Use cases
Integrations and parallelization
Pitfalls

Current testing problems

Testing on multiple versions of Python.

(py27):~/repex$ python --version
Python 2.7.6
(py27):~/repex$ nosetests
...................................
----------------------------------------------------------------------
Ran 35 tests in 0.162s

OK

:~/repex$ virtualenv -p /usr/bin/python2.6 py26
:~/repex$ source py26/bin/activate
(py26):~/repex$ python --version
Python 2.6.9
(py26):~/repex$ pip install . -r dev-requirements
(py26):~/repex$ nosetests

Managing test env dependencies.

(py27):~/repex$ cat dev-requirements.txt 
coverage==3.7.1
nose
nose-cov
testfixtures

(py27):~/repex$ pip install -r dev-requirements . --upgrade
(py27):~/repex$ nosetests --with-cov --cov-report term-missing --cov repex repex/tests -v

(py27):~/repex$ source py26/bin/activate
(py26):~/repex$ pip install -r dev-requirements . --upgrade
(py26):~/repex$ nosetests --with-cov --cov-report term-missing --cov repex repex/tests -v

Executing pre and post test related activities.

(py27):~/repex$ mypyserv --port 5000 &
(py27):~/repex$ nosetests --with-cov --cov-report term-missing \
    --cov repex repex/tests -v

Running tests in clean environments.

:~/repex$ virtualenv py27
:~/repex$ source py27/bin/activate
(py27):~/repex$ pip install -r dev-requirements.txt .
(py27):~/repex$ nosetests --with-cov --cov-report term-missing --cov repex repex/tests -v

:~/repex$ deactivate
:~/repex$ rm -rf py27
:~/repex$ virtualenv py27
:~/repex$ source py27/bin/activate
(py27):~/repex$ pip install -r dev-requirements.txt .
(py27):~/repex$ nosetests --with-cov --cov-report term-missing --cov repex repex/tests -v

Flake8 should be run on every commit - annoying.

(py27):~/repex$ nosetests --with-cov --cov-report term-missing --cov repex repex/tests -v
(py27):~/repex$ pip install flake8
(py27):~/repex$ flake8 repex

... add some code
(py27):~/repex$ git commit -am "yay!, moar!"
# because who knows.. maybe you added more dependencies
(py27):~/repex$ pip install -r dev-requirements.txt . --upgrade
(py27):~/repex$ nosetests --with-cov --cov-report term-missing --cov repex repex/tests -v
# meh! again!?
(py27):~/repex$ flake8 repex

Complex and repetitive execution of tests.
Most of the above must be solved for every testing platform (py.test, nose, unittest, etc..)
Executing tests in parallel.

Let's (de)tox

tox...

manages virtualenvs
very configurable
supports most common interpreters (from py24 to py34, jython and pypy).
more. let's see, shall we?

install tox

pip install tox

tox.ini

[tox]
# list to environments to test against. Either interpreters or arbitrary names. 
envlist=flake8,py27

# env for all interpreters
[testenv]
# install dependencies
deps =
    -rdev-requirements.txt
# execute commands within the virtualenv
commands=nosetests --with-cov --cov-report term-missing --cov repex repex/tests -v

# runs on default interpreter. name for specific configuration.
[testenv:flake8]
deps =
    flake8
    -rdev-requirements.txt
commands=flake8 repex

toxin'

(py27):~/repex$ ll
total 96
drwxrwxr-x  5 nir0s nir0s  4096 אוק 10 13:44 ./
drwxr-xr-x 55 nir0s nir0s  4096 אוק  9 11:07 ../
-rw-rw-r--  1 nir0s nir0s  4367 ספט 28 19:06 CHANGELOG
drwxrwxr-x  3 nir0s nir0s  4096 אפר  5  2015 check_validity/
-rw-rw-r--  1 nir0s nir0s    52 אפר  5  2015 .coveragerc
-rw-rw-r--  1 nir0s nir0s    52 אפר  1  2015 dev-requirements.txt
drwxrwxr-x  8 nir0s nir0s  4096 אוק 10 13:44 .git/
-rw-rw-r--  1 nir0s nir0s   587 אפר  1  2015 .gitignore
-rw-rw-r--  1 nir0s nir0s 11325 אפר  1  2015 LICENSE
-rw-rw-r--  1 nir0s nir0s  1092 אפר  1  2015 Makefile
-rw-rw-r--  1 nir0s nir0s  9159 ספט 28 19:06 README.md
-rw-rw-r--  1 nir0s nir0s 10099 ספט 28 19:06 README.rst
drwxrwxr-x  3 nir0s nir0s  4096 אוק 10 13:40 repex/
-rw-rw-r--  1 nir0s nir0s  1277 ספט 30 22:49 setup.py
-rw-rw-r--  1 nir0s nir0s   346 ספט 28 19:05 TODO.md
-rw-rw-r--  1 nir0s nir0s   296 אוק 10 13:20 tox.ini
-rw-rw-r--  1 nir0s nir0s   173 אפר  1  2015 .travis.yml

(py27):~/repex$ tox

results

(repex)nir0s@nir0s-x1:~/repos/repex$ tox
GLOB sdist-make: /home/nir0s/repos/repex/setup.py
flake8 inst-nodeps: /home/nir0s/repos/repex/.tox/dist/repex-0.3.0.zip
flake8 installed: click==4.0,cov-core==1.15.0,coverage==3.7.1,extras==0.0.3,flake8==2.4.1,linecache2==1.0.0,mccabe==0.3.1,nose==1.3.7,nose-cov==1.6,pbr==1.8.1,pep8==1.5.7,pyflakes==0.8.1,python-mimeparse==0.1.4,PyYAML==3.10,repex==0.3.0,six==1.10.0,testfixtures==4.3.3,testtools==1.8.0,traceback2==1.4.0,unittest2==1.1.0,wheel==0.24.0
flake8 runtests: PYTHONHASHSEED='3490089416'
flake8 runtests: commands[0] | flake8 repex
py27 inst-nodeps: /home/nir0s/repos/repex/.tox/dist/repex-0.3.0.zip
py27 installed: click==4.0,cov-core==1.15.0,coverage==3.7.1,extras==0.0.3,linecache2==1.0.0,nose==1.3.7,nose-cov==1.6,pbr==1.8.1,python-mimeparse==0.1.4,PyYAML==3.10,repex==0.3.0,six==1.10.0,testfixtures==4.3.3,testtools==1.8.0,traceback2==1.4.0,unittest2==1.1.0,wheel==0.24.0
py27 runtests: PYTHONHASHSEED='3490089416'
py27 runtests: commands[0] | nosetests
...................................
----------------------------------------------------------------------
Ran 35 tests in 0.152s

OK
______________________________________________________________________________________________ summary ______________________________________________________________________________________________
  flake8: commands succeeded
  py27: commands succeeded
  congratulations :)

dev

Elaborate docs at: http://goo.gl/utWvRp
Constantly developed: https://goo.gl/acS10a

Use Cases

you specified multiple Python versions to run on

but only want to run on one of them now.

[tox]
# tox uses sys.version_info to check if an interpreter is available
# the provided envs. 
# If not, it assumes that the env is a name (e.g. flake8, docs)
# these can also be passed via `export TOXENV=env,env,env`
envs=py26,py27

[testenv]
commands=nosetests

[testenv:py26]
commands=nosetests

:~/repex$ tox -e py27

...
py27 runtests: PYTHONHASHSEED='1593161913'
py27 runtests: commands[0] | nosetests
...

Your default Python interpreter is not the one required for

that specific test. You also want to install your module using `develop`

[testenv:py27]
# these will practically run /usr/local/bin/anaconda/python setup.py develop
basepython=/usr/local/bin/anaconda/python
usedevelop=True

you know that your code will not work on Python 2.6

on OS X (for some reason) and so you want to skip it.

[testenv]
commands=nosetests

[testenv:py26]
commands=nosetests
# tox uses sys.platform to check against the provided regex entry.
platform=linux2|win32

you have a Python script you want to run tests for, but it's not a module.

[tox]
# tox will not try to build the module before executing the command.
skipsdist = True

[testenv:py27]
commands =
    nosetests --with-cov --cov get-cloudify.py tests -v

you want to install your dependencies from a set a wheels in a wheelhouse dir.

[testenv]
commands=nosetests

[testenv]
commands=nosetests
# tox will replace the default pip install command with the below.
install_command = pip install --find-links wheelhouse/ --no-index {opts} {packages}

You want to pass down a specific env var from your shell to the test env and create a new env var for your tests.

[testenv]
passenv = 
    # this is passed from the shell running the test
    MY_TEST_RELATED_ENV_VAR
setenv =
    # by default, the port is 1003, but our tests override using this env var 
    PORT = 5000
commands =
    nosetests

you want to deploy your code but don't care about the exit code.

[testenv]
commands =
    nosetests
    flake8
    - deploy_carelessly

you want to create a --verbose vs. --debug mode for executing your tests.

[testenv]
commands =
    nosetests {posargs:-v}

# so with allowing to pass "-s" instead, this acts as verbose vs. debug
:~/repex$ tox -- -s

...
py27 runtests: PYTHONHASHSEED='1593161913'
py27 runtests: commands[0] | nosetests -s
...

you based your API docs on sphinx and want to generate them after every commit to make sure you didn't break anything in the docstrings.

[testenv:docs]
# this will cd to docs from the root dir of where tox.ini is located.
changedir=docs
deps =
    sphinx
    sphinx-rtd-theme
commands=make html

you want to test against multiple versions of the same dependency

[tox]
envlist = py26-legacy,py26-dev,py27-legacy,py27-dev

[testenv:py26-legacy]
deps = 
    dep==0.1
[testenv:py26-dev]
deps = 
    dep
[testenv:py27-legacy]
deps = 
    dep==0.1
[testenv:py27-dev]
deps = 
    dep

[tox]
envlist = {py26,py27}-{dev,legacy}

[testenv]
deps = 
    dev: dep==0.1
    legacy: dep

you want to use the same set of dependencies or changes to a virtualenv in all test environments.

[tox]
envlist = py26,py27

[testenv]
# all environments will use the same virtualenv.
# by default, tox creates the virtualenvs under {toxinidir}/.tox/NAME
# this allows to test on a modified virtualenv.
envdir = {toxinidir}/.env

commands =
    py26: nosetests
    py27: nosetests
    testserver: run_myserver_test_script
    test: more_test_commands

setup.py integration

# setup.py
class Tox(TestCommand):
    ...

setup(
    #...,
    tests_require=['tox'],
    cmdclass = {'test': Tox},
    )



:~/repex$ python setup.py test

detox

local parallelization

install detox

pip install detox

same tox.ini file!

parallel fun!

:~/repex$ pip install detox

# PARALLEL EXECUTION!
:~/repex$ detox

GLOB sdist-make: /home/nir0s/repos/repex/setup.py
py27 inst-nodeps: /home/nir0s/repos/repex/.tox/dist/repex-0.3.0.zip
flake8 inst-nodeps: /home/nir0s/repos/repex/.tox/dist/repex-0.3.0.zip
py27 runtests: PYTHONHASHSEED='2207186455'
py27 runtests: commands[0] | nosetests
flake8 runtests: PYTHONHASHSEED='2207186455'
flake8 runtests: commands[0] | flake8 repex
______________________________________________________________________________________________ summary ______________________________________________________________________________________________
  flake8: commands succeeded
  py27: commands succeeded
  congratulations :)

tox and Travis

remote parallelization

.travis.yml

sudo: false
language: python
python:
  - "2.7"
env:
    - TOX_ENV=flake8
    - TOX_ENV=py27
    - TOX_ENV=py26
install:
    - pip install tox
script:
    - tox -e $TOX_ENV

parallel fun!

Pitalls

getting intoxicated

you've been running tox for a while and decided to upgrade pyyaml from 3.10 to 3.11 and run tox again.

:~/repex$ tox -e py26
...
:~/repex$ .tox/py26/bin/pip freeze
...
PyYAML==3.10
repex==0.3.0
...
# WTF?!

# tox does not run --upgrade on depdendencies. you can use the --recreate
# flag to recreate the virtualenv and install the updated deps.
:~/repex$ tox -e py26 --recreate
...

:~/repex$ .tox/py26/bin/pip freeze
...
PyYAML==3.11
repex==0.3.0
...

# yAY!

detox requires test environments to be independent or clean.

Useful Links

https://tox.readthedocs.org/en/latest/
https://github.com/dstanek/tox-run-command
https://github.com/cloudify-cosmo/repex/blob/master/tox.ini
https://github.com/mitsuhiko/flask/blob/master/tox.ini
https://www.youtube.com/watch?v=Oldkj519o4A
https://www.youtube.com/watch?v=s_YjaODzM1E

tox

(not poison)

Agenda

current problems with testing

tox!

Use cases

Integrations and parallelization

Pitfalls

Current testing problems

Testing on multiple versions of Python.

Managing test env dependencies.

Executing pre and post test related activities.

Running tests in clean environments.

Flake8 should be run on every commit - annoying.

Let's (de)tox

tox...

install tox

tox.ini

toxin'

results

dev

Use Cases

you specified multiple Python versions to run on

but only want to run on one of them now.

Your default Python interpreter is not the one required for

that specific test. You also want to install your module using `develop`

you know that your code will not work on Python 2.6

on OS X (for some reason) and so you want to skip it.

you have a Python script you want to run tests for, but it's not a module.

you want to install your dependencies from a set a wheels in a wheelhouse dir.

You want to pass down a specific env var from your shell to the test env and create a new env var for your tests.

you want to deploy your code but don't care about the exit code.

you want to create a --verbose vs. --debug mode for executing your tests.

you based your API docs on sphinx and want to generate them after every commit to make sure you didn't break anything in the docstrings.

you want to test against multiple versions of the same dependency

you want to use the same set of dependencies or changes to a virtualenv in all test environments.

setup.py integration

detox

local parallelization

install detox

same tox.ini file!

parallel fun!

tox and Travis

remote parallelization

.travis.yml

parallel fun!

Pitalls

getting intoxicated

you've been running tox for a while and decided to upgrade pyyaml from 3.10 to 3.11 and run tox again.

detox requires test environments to be independent or clean.

Useful Links