tox
(not poison)
Nir Cohen@Gigaspaces
nir0s@github
@thinkops
Agenda
-
current problems with testing
-
tox!
-
Use cases
-
Integrations and parallelization
-
Pitfalls
Current testing problems
Testing on multiple versions of Python.
(py27):~/repex$ python --version
Python 2.7.6
(py27):~/repex$ nosetests
...................................
----------------------------------------------------------------------
Ran 35 tests in 0.162s
OK
:~/repex$ virtualenv -p /usr/bin/python2.6 py26
:~/repex$ source py26/bin/activate
(py26):~/repex$ python --version
Python 2.6.9
(py26):~/repex$ pip install . -r dev-requirements
(py26):~/repex$ nosetests
Managing test env dependencies.
(py27):~/repex$ cat dev-requirements.txt
coverage==3.7.1
nose
nose-cov
testfixtures
(py27):~/repex$ pip install -r dev-requirements . --upgrade
(py27):~/repex$ nosetests --with-cov --cov-report term-missing --cov repex repex/tests -v(py27):~/repex$ source py26/bin/activate
(py26):~/repex$ pip install -r dev-requirements . --upgrade
(py26):~/repex$ nosetests --with-cov --cov-report term-missing --cov repex repex/tests -v
Executing pre and post test related activities.
(py27):~/repex$ mypyserv --port 5000 &
(py27):~/repex$ nosetests --with-cov --cov-report term-missing \
--cov repex repex/tests -vRunning tests in clean environments.
:~/repex$ virtualenv py27
:~/repex$ source py27/bin/activate
(py27):~/repex$ pip install -r dev-requirements.txt .
(py27):~/repex$ nosetests --with-cov --cov-report term-missing --cov repex repex/tests -v:~/repex$ deactivate
:~/repex$ rm -rf py27
:~/repex$ virtualenv py27
:~/repex$ source py27/bin/activate
(py27):~/repex$ pip install -r dev-requirements.txt .
(py27):~/repex$ nosetests --with-cov --cov-report term-missing --cov repex repex/tests -vFlake8 should be run on every commit - annoying.
(py27):~/repex$ nosetests --with-cov --cov-report term-missing --cov repex repex/tests -v
(py27):~/repex$ pip install flake8
(py27):~/repex$ flake8 repex... add some code
(py27):~/repex$ git commit -am "yay!, moar!"
# because who knows.. maybe you added more dependencies
(py27):~/repex$ pip install -r dev-requirements.txt . --upgrade
(py27):~/repex$ nosetests --with-cov --cov-report term-missing --cov repex repex/tests -v
# meh! again!?
(py27):~/repex$ flake8 repex-
Complex and repetitive execution of tests.
- Most of the above must be solved for every testing platform (py.test, nose, unittest, etc..)
- Executing tests in parallel.
Let's (de)tox
tox...
- manages virtualenvs
- very configurable
- supports most common interpreters (from py24 to py34, jython and pypy).
- more. let's see, shall we?
install tox
pip install toxtox.ini
[tox]
# list to environments to test against. Either interpreters or arbitrary names.
envlist=flake8,py27
# env for all interpreters
[testenv]
# install dependencies
deps =
-rdev-requirements.txt
# execute commands within the virtualenv
commands=nosetests --with-cov --cov-report term-missing --cov repex repex/tests -v
# runs on default interpreter. name for specific configuration.
[testenv:flake8]
deps =
flake8
-rdev-requirements.txt
commands=flake8 repex
toxin'
(py27):~/repex$ ll
total 96
drwxrwxr-x 5 nir0s nir0s 4096 אוק 10 13:44 ./
drwxr-xr-x 55 nir0s nir0s 4096 אוק 9 11:07 ../
-rw-rw-r-- 1 nir0s nir0s 4367 ספט 28 19:06 CHANGELOG
drwxrwxr-x 3 nir0s nir0s 4096 אפר 5 2015 check_validity/
-rw-rw-r-- 1 nir0s nir0s 52 אפר 5 2015 .coveragerc
-rw-rw-r-- 1 nir0s nir0s 52 אפר 1 2015 dev-requirements.txt
drwxrwxr-x 8 nir0s nir0s 4096 אוק 10 13:44 .git/
-rw-rw-r-- 1 nir0s nir0s 587 אפר 1 2015 .gitignore
-rw-rw-r-- 1 nir0s nir0s 11325 אפר 1 2015 LICENSE
-rw-rw-r-- 1 nir0s nir0s 1092 אפר 1 2015 Makefile
-rw-rw-r-- 1 nir0s nir0s 9159 ספט 28 19:06 README.md
-rw-rw-r-- 1 nir0s nir0s 10099 ספט 28 19:06 README.rst
drwxrwxr-x 3 nir0s nir0s 4096 אוק 10 13:40 repex/
-rw-rw-r-- 1 nir0s nir0s 1277 ספט 30 22:49 setup.py
-rw-rw-r-- 1 nir0s nir0s 346 ספט 28 19:05 TODO.md
-rw-rw-r-- 1 nir0s nir0s 296 אוק 10 13:20 tox.ini
-rw-rw-r-- 1 nir0s nir0s 173 אפר 1 2015 .travis.yml
(py27):~/repex$ toxresults
(repex)nir0s@nir0s-x1:~/repos/repex$ tox
GLOB sdist-make: /home/nir0s/repos/repex/setup.py
flake8 inst-nodeps: /home/nir0s/repos/repex/.tox/dist/repex-0.3.0.zip
flake8 installed: click==4.0,cov-core==1.15.0,coverage==3.7.1,extras==0.0.3,flake8==2.4.1,linecache2==1.0.0,mccabe==0.3.1,nose==1.3.7,nose-cov==1.6,pbr==1.8.1,pep8==1.5.7,pyflakes==0.8.1,python-mimeparse==0.1.4,PyYAML==3.10,repex==0.3.0,six==1.10.0,testfixtures==4.3.3,testtools==1.8.0,traceback2==1.4.0,unittest2==1.1.0,wheel==0.24.0
flake8 runtests: PYTHONHASHSEED='3490089416'
flake8 runtests: commands[0] | flake8 repex
py27 inst-nodeps: /home/nir0s/repos/repex/.tox/dist/repex-0.3.0.zip
py27 installed: click==4.0,cov-core==1.15.0,coverage==3.7.1,extras==0.0.3,linecache2==1.0.0,nose==1.3.7,nose-cov==1.6,pbr==1.8.1,python-mimeparse==0.1.4,PyYAML==3.10,repex==0.3.0,six==1.10.0,testfixtures==4.3.3,testtools==1.8.0,traceback2==1.4.0,unittest2==1.1.0,wheel==0.24.0
py27 runtests: PYTHONHASHSEED='3490089416'
py27 runtests: commands[0] | nosetests
...................................
----------------------------------------------------------------------
Ran 35 tests in 0.152s
OK
______________________________________________________________________________________________ summary ______________________________________________________________________________________________
flake8: commands succeeded
py27: commands succeeded
congratulations :)
dev
- Elaborate docs at: http://goo.gl/utWvRp
- Constantly developed: https://goo.gl/acS10a
Use Cases
you specified multiple Python versions to run on
but only want to run on one of them now.
[tox]
# tox uses sys.version_info to check if an interpreter is available
# the provided envs.
# If not, it assumes that the env is a name (e.g. flake8, docs)
# these can also be passed via `export TOXENV=env,env,env`
envs=py26,py27
[testenv]
commands=nosetests
[testenv:py26]
commands=nosetests:~/repex$ tox -e py27
...
py27 runtests: PYTHONHASHSEED='1593161913'
py27 runtests: commands[0] | nosetests
...Your default Python interpreter is not the one required for
that specific test. You also want to install your module using `develop`
[testenv:py27]
# these will practically run /usr/local/bin/anaconda/python setup.py develop
basepython=/usr/local/bin/anaconda/python
usedevelop=Trueyou know that your code will not work on Python 2.6
on OS X (for some reason) and so you want to skip it.
[testenv]
commands=nosetests
[testenv:py26]
commands=nosetests
# tox uses sys.platform to check against the provided regex entry.
platform=linux2|win32you have a Python script you want to run tests for, but it's not a module.
[tox]
# tox will not try to build the module before executing the command.
skipsdist = True
[testenv:py27]
commands =
nosetests --with-cov --cov get-cloudify.py tests -vyou want to install your dependencies from a set a wheels in a wheelhouse dir.
[testenv]
commands=nosetests
[testenv]
commands=nosetests
# tox will replace the default pip install command with the below.
install_command = pip install --find-links wheelhouse/ --no-index {opts} {packages}You want to pass down a specific env var from your shell to the test env and create a new env var for your tests.
[testenv]
passenv =
# this is passed from the shell running the test
MY_TEST_RELATED_ENV_VAR
setenv =
# by default, the port is 1003, but our tests override using this env var
PORT = 5000
commands =
nosetestsyou want to deploy your code but don't care about the exit code.
[testenv]
commands =
nosetests
flake8
- deploy_carelesslyyou want to create a --verbose vs. --debug mode for executing your tests.
[testenv]
commands =
nosetests {posargs:-v}# so with allowing to pass "-s" instead, this acts as verbose vs. debug
:~/repex$ tox -- -s
...
py27 runtests: PYTHONHASHSEED='1593161913'
py27 runtests: commands[0] | nosetests -s
...you based your API docs on sphinx and want to generate them after every commit to make sure you didn't break anything in the docstrings.
[testenv:docs]
# this will cd to docs from the root dir of where tox.ini is located.
changedir=docs
deps =
sphinx
sphinx-rtd-theme
commands=make html
you want to test against multiple versions of the same dependency
[tox]
envlist = py26-legacy,py26-dev,py27-legacy,py27-dev
[testenv:py26-legacy]
deps =
dep==0.1
[testenv:py26-dev]
deps =
dep
[testenv:py27-legacy]
deps =
dep==0.1
[testenv:py27-dev]
deps =
dep
[tox]
envlist = {py26,py27}-{dev,legacy}
[testenv]
deps =
dev: dep==0.1
legacy: dep
you want to use the same set of dependencies or changes to a virtualenv in all test environments.
[tox]
envlist = py26,py27
[testenv]
# all environments will use the same virtualenv.
# by default, tox creates the virtualenvs under {toxinidir}/.tox/NAME
# this allows to test on a modified virtualenv.
envdir = {toxinidir}/.env
commands =
py26: nosetests
py27: nosetests
testserver: run_myserver_test_script
test: more_test_commandssetup.py integration
# setup.py
class Tox(TestCommand):
...
setup(
#...,
tests_require=['tox'],
cmdclass = {'test': Tox},
)
:~/repex$ python setup.py testdetox
local parallelization
install detox
pip install detoxsame tox.ini file!
parallel fun!
:~/repex$ pip install detox
# PARALLEL EXECUTION!
:~/repex$ detox
GLOB sdist-make: /home/nir0s/repos/repex/setup.py
py27 inst-nodeps: /home/nir0s/repos/repex/.tox/dist/repex-0.3.0.zip
flake8 inst-nodeps: /home/nir0s/repos/repex/.tox/dist/repex-0.3.0.zip
py27 runtests: PYTHONHASHSEED='2207186455'
py27 runtests: commands[0] | nosetests
flake8 runtests: PYTHONHASHSEED='2207186455'
flake8 runtests: commands[0] | flake8 repex
______________________________________________________________________________________________ summary ______________________________________________________________________________________________
flake8: commands succeeded
py27: commands succeeded
congratulations :)
tox and Travis
remote parallelization
.travis.yml
sudo: false
language: python
python:
- "2.7"
env:
- TOX_ENV=flake8
- TOX_ENV=py27
- TOX_ENV=py26
install:
- pip install tox
script:
- tox -e $TOX_ENV
parallel fun!
Pitalls
getting intoxicated
you've been running tox for a while and decided to upgrade pyyaml from 3.10 to 3.11 and run tox again.
:~/repex$ tox -e py26
...
:~/repex$ .tox/py26/bin/pip freeze
...
PyYAML==3.10
repex==0.3.0
...
# WTF?!# tox does not run --upgrade on depdendencies. you can use the --recreate
# flag to recreate the virtualenv and install the updated deps.
:~/repex$ tox -e py26 --recreate
...
:~/repex$ .tox/py26/bin/pip freeze
...
PyYAML==3.11
repex==0.3.0
...
# yAY!detox requires test environments to be independent or clean.
Useful Links
- https://tox.readthedocs.org/en/latest/
- https://github.com/dstanek/tox-run-command
- https://github.com/cloudify-cosmo/repex/blob/master/tox.ini
- https://github.com/mitsuhiko/flask/blob/master/tox.ini
- https://www.youtube.com/watch?v=Oldkj519o4A
- https://www.youtube.com/watch?v=s_YjaODzM1E
tox (not poison) Nir Cohen@Gigaspaces nir0s@github @thinkops