i wrote a             plugin

(and so can you!)

py

test

jay luker | @lbjay

senior software engineer, harvard division of continuing education

• pytest
  
• pytest plugins
• ghost inspector
• pytest-ghostinspector

what i'm going to talk about

pytest is ...

• an open source python testing framework
• roughly analogous to unittest or nose
• a framework-agnostic python test runner
• documented at http://pytest.org
• developed at https://github.com/pytest-dev/pytest
• installable via `pip install pytest`

why pytest?

less work & boilerplate for your 80% cases

# src/foobar.py

def foo(x):
    return x * x

def bar(x):
    return x ** x

# tests/test_foobar.py

def test_foo():
    assert foo(3) == 9

def test_bar():
    assert bar(3) == 30

more functionality for your 20% cases

# tests/conftest.py
import pytest

@pytest.fixture(autouse=True)
def disable_requests(monkeypatch):
    monkeypatch.delattr("requests.sessions.Session.request")

# tests/test_user.py
import pytest

@pytest.mark.parametrize('username,password', [
    ('miguel', 'abc123'),
    ('sigríðr', '3CpwDxApl8lw'),
    pytest.mark.xfail(('naomi', 'bad_password'))
])
def test_login(username, password)
    user = User(username)
    assert user.login(password) == True

all assertions with "assert"...

def test_foo():
    assert 3 ** 3 == 27
    assert "this"[::-1] == "siht"
    assert 5 in range(10)
    assert isinstance([], list)
    assert "key" not in {}
    assert frobnozzle(), "Failed to frob the nozzle"

    # but what about "assert{Not}AlmostEqual"?
    assert round(9.23585 - 9.23661, 2) == 0

... unless expecting an exception

import pytest

def test_zero_division():
    with pytest.raises(ZeroDivisionError) as excinfo:
        1 / 0
    assert 'division or modulo by zero' in str(excinfo.value)

plugins!

plugin basics

• pytest core == 300 lines of code + plugins
• 1:N hook-based architecture
• 2 types of plugins:
  • conftest.py, aka "local"
  • 3rd-party
• hooks grouped into roughly five phases:
  • initialization
  • collection
  • running
  • reporting
  • debugging

• per-directory customization
• pytest executes (first) any conftest.py it finds
• can contain fixtures or hook functions

conftest.py, aka "local" plugins

# tests/foo/conftest.py
def pytest_runtest_setup(item):
    print ("setup", item)

# tests/foo/test_foo.py
def test_something():
    pass

# tests/bar/test_bar.py
def test_something():
    pass

3rd-party plugins

# pytest-vw/plugin.py
import pytest

@pytest.hookimpl(hookwrapper=True)
def pytest_runtest_makereport(item, call):
    outcome = yield
    outcome.result.__dict__['passed'] = True
    outcome.result.outcome = 'passed'

• registered/discovered via setuptools entry points
• install via pip
• full list + build status: plugincompat.herokuapp.com

# setup.py

setup(...
    entry_points={
        'pytest11': ['vw = pytest_vw.plugin']
    }
)

initialization phase

# Automatically parallelize over all cpus
def pytest_cmdline_preparse(args):
    if 'xdist' in sys.modules: # pytest-xdist plugin
        import multiprocessing
        num = multiprocessing.cpu_count()
        args[:] = ["-n", str(num)] + args

modify command-line args

def pytest_addoption(parser):
    parser.addoption("--db",
      action = "store",
      default = "sqlite",
      help = "what db backend to test"
    )

add additional command-line args

collection phase

def pytest_collection_modifyitems(items):
    items[:] = reversed(sorted(
                   items, 
                   key=lambda a: a.module.__name__ + a.name
               ))

run your test items in reverse lexical order (?)

@pytest.mark.tryfirst
def pytest_pycollect_makeitem(collector, name, obj):
    if collector.funcnamefilter(name) and _is_coroutine(obj):
        item = pytest.Function(name, parent=collector)
        if ('asyncio' in item.keywords or
           'asyncio_process_pool' in item.keywords):
            return list(collector._genfunctions(name, obj))

only generate specially marked test items

running phase

def pytest_runtest_call(item):
    try:
        item.runtest()
    except Exception:
        # Store trace info to allow postmortem debugging
        type, value, tb = sys.exc_info()
        [...]
        raise

control actual test execution

@pytest.hookimpl(hookwrapper=True)
def pytest_runtest_call(self, item):
    self.resumecapture()
    self.activate_funcargs(item)
    yield
    self.suspendcapture_item(item, "call")

wrap test execution

reporting phase

def pytest_report_teststatus(report):
    if report.passed:
        letter = colored('✓', THEME['success'])
    elif report.skipped:
        letter = colored('⚫', THEME['skipped'])
    elif report.failed:
        letter = colored('⨯', THEME['fail'])
        if report.when != "call":
            letter = colored('ₓ', THEME['fail'])

    if hasattr(report, "wasxfail"):
        if report.skipped:
            return "xfailed", 
                    colored('x', THEME['xfailed']), "xfail"
        ...

    return report.outcome, letter, report.outcome.upper()

show alternate test status indicators

pytest-sugar

ghost inspector

• an automated website testing & monitoring service
• a web UI for creating and editing web tests
• a chrome extension for recording web tests
• free for 1 user + 100 tests/month*

ghost inspector is...

* 1000/month if you also sign up for a free runscope.com account

demo 

ghost inspector api

api base url: https://api.ghostinspector.com/v1

suites

tests

test/suite execution parameters

pytest-ghostinspector

• pip install pytest-ghostinspector
• uses the API to execute tests
• specify tests by...

the basics

# tests/gi_test_foo.yml

suites:
  - id: abcd1234

writing .yml files

%> py.test --gi_key=abc123 --gi_suite=hjkl1234

or command-line options

command-line options

%> py.test --help
[...]
ghostinspector:
  --gi_key=GI_KEY       Set value of Ghost Inspector API key
  --gi_start_url=GI_START_URL
                        Override starting url value
  --gi_suite=GI_SUITE   Id of a Ghost Inspector suite to exec
  --gi_test=GI_TEST     Id of a Ghost Inspector test to exec
  --gi_param=GI_PARAM   Querystring param (repeatable) to inc
                        in the API execute request. 
                        Example: "--gi_param foo=bar"

options via...

[pytest]
addopts =
    --gi_key=abcd123
    --gi_start_url=http://ec2-1-2-3-4.compute-1.amazonaws.com
    --gi_suite=56c77b4513888c3373bdb4f4
    --gi_param=username=guest
    --gi_param=password=foobar

%> py.test --gi_key=abcd123 \
    --gi_start_url=http://ec2-1-2-3-4.compute-1.amazonaws.com \
    --gi_suite=56c77b4513888c3373bdb4f4 \
    --gi_param=username=guest \
    --gi_param=password=foobar

command line

pytest.ini

• checks for --gi_test / --gi_suite args
  • creates temporary yaml files
  • adds temp file paths to command-line args
• looks for files named "gi_test_*.yml"
• issues api requests to fetch test metadata
• generates pytest test items
• issues api requests to execute tests
• generates test status report from api responses

how it works

plugin.py - adding options

def pytest_addoption(parser):
    group = parser.getgroup('ghostinspector')
    group.addoption(
        '--gi_key',
        action='store',
        dest='gi_key',
        default=env('GI_API_KEY'),
        help='Set the value for the Ghost Inspector API key'
    )
    group.addoption('--gi_start_url', ...)
    group.addoption('--gi_suite', ...)
    group.addoption('--gi_test', ...)
    group.addoption('--gi_param', ...)
   

plugin.py - collecting tests

def pytest_collect_file(path, parent):
    """Collection hook for ghost inspector tests [...]
    """
    if (path.basename.startswith('gi_test_') 
            and path.ext == '.yml'):
        if parent.config.option.gi_key is None:
            raise pytest.UsageError("Missing --gi_key option")
        return GIYamlCollector(path, parent=parent)

plugin.py - collecting tests (cont)

class GIYamlCollector(pytest.File, GIAPIMixin):
    """Collect and generate pytest test items [...]"""

    ...

    def collect(self):
        raw = yaml.safe_load(self.fspath.open())

        for suite in raw.get('suites', []):
            for test_item in self._collect_suite(suite):
                yield test_item

        for test in raw.get('tests', []):
            yield self._collect_test(test)
    ...

plugin.py - collecting tests (cont)

@pytest.hookimpl(hookwrapper=True)
def pytest_collection(session):
    ...
    if not session.config.option.gi_suite \
            and not session.config.option.gi_test:
        yield
    else:
        with _make_tmp_dir() as tmpdir:
            tmp_files = []
            for id in session.config.option.gi_suite:
                test_yaml = {'suites': [{'id': id}]}
                tmp_file = _make_tmp_yaml(tmpdir, test_yaml)
                tmp_files.append(tmp_file)
            for id in session.config.option.gi_test:
                test_yaml = {'tests': [{'id': id}]}
                tmp_file = _make_tmp_yaml(tmpdir, test_yaml)
                tmp_files.append(tmp_file)
            session.config.args += tmp_files
            yield

plugin.py - running tests

class GITestItem(pytest.Item, GIAPIMixin):
    ... 
    def runtest(self):
        url = API_URL + \
              ('tests/%s/execute/' % self.spec['id'])
        result = self._api_request(url, self.spec['params'])
        if not result['passing']:
            raise GIException(self, result)
    ...

plugin.py - running tests (cont)

    ...
    def repr_failure(self, excinfo):
        """ format failure info from GI API response """
        if isinstance(excinfo.value, GIException):
            resp_data = excinfo.value.args[1]
            failing_step = next(
                step for step in resp_data['steps'] 
                    if not step['passing']
            )
            result_url = '%s/%s' % (RESULT_BASE, resp_data['_id'])
            return "\n".join([
                "Ghost Inspector test failed",
                "   name: %s" % resp_data['test']['name'],
                "   result url: %s" % result_url,
                "   sequence: %d" % failing_step['sequence'],
                "   target: %s" % failing_step['target'],
                "   command: %s" % failing_step['command'],
                "   value: %s" % failing_step['value'],
                "   error: %s" % failing_step['error']
            ])
    ...

more demoing

• use the "pytester" plugin
• read other people's tests :) 

plugin testing

# tests/conftest.py
pytest_plugins = "pytester"

# tests/test_ghostinspector.py

@pytest.mark.httpretty
def test_cmdline_collect_test(testdir, test_resp, gi_api_test_re):
    ...
    result = testdir.runpytest(
        '--collect-only',
        '--gi_key=foo',
        '--gi_test=xyz789'
    )
    result.stdout.fnmatch_lines([
        u'collected 1 items',
        u"*GITestItem*'test xyz789'*",
    ])

• y'all

• especially Ned, et al

• pytest contributors

• @aafshar for pytest-yamlwsgi

• @hackebrot for cookiecutter-pytest-plugin

• @ghostinspector team

thanks to...