Don't just test, my friend, test better

Get the slides: slides.com/cheukting_ho/test-better/

Cheuk Ting Ho

@cheukting_ho

Cheukting

https://cheuk.dev

@cheuktingho

Hello I am Cheuk

  • Open-Source contributor


     
  • Organisers of community events


     
  • PSF director and fellow
     
  • Community manager at OpenSSF

Who uses pytest?

Who thinks that you know pytest really well?

What makes pytest popular?

Unittest

  • Default on that comes with Python
  • Universally understand
  • Require importing the module and defining test classes

Pytest

  • More straightforward to use
  • Just add "test_"
  • Extra features

Pytest features

How many are you using?

parameterize

Have you ever realized

you are writing the same test OVER and OVER?

import reminder as app
from reminder import Task


def test_find_task():
    task_list = [Task(name="pay rent"), Task(name="buy bread")]
    assert app._find_task("buy bread", task_list) == Task(name="buy bread")

def test_find_task_upper_case():
    task_list = [Task(name="pay rent"), Task(name="buy bread")]
    assert app._find_task("PAY RENT", task_list) == Task(name="pay rent")

def test_find_task_none():
    task_list = [Task(name="pay rent"), Task(name="buy bread")]
    assert app._find_task("buy banana", task_list) is None

If you have done this...

import reminder as app
from reminder import Task

import pytest


@pytest.mark.parametrize(
    "test_input, expected",
    [
        ("buy bread", Task(name="buy bread")),
        ("buy banana", None),
        ("PAY RENT", Task(name="pay rent")),
    ],
)
def test_find_task(test_input, expected):
    task_list = [Task(name="pay rent"), Task(name="buy bread")]
    assert app._find_task(test_input, task_list) == expected

Maybe you can do this

I don't know what to test...

hypothesis

not included in pytest

Property

  • the given is obvious

     
  • works extra well with typing

     
  • edge case automatically be found

Example

  • need to think of what is and what is not
     
  • take extra steps to write examples

     
  • may overlook edge cases

Testing by...

 Hypothesis uses

decorators

entry point to modify the test

strategies

generating test data

import reminder as app
from reminder import Task

import pytest
from hypothesis import given, strategies as st

@given(st.text())
def test_cant_find_task(test_input):
    task_list = [Task(name="pay rent"), Task(name="buy bread")]
    assert app._find_task(test_input, task_list) is None

@given(st.lists(st.text(), unique=True))
def test_find_task(test_list):
    task_list = map(lambda x: Task(name=x), test_list)
    for test_input in test_list:
        assert app._find_task(test_input, task_list) == Task(name=test_input)

fixture

What is fixture?

What is fixture?

(and why it's useful?)

  • Takes elements out from your test 
  • Make tests cleaner
  • Reuse those elements
  • e.g. need external elements in your test
  • for building pytest extension (advance)
import reminder as app
from reminder import Task

import pytest


@pytest.mark.parametrize(
    "test_input, expected",
    [
        ("buy bread", Task(name="buy bread")),
        ("buy banana", None),
        ("PAY RENT", Task(name="pay rent")),
    ],
)
def test_find_task(test_input, expected):
    tast_list = [Task(name="pay rent"), Task(name="buy bread")]
    assert app._find_task(test_input, tast_list) == expected
    
def test_save_load_task_list():
    tast_list = [Task(name="pay rent"), Task(name="buy bread")]
    app._save_task_list(task_list)
    load_list = app._get_task_list()
    assert task_list == load_list
import reminder as app
from reminder import Task

import pytest


@pytest.fixture
def task_list():
    return [Task(name="pay rent"), Task(name="buy bread")]

@pytest.mark.parametrize("test_input, expected",
        [("buy bread", Task(name="buy bread")),
         ("buy banana", None),
         ("PAY RENT", Task(name="pay rent")),
         ])
def test_find_task(test_input, expected, task_list):
    assert app._find_task(test_input, task_list) == expected

def test_save_load_task_list(task_list):
    app._save_task_list(task_list)
    load_list = app._get_task_list()
    assert task_list == load_list

Sharing fixtures

import pytest
import smtplib


@pytest.fixture(scope="module")
def smtp_connection():
    return smtplib.SMTP("smtp.gmail.com", 587, timeout=5)
def test_ehlo(smtp_connection):
    response, msg = smtp_connection.ehlo()
    assert response == 250
    assert b"smtp.gmail.com" in msg
    assert 0  # for demo purposes


def test_noop(smtp_connection):
    response, msg = smtp_connection.noop()
    assert response == 250
    assert 0  # for demo purposes

conftest.py

test_module.py

Can be shared among different scopes:

  • classes
  • modules
  • packages
  • session

don't get me started....

FIXTURES are one of the most powerful features in PYTEST

Allow 3rd-party plugins

Check out: https://docs.pytest.org/en/6.2.x/fixture.html

mark xfail

You want to test a new feature

That is only available to the newest versions of Python

Imagine...

You know when the test suit is run on certain versions of Python...

these tests will fail

How can you let the CICD pipeline knows not to fail the whole test suit?

A.k.a

How to skip a test gracefully?

use mark.skip

or

mark.skipif

@pytest.mark.skip(reason="coming soon")
def test_new_feature():
    ...

from sys import version_info

@pytest.mark.skipif(version_info < (3, 8), reason="requires >= python3.8")
def test_function():
    ...

Or use mark.xfail

to temporary make a failed test pass

To be fixed later

@pytest.mark.xfail(reason="known issue")
def test_function():
    ...

from sys import platform

@pytest.mark.xfail(platform == "win32")
def test_function():
    ...

Please state the reason

Top tips

  • parameterize
  • fixture
  • mark xfail