Test Faster, Fix More

Property based testing with Hypothesis

Alexander Hultnér @ PyCon SE 2019

@ahultner

Alexander Hultnér

Founder of Hultnér Technologies

linkedin.com/in/hultner

@ahultner

Test Faster, Fix More

Outline

What is property based testing? Why?

Cheatsheet, two minute abstract

Hypothesis

Usage examples

Advanced strategies

Rule state machines

Interactive demo in Jupyter

Conclusion

Questions and links

@ahultner

Test Faster, Fix More

What & Why

Traditional
- Specific examples
- Repetition
- Leads to "happy path testing"
Property based testing
- QuickCheck
- Test behaviour, find unexpected
- Generate thousands of examples
- Shrink errors
- Detect subtle bugs
- Compare when refactoring

@ahultner

Test Faster, Fix More

Cheatsheet

Define input & behaviour, not specific examples
QuickCheck by John Hughes
Hypothesis
- ```
pip install hypothesis
```
Beyond Unit Tests
- Hillel Wayne, PyCon 2018
Better Testing With Less Code
- Matt Bachmann, PyCon 2016
Choosing properties for property-based testing (F#)
GitHub Repository

@ahultner

Test Faster, Fix More

Hypothesis

Python Library
http://hypothesis.works
Tests for regressions
Large library of "strategies"
- (dp)contracts
- Swagger conformance testing
- Pandas, Numpy, Django, etc
Rules
- State machines, shows steps to reproduce a bug

@ahultner

Test Faster, Fix More

Minimal usage

Generates positional arguments x, y

Strategies ensure integers are given

Strategies exists for a vast number of input types, they can be inferred and passed constraints via arguments.

from hypothesis import given
import hypothesis.strategies as st

@given(
    st.integers(), 
    st.integers()
)
def test_ints_are_commutative(x, y):
    assert x + y == y + x

@ahultner

Test Faster, Fix More

Shrinking errors

When an error is found it will be shrunk

to smallest possible example

=== FAILURES ===
___ test_add ___

    @given(st.integers(), st.integers())
>   def test_add(a, b):


_ _ _ _ _ _ _ 

a = -1, b = 1

    @given(st.integers(), st.integers())
    def test_add(a, b):
>       assert add(a, b) == a + b
E       assert -1 == 0
E         --1
E         +0

AssertionError
--- Hypothesis ---
Falsifying example: test_add(a=-1, b=1)

# Buggy implementation of add
def add(a, b): 
    return a if a < 0 else a + b

@ahultner

Test Faster, Fix More

Using @example

We can explicitly define input with @example.

Useful to keep track of known tricky cases

@given(st.integers(), st.integers())
@example(
    -1, 1
)
def test_add(a, b):
    assert add(a, b) == a + b

@ahultner

Test Faster, Fix More

Python 2 to 3

Migrating legacy code
- 2 –> 3
Old "correct" application as oracle
- As in source of truth,
  not the company
Align new implementation with new

@ahultner

Test Faster, Fix More

…
@given(
  st.integers(), 
  st.integers()
)
def test_add(a, b):
    assert add_py3(a, b) == add_py2(a, b)
…

Strategies

Tuples, Lists
Text, Characters
Floats, Integers
Datetimes, timezones
Narrowing with max/min, etc
API Fuzzing example in docs.
First and third party extensions
Inference through type annotations

@ahultner

Test Faster, Fix More

Rules

With Hypothesis’s stateful testing, Hypothesis instead tries to generate not just data but entire tests.

Rule based state machines will allow you to define primitives from which tests are generated, gives you detailed steps taken to produce the bug, read more.