Test Faster, Fix More

Property based testing with Hypothesis Alexander Hultnér @ foss-north, 2019

@ahultner

Alexander Hultnér

Founder of Hultnér Technologies

@ahultner

contact@hultner.se

hultner.se

slides.com/hultner

linkedin.com/in/hultner

Outline

What is property based testing? Why?

Cheatsheet, two minute abstract

Hypothesis

Usage examples

Advanced strategies

Rule state machines

Interactive demo in Jupyter

Conclusion

Questions and links

@ahultner

What & Why

Traditional
- Specific examples
- Repetition
- Leads to "happy path testing"
Property based testing
- QuickCheck
- Test behaviour, find unexpected
- Generate thousands of examples
- Shrink errors
- Detect subtle bugs
- Compare when refactoring

@ahultner

Cheatsheet

Define input & behaviour, not specific examples
QuickCheck by John Hughes
Hypothesis
- ```
pip install hypothesis
```
Beyond Unit Tests
- Hillel Wayne, PyCon 2018
Better Testing With Less Code
- Matt Bachmann, PyCon 2016
Choosing properties for property-based testing (F#)
GitHub Repository

@ahultner

Hypothesis

Python Library
http://hypothesis.works
Tests for regressions
Large library of "strategies"
- (dp)contracts
- Swagger conformance testing
- Pandas, Numpy, Django, etc
Rules
- State machines, shows steps to reproduce a bug

@ahultner

Minimal usage

Generates positional arguments x, y

Strategies ensure integers are given

Strategies exists for a vast number of input types, they can be inferred and passed constraints via arguments.

@ahultner

from hypothesis import given
import hypothesis.strategies as st

@given(
    st.integers(), 
    st.integers()
)
def test_ints_are_commutative(x, y):
    assert x + y == y + x

Shrinking errors

When an error is found it will be shrunk

to smallest possible example

@ahultner

=== FAILURES ===
___ test_add ___

    @given(st.integers(), st.integers())
>   def test_add(a, b):

<ipython-input-35-fa7bbf382a62>:6: 
_ _ _ _ _ _ _ 

a = -1, b = 1

    @given(st.integers(), st.integers())
    def test_add(a, b):
>       assert add(a, b) == a + b
E       assert -1 == 0
E         --1
E         +0

<ipython-input-35-fa7bbf382a62>:10: AssertionError
--- Hypothesis ---
Falsifying example: test_add(a=-1, b=1)

# Buggy implementation of add
def add(a, b): 
    return a if a < 0 else a + b

Using @example

We can explicitly define input with @example.

Useful to keep track of known tricky cases

@ahultner

@given(st.integers(), st.integers())
@example(
    -1, 1
)
def test_add(a, b):
    assert add(a, b) == a + b

Strategies

Tuples, Lists
Text, Characters
Floats, Integers
Datetimes, timezones
Narrowing with max/min, etc
API Fuzzing example in docs.
First and third party extensions

@ahultner

Rules

With Hypothesis’s stateful testing, Hypothesis instead tries to generate not just data but entire tests.

Rule based state machines will allow you to define primitives from which tests are generated, gives you detailed steps taken to produce the bug, read more.