Alexander Hultnér
Founder, Hultnér Technologies (https://hultner.se). Want me to speak at your company? Corporate training? Or just a fresh pair of eyes on your project? Contact me for contracts.
Test Faster, Fix More
Property based testing with Hypothesis
Alexander Hultnér @ PyCon SE 2019
@ahultner
Alexander Hultnér
Founder of Hultnér Technologies
@ahultner
Test Faster, Fix More
Outline
What is property based testing? Why?
Cheatsheet, two minute abstract
Hypothesis
Usage examples
Advanced strategies
Rule state machines
Interactive demo in Jupyter
Conclusion
Questions and links
@ahultner
Test Faster, Fix More
What & Why
- Traditional
- Specific examples
- Repetition
- Leads to "happy path testing"
- Property based testing
- QuickCheck
- Test behaviour, find unexpected
- Generate thousands of examples
- Shrink errors
- Detect subtle bugs
- Compare when refactoring
@ahultner
Test Faster, Fix More
Cheatsheet
- Define input & behaviour, not specific examples
- QuickCheck by John Hughes
- Hypothesis
-
pip install hypothesis
-
-
Beyond Unit Tests
- Hillel Wayne, PyCon 2018
-
Better Testing With Less Code
- Matt Bachmann, PyCon 2016
- Choosing properties for property-based testing (F#)
- GitHub Repository
@ahultner
Test Faster, Fix More
Hypothesis
- Python Library
- http://hypothesis.works
- Tests for regressions
-
Large library of "strategies"
- (dp)contracts
- Swagger conformance testing
- Pandas, Numpy, Django, etc
-
Rules
- State machines, shows steps to reproduce a bug
@ahultner
Test Faster, Fix More
Minimal usage
Generates positional arguments x, y
Strategies ensure integers are given
Strategies exists for a vast number of input types, they can be inferred and passed constraints via arguments.
from hypothesis import given
import hypothesis.strategies as st
@given(
st.integers(),
st.integers()
)
def test_ints_are_commutative(x, y):
assert x + y == y + x@ahultner
Test Faster, Fix More
Shrinking errors
When an error is found it will be shrunk
to smallest possible example
=== FAILURES ===
___ test_add ___
@given(st.integers(), st.integers())
> def test_add(a, b):
_ _ _ _ _ _ _
a = -1, b = 1
@given(st.integers(), st.integers())
def test_add(a, b):
> assert add(a, b) == a + b
E assert -1 == 0
E --1
E +0
AssertionError
--- Hypothesis ---
Falsifying example: test_add(a=-1, b=1)# Buggy implementation of add
def add(a, b):
return a if a < 0 else a + b@ahultner
Test Faster, Fix More
Using @example
We can explicitly define input with @example.
Useful to keep track of known tricky cases
@given(st.integers(), st.integers())
@example(
-1, 1
)
def test_add(a, b):
assert add(a, b) == a + b@ahultner
Test Faster, Fix More
Python 2 to 3
- Migrating legacy code
- 2 –> 3
- Old "correct" application as oracle
- As in source of truth,
not the company
- As in source of truth,
- Align new implementation with new
@ahultner
Test Faster, Fix More
…
@given(
st.integers(),
st.integers()
)
def test_add(a, b):
assert add_py3(a, b) == add_py2(a, b)
…Strategies
- Tuples, Lists
- Text, Characters
- Floats, Integers
- Datetimes, timezones
- Narrowing with max/min, etc
- API Fuzzing example in docs.
- First and third party extensions
- Inference through type annotations
@ahultner
Test Faster, Fix More
Rules
With Hypothesis’s stateful testing, Hypothesis instead tries to generate not just data but entire tests.
Rule based state machines will allow you to define primitives from which tests are generated, gives you detailed steps taken to produce the bug, read more.
@ahultner
Test Faster, Fix More
Demo time
I've prepared a Jupyter Notebook with a few interactive and more thorough examples.
@ahultner
Test Faster, Fix More
Conclusion
- Write more extensive tests faster
- Find more bugs
- Not a silver bullet for all testing
but a powerful tool in your belt - Think about defining properties not specific examples
- Start playing around with property-based testing!
@ahultner
Test Faster, Fix More
Questions
Contact me if you have any further
questions.
Want to learn more?
Available for training, workshops and
consulting.
Links
@ahultner
Test Faster, Fix More
PyCon Sweden 2019: Test Fast, Fix More – Property based in Python testing with Hypothesis
By Alexander Hultnér
PyCon Sweden 2019: Test Fast, Fix More – Property based in Python testing with Hypothesis
Did you ever miss that corner case bug? Maybe it was a negative integer, strange timezone conversion behaviour, off by one error or something entirely else. These subtle bugs are often hard to catch and are easily missed in test cases. You like me have probably ran into plenty of code utilising only happy path testing, only to later discover subtle bugs which are easily fixed once pointed out. This is where property based testing comes into the picture. In this talk I will focus on a wonderful Python library called Hypothesis but the concepts apply to other languages as well. Hypethesis is based on the same concept as the famous QuickCheck library for Haskell, which in turn have been ported a large number of languages. Hypothesis uses a wide range of input to find edge cases that you could otherwise easily miss, once it finds these cases it narrows down the input to the minimal breaking example to provide failures which are easier to understand.
