Alexander Hultnér
Founder, Hultnér Technologies (https://hultner.se). Want me to speak at your company? Corporate training? Or just a fresh pair of eyes on your project? Contact me for contracts.
Test Faster, Fix More
Property based testing with Hypothesis Alexander Hultnér @ GothPy, 2019
@ahultner
Alexander Hultnér
Founder of Hultnér Technologies
@ahultner
Outline
What is property based testing? Why?
Cheatsheet, two minute abstract
Hypothesis
Usage examples
Advanced strategies
Rule state machines
Interactive demo in Jupyter
Conclusion
Questions and links
@ahultner
What & Why
- Traditional
- Specific examples
- Repetition
- Leads to "happy path testing"
- Property based testing
- QuickCheck
- Test behaviour, find unexpected
- Generate thousands of examples
- Shrink errors
- Detect subtle bugs
- Compare when refactoring
@ahultner
Cheatsheet
- Define input & behaviour, not specific examples
- QuickCheck by John Hughes
- Hypothesis
-
pip install hypothesis
-
-
Beyond Unit Tests
- Hillel Wayne, PyCon 2018
-
Better Testing With Less Code
- Matt Bachmann, PyCon 2016
- Choosing properties for property-based testing (F#)
- GitHub Repository
@ahultner
Hypothesis
- Python Library
- http://hypothesis.works
- Tests for regressions
-
Large library of "strategies"
- (dp)contracts
- Swagger conformance testing
- Pandas, Numpy, Django, etc
-
Rules
- State machines, shows steps to reproduce a bug
@ahultner
Minimal usage
Generates positional arguments x, y
Strategies ensure integers are given
Strategies exists for a vast number of input types, they can be inferred and passed constraints via arguments.
@ahultner
from hypothesis import given
import hypothesis.strategies as st
@given(
st.integers(),
st.integers()
)
def test_ints_are_commutative(x, y):
assert x + y == y + xShrinking errors
When an error is found it will be shrunk
to smallest possible example
@ahultner
=== FAILURES ===
___ test_add ___
@given(st.integers(), st.integers())
> def test_add(a, b):
<ipython-input-35-fa7bbf382a62>:6:
_ _ _ _ _ _ _
a = -1, b = 1
@given(st.integers(), st.integers())
def test_add(a, b):
> assert add(a, b) == a + b
E assert -1 == 0
E --1
E +0
<ipython-input-35-fa7bbf382a62>:10: AssertionError
--- Hypothesis ---
Falsifying example: test_add(a=-1, b=1)# Buggy implementation of add
def add(a, b):
return a if a < 0 else a + bUsing @example
We can explicitly define input with @example.
Useful to keep track of known tricky cases
@ahultner
@given(st.integers(), st.integers())
@example(
-1, 1
)
def test_add(a, b):
assert add(a, b) == a + bStrategies
- Tuples, Lists
- Text, Characters
- Floats, Integers
- Datetimes, timezones
- Narrowing with max/min, etc
- API Fuzzing example in docs.
- First and third party extensions
@ahultner
Rules
With Hypothesis’s stateful testing, Hypothesis instead tries to generate not just data but entire tests.
Rule based state machines will allow you to define primitives from which tests are generated, gives you detailed steps taken to produce the bug, read more.
@ahultner
Demo time
I've prepared a Jupyter Notebook with a few interactive and more thorough examples.
@ahultner
Conclusion
- Write more extensive tests faster
- Find more bugs
- Not a silver bullet for all testing
but a powerful tool in your belt - Think about defining properties not specific examples
- Start playing around with property-based testing!
@ahultner
Questions
Contact me if you have any further
questions.
Want to learn more?
Available for training, workshops and
consulting at your company.
@ahultner
Links
GothPy 2019: Test Fast, Fix More – Property based in Python testing with Hypothesis
By Alexander Hultnér
GothPy 2019: Test Fast, Fix More – Property based in Python testing with Hypothesis
Did you ever miss that corner case bug? Maybe it was a negative integer, strange timezone conversion behaviour, off by one error or something entirely else. These subtle bugs are often hard to catch and are easily missed in test cases. You like me have probably ran into plenty of code utilising only happy path testing, only to later discover subtle bugs which are easily fixed once pointed out. This is where property based testing comes into the picture. In this talk I will focus on a wonderful Python library called Hypothesis but the concepts apply to other languages as well. Hypethesis is based on the same concept as the famous QuickCheck library for Haskell, which in turn have been ported a large number of languages. Hypothesis uses a wide range of input to find edge cases that you could otherwise easily miss, once it finds these cases it narrows down the input to the minimal breaking example to provide failures which are easier to understand.
