Test Faster, Fix More

 

Property based testing with Hypothesis

Alexander Hultnér @ PyCon SE 2019

@ahultner

Alexander Hultnér

Founder of Hultnér Technologies

@ahultner

Test Faster, Fix More

Outline

What is property based testing? Why?

Cheatsheet, two minute abstract

Hypothesis

Usage examples

Advanced strategies

Rule state machines

Interactive demo in Jupyter

Conclusion

Questions and links

@ahultner

Test Faster, Fix More

What & Why

  • Traditional
    • Specific examples
    • Repetition
    • Leads to "happy path testing"
  • Property based testing
    • QuickCheck
    • Test behaviour, find unexpected
    • Generate thousands of examples
    • Shrink errors
    • Detect subtle bugs
    • Compare when refactoring

@ahultner

Test Faster, Fix More

Cheatsheet

@ahultner

Test Faster, Fix More

Hypothesis

  • Python Library
  • http://hypothesis.works
  • Tests for regressions
  • Large library of "strategies"
    • ​(dp)contracts
    • Swagger conformance testing
    • Pandas, Numpy, Django, etc
  • Rules
    • State machines, shows steps to reproduce a bug

@ahultner

Test Faster, Fix More

Minimal usage

Generates positional arguments x, y

Strategies ensure integers are given

 

Strategies exists for a vast number of input types, they can be inferred and passed constraints via arguments.

from hypothesis import given
import hypothesis.strategies as st

@given(
    st.integers(), 
    st.integers()
)
def test_ints_are_commutative(x, y):
    assert x + y == y + x

@ahultner

Test Faster, Fix More

Shrinking errors

When an error is found it will be shrunk

to smallest possible example

=== FAILURES ===
___ test_add ___

    @given(st.integers(), st.integers())
>   def test_add(a, b):


_ _ _ _ _ _ _ 

a = -1, b = 1

    @given(st.integers(), st.integers())
    def test_add(a, b):
>       assert add(a, b) == a + b
E       assert -1 == 0
E         --1
E         +0

AssertionError
--- Hypothesis ---
Falsifying example: test_add(a=-1, b=1)
# Buggy implementation of add
def add(a, b): 
    return a if a < 0 else a + b

@ahultner

Test Faster, Fix More

Using @example

We can explicitly define input with @example.

Useful to keep track of known tricky cases

@given(st.integers(), st.integers())
@example(
    -1, 1
)
def test_add(a, b):
    assert add(a, b) == a + b

@ahultner

Test Faster, Fix More

Python 2 to 3

  • Migrating legacy code
    • 2 –> 3
  • Old "correct" application as oracle
    • As in source of truth,
      not the company
  • Align new implementation with new

@ahultner

Test Faster, Fix More

…
@given(
  st.integers(), 
  st.integers()
)
def test_add(a, b):
    assert add_py3(a, b) == add_py2(a, b)
…

Strategies

  • Tuples, Lists
  • Text, Characters
  • Floats, Integers
  • Datetimes, timezones
  • Narrowing with max/min, etc
  • API Fuzzing example in docs.
  • First and third party extensions
  • Inference through type annotations

@ahultner

Test Faster, Fix More

Rules

With Hypothesis’s stateful testing, Hypothesis instead tries to generate not just data but entire tests.

 

Rule based state machines will allow you to define primitives from which tests are generated, gives you detailed steps taken to produce the bug, read more.

@ahultner

Test Faster, Fix More

Demo time

I've prepared a Jupyter Notebook with a few interactive and more thorough examples.

@ahultner

Test Faster, Fix More

Conclusion

  • Write more extensive tests faster
  • Find more bugs
  • Not a silver bullet for all testing
    but a powerful tool in your belt
  • Think about defining properties not specific examples
  • Start playing around with property-based testing!

@ahultner

Test Faster, Fix More

Questions

Contact me if you have any further

questions.

 

 

Want to learn more?

Available for training, workshops and

consulting.

Links

@ahultner

Test Faster, Fix More

PyCon Sweden 2019: Test Fast, Fix More – Property based in Python testing with Hypothesis

By Alexander Hultnér

PyCon Sweden 2019: Test Fast, Fix More – Property based in Python testing with Hypothesis

Did you ever miss that corner case bug? Maybe it was a negative integer, strange timezone conversion behaviour, off by one error or something entirely else. These subtle bugs are often hard to catch and are easily missed in test cases. You like me have probably ran into plenty of code utilising only happy path testing, only to later discover subtle bugs which are easily fixed once pointed out. This is where property based testing comes into the picture. In this talk I will focus on a wonderful Python library called Hypothesis but the concepts apply to other languages as well. Hypethesis is based on the same concept as the famous QuickCheck library for Haskell, which in turn have been ported a large number of languages. Hypothesis uses a wide range of input to find edge cases that you could otherwise easily miss, once it finds these cases it narrows down the input to the minimal breaking example to provide failures which are easier to understand.

  • 2,277