# Test Faster, Fix More

Property based testing with Hypothesis

Alexander Hultnér @ PyCon SE 2019

@ahultner

## Alexander Hultnér

Founder of Hultnér Technologies

@ahultner

Test Faster, Fix More

## Outline

What is property based testing? Why?

Cheatsheet, two minute abstract

Hypothesis

Usage examples

Rule state machines

Interactive demo in Jupyter

Conclusion

@ahultner

Test Faster, Fix More

## What & Why

• Specific examples
• Repetition
• Leads to "happy path testing"
• Property based testing
• QuickCheck
• Test behaviour, find unexpected
• Generate thousands of examples
• Shrink errors
• Detect subtle bugs
• Compare when refactoring

@ahultner

Test Faster, Fix More

## Cheatsheet

@ahultner

Test Faster, Fix More

## Hypothesis

• Python Library
• http://hypothesis.works
• Tests for regressions
• Large library of "strategies"
• ​(dp)contracts
• Swagger conformance testing
• Pandas, Numpy, Django, etc
• Rules
• State machines, shows steps to reproduce a bug

@ahultner

Test Faster, Fix More

## Minimal usage

Generates positional arguments x, y

Strategies ensure integers are given

Strategies exists for a vast number of input types, they can be inferred and passed constraints via arguments.

``````from hypothesis import given
import hypothesis.strategies as st

@given(
st.integers(),
st.integers()
)
def test_ints_are_commutative(x, y):
assert x + y == y + x``````

@ahultner

Test Faster, Fix More

## Shrinking errors

When an error is found it will be shrunk

to smallest possible example

``````=== FAILURES ===

@given(st.integers(), st.integers())

_ _ _ _ _ _ _

a = -1, b = 1

@given(st.integers(), st.integers())
>       assert add(a, b) == a + b
E       assert -1 == 0
E         --1
E         +0

AssertionError
--- Hypothesis ---
``````# Buggy implementation of add
return a if a < 0 else a + b``````

@ahultner

Test Faster, Fix More

## Using @example

We can explicitly define input with @example.

Useful to keep track of known tricky cases

``````@given(st.integers(), st.integers())
@example(
-1, 1
)
assert add(a, b) == a + b``````

@ahultner

Test Faster, Fix More

## Python 2 to 3

• Migrating legacy code
• 2 –> 3
• Old "correct" application as oracle
• As in source of truth,
not the company
• Align new implementation with new

@ahultner

Test Faster, Fix More

``````…
@given(
st.integers(),
st.integers()
)
…``````

## Strategies

• Tuples, Lists
• Text, Characters
• Floats, Integers
• Datetimes, timezones
• Narrowing with max/min, etc
• API Fuzzing example in docs.
• First and third party extensions
• Inference through type annotations

@ahultner

Test Faster, Fix More

## Rules

With Hypothesis’s stateful testing, Hypothesis instead tries to generate not just data but entire tests.

Rule based state machines will allow you to define primitives from which tests are generated, gives you detailed steps taken to produce the bug, read more.

@ahultner

Test Faster, Fix More

## Demo time

I've prepared a Jupyter Notebook with a few interactive and more thorough examples.

@ahultner

Test Faster, Fix More

## Conclusion

• Write more extensive tests faster
• Find more bugs
• Not a silver bullet for all testing
but a powerful tool in your belt
• Think about defining properties not specific examples
• Start playing around with property-based testing!

@ahultner

Test Faster, Fix More

## Questions

Contact me if you have any further

questions.

Available for training, workshops and

consulting.

@ahultner

Test Faster, Fix More

#### PyCon Sweden 2019: Test Fast, Fix More – Property based in Python testing with Hypothesis

By Alexander Hultnér

# PyCon Sweden 2019: Test Fast, Fix More – Property based in Python testing with Hypothesis

Did you ever miss that corner case bug? Maybe it was a negative integer, strange timezone conversion behaviour, off by one error or something entirely else. These subtle bugs are often hard to catch and are easily missed in test cases. You like me have probably ran into plenty of code utilising only happy path testing, only to later discover subtle bugs which are easily fixed once pointed out. This is where property based testing comes into the picture. In this talk I will focus on a wonderful Python library called Hypothesis but the concepts apply to other languages as well. Hypethesis is based on the same concept as the famous QuickCheck library for Haskell, which in turn have been ported a large number of languages. Hypothesis uses a wide range of input to find edge cases that you could otherwise easily miss, once it finds these cases it narrows down the input to the minimal breaking example to provide failures which are easier to understand.

• 590