Property based testing

Automating the creation of test cases

by Daniel Bachler (@danyx23)

Kinds of test

Unit tests

Integration tests

End-to-End tests

Kinds of test

Manual tests


Automated test generation

Manual tests

Every example that is tested is created by a human

def add(a, b):
  return a + b
def test_add():
  assert add(1, 1) == 2

def test_add():
  assert add(2, 2) == 4

The state space

This code fragment has 100% code coverage of the add function

But only a single point in the possible state space is tested

def add(a, b):
  return a + b
def test_add():
  assert add(1, 1) == 2

Property based testing

Find properties that should be true about your code

With any arguments

Then create test data automatically and test these properties

In Python with Hypothesis

def add(a, b):
  return a + b

def test_adds_zero(a):
  assert add(a, 0) == a

What can be properties?

  • mathematical properties (commutativity, associativity, ...)
  • works correctly with it's inverse counterpart (encode/decode)
  • gives same result as a different (e.g. slower) implementation

RLE encoding/decoding



RLE encoding/decoding

def rle_encode(input):

def rle_decode(input):

def test_decode_inverts_encode(s):
    assert rle_decode(rle_encode(s)) == s


Works great for any kind of serialisation

Test against alternative implementation

def amazing_sort(items):
def test_is_sorted(items):
  amazing_sort(items) == sorted(items)

"Obvious" properties

def amazing_sort(items):
def is_sorted(items):
  return all(items[i] <= items[i+1] for i in range(len(items)-1))
def test_is_sorted(items):
  return is_sorted(amazing_sort(items))


Generate random values (many of them)

e.g. text(), integers()

Usually they try "tricky" values

"", "#\0äz象形😂", ...

Some languages can auto-generate Generators

{-# LANGUAGE DeriveGeneric #-}
import GHC.Generics
import Generic.Random.Generic

data MyType = MyType {
    foo :: Int
  , bar :: Bool
  , baz :: Float
  } deriving (Show, Generic)

generate (genericArbitrary :: Gen MyType)


Many PBT libraries offer shrinking

Find simpler test cases if a test fails

Instead of 3KB of text, tells you "象" makes the test fail

Generating from output -> input

e.g. for image classification

hard to write a random generator for pixels and know what  classification it should be

but easy to pick example "input" images for known labels

PBT can be used for End-to-End testing!

Web service that extracts tables from PDF into CSV


Generate PDFs from randomly generated CSV


extract_csv_from_pdf(csv_to_pdf(csv)) == csv

Further reading

Thank you!

follow me on Twitter: @danyx23


These slides:

Made with