Property based testing

Automating the creation of test cases

by Daniel Bachler (@danyx23)

https://slides.com/danielbachler/pbt

Kinds of test

Unit tests

Integration tests

End-to-End tests

Kinds of test

Manual tests

 

Automated test generation

Manual tests

Every example that is tested is created by a human

def add(a, b):
  return a + b
    
def test_add():
  assert add(1, 1) == 2

def test_add():
  assert add(2, 2) == 4

The state space

This code fragment has 100% code coverage of the add function

But only a single point in the possible state space is tested

def add(a, b):
  return a + b
    
def test_add():
  assert add(1, 1) == 2

Property based testing

Find properties that should be true about your code

With any arguments

Then create test data automatically and test these properties

In Python with Hypothesis

def add(a, b):
  return a + b

@given(integers())
def test_adds_zero(a):
  assert add(a, 0) == a

What can be properties?

  • mathematical properties (commutativity, associativity, ...)
  • works correctly with it's inverse counterpart (encode/decode)
  • gives same result as a different (e.g. slower) implementation

RLE encoding/decoding

WWWWBBBA

4W3BA

RLE encoding/decoding

def rle_encode(input):
	...

def rle_decode(input):
	...


@given(text())
def test_decode_inverts_encode(s):
    assert rle_decode(rle_encode(s)) == s

Encoding/Decoding

Works great for any kind of serialisation

Test against alternative implementation

def amazing_sort(items):
  ...
 
@given(lists(integers()))
def test_is_sorted(items):
  amazing_sort(items) == sorted(items)

"Obvious" properties

def amazing_sort(items):
  ...
 
def is_sorted(items):
  return all(items[i] <= items[i+1] for i in range(len(items)-1))
 
@given(lists(integers()))
def test_is_sorted(items):
  return is_sorted(amazing_sort(items))

Generators

Generate random values (many of them)

e.g. text(), integers()

Usually they try "tricky" values

"", "#\0äz象形😂", ...

Some languages can auto-generate Generators

{-# LANGUAGE DeriveGeneric #-}
import GHC.Generics
import Generic.Random.Generic

data MyType = MyType {
    foo :: Int
  , bar :: Bool
  , baz :: Float
  } deriving (Show, Generic)

generate (genericArbitrary :: Gen MyType)

Shrinking

Many PBT libraries offer shrinking

Find simpler test cases if a test fails

Instead of 3KB of text, tells you "象" makes the test fail

Generating from output -> input

e.g. for image classification

hard to write a random generator for pixels and know what  classification it should be

but easy to pick example "input" images for known labels

PBT can be used for End-to-End testing!

Web service that extracts tables from PDF into CSV

 

Generate PDFs from randomly generated CSV

 

extract_csv_from_pdf(csv_to_pdf(csv)) == csv

Further reading

Oskar Wickströms book and blog

https://wickstrom.tech

FSharp For Fun and Profit

https://fsharpforfunandprofit.com/posts/property-based-testing/

Hillel Waynes blog

e.g. https://www.hillelwayne.com/post/hypothesis-oracles/

Joe "begriffs" Nelson

https://begriffs.com/posts/2017-01-14-design-use-quickcheck.html

Thank you!

follow me on Twitter: @danyx23

 

These slides:

https://slides.com/danielbachler/pbt

Made with Slides.com