Property based testing
Automating the creation of test cases
by Daniel Bachler (@danyx23)
https://slides.com/danielbachler/pbt
Kinds of test
Unit tests
Integration tests
End-to-End tests
Kinds of test
Manual tests
Automated test generation
Manual tests
Every example that is tested is created by a human
def add(a, b):
return a + b
def test_add():
assert add(1, 1) == 2
def test_add():
assert add(2, 2) == 4
The state space
This code fragment has 100% code coverage of the add function
But only a single point in the possible state space is tested
def add(a, b):
return a + b
def test_add():
assert add(1, 1) == 2
Property based testing
Find properties that should be true about your code
With any arguments
Then create test data automatically and test these properties
In Python with Hypothesis
def add(a, b):
return a + b
@given(integers())
def test_adds_zero(a):
assert add(a, 0) == a
What can be properties?
- mathematical properties (commutativity, associativity, ...)
- works correctly with it's inverse counterpart (encode/decode)
- gives same result as a different (e.g. slower) implementation
RLE encoding/decoding
WWWWBBBA
4W3BA
RLE encoding/decoding
def rle_encode(input):
...
def rle_decode(input):
...
@given(text())
def test_decode_inverts_encode(s):
assert rle_decode(rle_encode(s)) == s
Encoding/Decoding
Works great for any kind of serialisation
Test against alternative implementation
def amazing_sort(items):
...
@given(lists(integers()))
def test_is_sorted(items):
amazing_sort(items) == sorted(items)
"Obvious" properties
def amazing_sort(items):
...
def is_sorted(items):
return all(items[i] <= items[i+1] for i in range(len(items)-1))
@given(lists(integers()))
def test_is_sorted(items):
return is_sorted(amazing_sort(items))
Generators
Generate random values (many of them)
e.g. text(), integers()
Usually they try "tricky" values
"", "#\0äz象形😂", ...
Some languages can auto-generate Generators
{-# LANGUAGE DeriveGeneric #-}
import GHC.Generics
import Generic.Random.Generic
data MyType = MyType {
foo :: Int
, bar :: Bool
, baz :: Float
} deriving (Show, Generic)
generate (genericArbitrary :: Gen MyType)
Shrinking
Many PBT libraries offer shrinking
Find simpler test cases if a test fails
Instead of 3KB of text, tells you "象" makes the test fail
Generating from output -> input
e.g. for image classification
hard to write a random generator for pixels and know what classification it should be
but easy to pick example "input" images for known labels
PBT can be used for End-to-End testing!
Web service that extracts tables from PDF into CSV
Generate PDFs from randomly generated CSV
extract_csv_from_pdf(csv_to_pdf(csv)) == csv
Further reading
Oskar Wickströms book and blog
FSharp For Fun and Profit
https://fsharpforfunandprofit.com/posts/property-based-testing/
Hillel Waynes blog
e.g. https://www.hillelwayne.com/post/hypothesis-oracles/
Joe "begriffs" Nelson
https://begriffs.com/posts/2017-01-14-design-use-quickcheck.html
Thank you!
follow me on Twitter: @danyx23
These slides:
https://slides.com/danielbachler/pbt
Property based Testing
By Daniel Bachler
Property based Testing
- 1,157