Property Based Testing Intro

or Generative Testing

or Test Generation

What it is?

Instead of unit testing with individual example inputs eg.

"Sorting a list with values 3,1,2 should return 1,2,3.",

"Sorting a list with values 2,3,1 should return 1,2,3.",

"Sorting a list with values 1,3,2 should return 1,2,3."

...

Test the properties of a function eg.

"After sorting a list it should have same amount of elements.",

"And it should have the same elements.",

...

Coverage of testing, property-based testing and proof*

* Source: "Haskell the craft of functional programming" 3rd ed. by Simon Thompson

Advantages over example-based tests

• Property-based tests are more general, and thus are less brittle.
   
• Property-based tests provide a better and more concise description of requirements than a bunch of examples.
   
• As a consequence, one property-based test can replace many, many, example-based tests.
   
• By generating random input, property-based tests often reveal issues that you have overlooked, such as dealing with nulls, missing data, divide by zero, negative numbers, etc.
   
• Property-based tests force you to think.
   
• Property-based tests force you to have a clean design.

Test input generation in practice

1. Generate input for a unit or an integration test
   • ​Think more about properties of a functionality than isolated examples
     
     ​
2. Can be run multiple times when testing -> More coverage
   • Works best with unit tests
   • Integration tests might be too slow to be run hundreds of times in succession 

Categories of properties

• Different paths, same destination
  • Combining operations in different orders, but getting the same result. add 1, add 2 == add 2, add 1
• There and back again
  • addition/subtraction, write/read, setProperty/getProperty, toJSON/fromJSON
• Some things never change
  • Common invariants include size of a collection (after map), the contents of a collection (after sort), the height or depth of something in proportion to size (e.g. balanced trees).
• The more things change, the more they stay the same
  • These kinds of properties are based on “idempotence”. Eg. distinct, distinct...

Categories of properties, pt 2

• Solve a smaller problem first
  • These kinds of properties are based on “structural induction”. Induction properties are often naturally applicable to recursive structures such as lists and trees.
         In sort n <= n+1 and n+1 <= n+2...
• Hard to prove, easy to verify
  • Often an algorithm to find a result can be complicated, but verifying the answer is easy. For example, you might check that a string tokenizer works by just concatenating all the tokens again.
• The test oracle
  • An alternate version of an algorithm or process (a “test oracle”) that you can use to check your results. Fast algorithm vs. brute force. Map in memory vs. data in DB.

PBT in different languages

• Haskell https://hackage.haskell.org/package/QuickCheck
• Clojure https://github.com/clojure/test.check
• Java
  • https://github.com/pholser/junit-quickcheck Property-based testing, JUnit-style
  • https://github.com/ncredinburgh/QuickTheories Property based testing for Java 8
  • Some of these might require own separate test runner!
• JavaScript
  • https://github.com/jsverify/jsverify
  • https://www.npmjs.com/browse/keyword/quickcheck
• F# https://github.com/fscheck/FsCheck
• ... https://en.wikipedia.org/wiki/QuickCheck

Sources

• Scott Wlaschin
  • https://fsharpforfunandprofit.com/posts/property-based-testing/
  •  https://fsharpforfunandprofit.com/posts/property-based-testing-2/
• "Haskell the craft of functional programming" 3rd ed. Simon Thompson
• Clojure test check
  • https://github.com/clojure/test.check
• Testing Stateful and Concurrent Systems Using test.check Eric Normand
  • https://www.youtube.com/watch?v=r5i_OiZw6Sw (35 min)