Property-based testing

Magda Stożek

@magdastozek

Let your testing library work for you

BigDecimal calculateDiscount(Customer customer, LocalDate now);

Example: Calculating discount

Years:

0-1

1-2

2 +

0%

30%

10%

20%

birthday

class Customer {
    private final String name;
    private final LocalDate joinedAt;
    private final LocalDate dateOfBirth;
// (...)
}

Example-based tests

@Test
void shouldCalculateNoDiscount() {...}
@Test
void shouldCalculate10percentDiscount() {...}
@Test
void shouldCalculate20percentDiscount() {...}
@Test
void shouldCalculateBirthdayDiscount() {...}
@Test
void shouldCalculate20percentDiscountFor3Years() {...}

Happy path

Edge cases

What about properties?

BigDecimal calculateDiscount(Customer customer, LocalDate now);

Example: Calculating discount

Years:

0-1

1-2

2 +

0%

30%

10%

20%

birthday

class Customer {
    private final String name;
    private final LocalDate joinedAt;
    private final LocalDate dateOfBirth;
// (...)
}

Property-based tests

@Property
void shouldNotBeLowerThan0() {...}
@Property
void shouldNotBeOver30percent() {...}
@Property
void shouldBeProportionalToMembershipYears() {...}

Properties:

@Property
void shouldNotBeGreaterThanBirthdayDiscount() {...}

Libraries

  • First:
  • Later:

    • C, C++, C#, Chicken, Clojure, Common Lisp, D, Elm, Elixir, Erlang, F#, Factor, Go, Io, Java, JavaScript, Julia, Kotlin, Logtalk, Lua, Node.js, Objective-C, OCaml, Perl, Prolog, PHP, Pony, Python, R, Racket, Ruby, Rust, Scala, Scheme, Smalltalk, Standard ML, Swift, Typescript, VB.NET, ...

What about Java?

Property: discount is never larger than 30%

@Property
boolean shouldNotBeOver30percent(
    @ForAll Customer customer, 
    @ForAll("futureDate") LocalDate now) 
{
    BigDecimal discount = calculator.calculateDiscount(customer, now);
    Assertions.assertThat(discount).compareTo(new BigDecimal("0.3")) <= 0;
}
                              |-------------------jqwik-------------------
tries = 374                   | # of calls to property
checks = 374                  | # of not rejected calls
generation-mode = RANDOMIZED  | parameters are randomly generated
after-failure = PREVIOUS_SEED | use the previous seed
seed = -272292185905571017    | random seed to reproduce generated values
sample = 
[Customer{name='AAA', joinedAt=2010-01-01, dateOfBirth=1980-02-29}, 2019-10-06]
original-sample = 
[Customer{name='DtQDpPfEx', joinedAt=2010-04-21, dateOfBirth=1980-02-29}, 2185-03-17]

Discovered error

Generators

@Property
boolean shouldBeLongerThanTwo(@ForAll("shortStrings") String string, @ForAll("10 to 99") int number) {
    return (string + number).length() > 2;
}

@Provide
Arbitrary<String> shortStrings() {
    return Arbitraries.strings().withCharRange('a', 'z').filter('q')
        .ofMinLength(1).ofMaxLength(8);
}

@Provide("10 to 99")
Arbitrary<Integer> numbers() {
    return Arbitraries.integers().between(10, 99);
}

Custom generators

Challenges

Difficult questions earlier

Challenge #1

Nondeterminism

Challenge #2

Documenting

Challenge #3

Coming up with properties

Challenge #4

fsharpforfunandprofit.com/posts/property-based-testing-2/

Strategies

1. Examples first, properties later

2. Properties for key components

Summary

  1. Higher cost, higher value
  2. Edge cases and misconceptions
  3. Coming up with properties
  4. Balance

More

Questions?

@Property
boolean endOfPresentation(@ForAll Attendee attendee) {
    Assume.that(attendee.hasQuestions());

    return attendee.questionsAsked > 0;
}

@magdastozek