Property-based testing

Magda Stożek

Let your testing library work for you

BigDecimal calculateDiscount(Customer customer, LocalDate now);

Example: Calculating discount

Years:

0-1

1-2

2 +

0%

30%

10%

20%

birthday

class Customer {
    private final String name;
    private final LocalDate joinedAt;
    private final LocalDate dateOfBirth;
// (...)
}
@Test
void shouldCalculate10percentDiscount() {
    LocalDate now = LocalDate.of(2019, 9, 15);
    LocalDate dateOfBirth = LocalDate.of(1980, 1, 20);
    Customer customer = new Customer("Sue Smith", now.minusYears(1), dateOfBirth);

    BigDecimal discount = calculator.calculateDiscount(customer, now);

    Assertions.assertThat(discount).isEqualTo(new BigDecimal("0.1"));
}

@Test
void shouldCalculate20percentDiscount() {
    LocalDate now = LocalDate.of(2019, 9, 15);
    LocalDate dateOfBirth = LocalDate.of(1980, 1, 20);
    Customer customer = new Customer("Sue Smith", now.minusYears(2), dateOfBirth);

    BigDecimal discount = calculator.calculateDiscount(customer, now);

    Assertions.assertThat(discount).isEqualTo(new BigDecimal("0.2"));
}

Example-based tests

@Test
void shouldCalculateBirthdayDiscount() {
    Customer customer = new Customer("Sue Smith", NOW, NOW.withYear(1980));

    BigDecimal discount = calculator.calculateDiscount(customer, NOW);

    Assertions.assertThat(discount).isEqualTo(new BigDecimal("0.3"));
}
@Test
void shouldCalculateNoDiscount() {
    Customer customer = new Customer("Sue Smith", NOW, LocalDate.of(1980, 1, 20));

    BigDecimal discount = calculator.calculateDiscount(customer, NOW);

    Assertions.assertThat(discount).isEqualTo(new BigDecimal("0"));
}

@Test
void shouldCalculate20percentDiscountFor3Years() {
    LocalDate now = LocalDate.of(2019, 9, 15);
    LocalDate dateOfBirth = LocalDate.of(1980, 1, 20);
    Customer customer = new Customer("Sue Smith", now.minusYears(3), dateOfBirth);

    BigDecimal discount = calculator.calculateDiscount(customer, now);

    Assertions.assertThat(discount).isEqualTo(new BigDecimal("0.2"));
}

What about properties?

BigDecimal calculateDiscount(Customer customer, LocalDate now);

Example: Calculating discount

Years:

0-1

1-2

2 +

0%

30%

10%

20%

birthday

class Customer {
    private final String name;
    private final LocalDate joinedAt;
    private final LocalDate dateOfBirth;
// (...)
}

Libraries

  • First:
  • Later:
    • C, C++, C#, Chicken, Clojure, Common Lisp, D, Elm, Elixir, Erlang, F#, Factor, Go, Io, Java, JavaScript, Julia, Kotlin, Logtalk, Lua, Node.js, Objective-C, OCaml, Perl, Prolog, PHP, Pony, Python, R, Racket, Ruby, Rust, Scala, Scheme, Smalltalk, Standard ML, Swift, Typescript, VB.NET, ...

What about Java?

@Property
boolean discountShouldNotBeOver30percent(@ForAll("validCustomer") Customer customer, @ForAll("futureDate") LocalDate now) {

    BigDecimal discount = calculator.calculateDiscount(customer, now);
    Assertions.assertThat(discount).isLessThan(new BigDecimal("0.3"));
}

Property: discount is never larger than 30%

                              |-------------------jqwik-------------------
tries = 374                   | # of calls to property
checks = 374                  | # of not rejected calls
generation-mode = RANDOMIZED  | parameters are randomly generated
after-failure = PREVIOUS_SEED | use the previous seed
seed = -272292185905571017    | random seed to reproduce generated values
sample = 
[Customer{name='AAA', joinedAt=2010-01-01, dateOfBirth=1980-02-29}, 2019-10-06]
original-sample = 
[Customer{name='DtQDpPfEx', joinedAt=2010-04-21, dateOfBirth=1980-02-29}, 2185-03-17]

Discovered error

Generators

@ForAll String sentence

@ForAll @StringLength(min = 3, max = 100) String sentence

@ForAll @AlphaChars @NumericChars @Chars({',', '.', ':', ';', '-'}) @Whitespace String sentence

Text

@ForAll Integer number

@ForAll @Positive Integer number

@ForAll @BigRange(min = "1", max = "99.99") @Scale(2) BigDecimal price

Numbers

@ForAll List<@Alpha String> words

@ForAll @Size(10) Set<@Negative Integer> numbers

Collections

@ForAll Optional<String> middleName

@ForAll @WithNull String middleName

Optional

@Property
boolean myProperty(@ForAll("shortStrings") String string, @ForAll("10 to 99") int number) {
    String concatenated = string + number;
    return concatenated.length() > 2 && concatenated.length() < 11;
}

@Provide
Arbitrary<String> shortStrings() {
    return Arbitraries.strings().withCharRange('a', 'z').filter('q')
        .ofMinLength(1).ofMaxLength(8);
}

@Provide("10 to 99")
Arbitrary<Integer> numbers() {
    return Arbitraries.integers().between(10, 99);
}

Custom generators

@Property
boolean discountShouldNotBeOver30percent(@ForAll("validCustomer") Customer customer) {
    BigDecimal discount = calculator.calculateDiscount(customer);
    Assertions.assertThat(discount).isLessThan(new BigDecimal("0.3"));
}

@Provide
Arbitrary<Customer> validCustomer() {
    Arbitrary<String> names = Arbitraries.strings().alpha().ofMinLength(3).ofMaxLength(21);
    Arbitrary<LocalDate> joined = datesGen(BUSINESS_START, LocalDate.now());
    Arbitrary<LocalDate> born = datesGen(MIN_DOB, LocalDate.now().minusYears(18));
    return Combinators.combine(names, joined, born).as(Customer::new);
}

Arbitrary<LocalDate> datesGen(LocalDate minDate, LocalDate maxDate) {
    long range = DAYS.between(minDate, maxDate);
    return Arbitraries.longs().between(0, range).map(n -> minDate.plus(n, DAYS));
}

Combining generators

@Property
boolean discountShouldBeProportionalToMembershipYears(@ForAll Customer older, @ForAll Customer newer) {
    Assume.that(isNotBirthday(older, now));
    Assume.that(isNotBirthday(newer, now));
    Assume.that(older.getJoinedAt().isBefore(newer.getJoinedAt()));

    BigDecimal discountForOlderCustomer = calculator.calculateDiscount(older);
    BigDecimal discountForNewerCustomer = calculator.calculateDiscount(newer);
    Assertions.assertThat(discountForOlderCustomer.isGreaterThan(discountForNewerCustomer);
}

Assumptions

@Property
void shouldCalculateDiscount(@ForAll("membershipYears") Short years) {
    BigDecimal discount = calculator.prepareDiscount(years);
    //(...)
}

@Provide
Arbitrary<Short> membershipYears() {
    return Arbitraries.frequency(
        Tuple.of(30, 0),
        Tuple.of(60, 1),
        Tuple.of(7, 2),
        Tuple.of(3, 3)
    );
}

Frequencies

Challenges

Difficult questions earlier

Challenge #1

Nondeterminism

Challenge #2

Documenting

Challenge #3

Coming up with properties

Challenge #4

Oracle

Alternate wording

Invariants

Symmetry

Strategies

1. Examples first, properties later

2. Properties for key components

Summary

  1. Higher cost, higher value
  2. Edge cases and misconceptions
  3. Coming up with properties
  4. Balance

More

Questions?

@Property
boolean endOfPresentation(@ForAll Attendee attendee) {
    Assume.that(attendee.hasQuestions());

    return attendee.questionsAsked > 0;
}

Property-based testing [Java, 45 min]

By Magda Stożek

Property-based testing [Java, 45 min]

  • 1,298