Property-based testing
Magda Stożek
Let your testing library work for you
BigDecimal calculateDiscount(Customer customer, LocalDate now);
Example: Calculating discount
Years:
0-1
1-2
2 +
0%
30%
10%
20%
birthday
class Customer {
private final String name;
private final LocalDate joinedAt;
private final LocalDate dateOfBirth;
// (...)
}
@Test
void shouldCalculate10percentDiscount() {
LocalDate now = LocalDate.of(2019, 9, 15);
LocalDate dateOfBirth = LocalDate.of(1980, 1, 20);
Customer customer = new Customer("Sue Smith", now.minusYears(1), dateOfBirth);
BigDecimal discount = calculator.calculateDiscount(customer, now);
Assertions.assertThat(discount).isEqualTo(new BigDecimal("0.1"));
}
@Test
void shouldCalculate20percentDiscount() {
LocalDate now = LocalDate.of(2019, 9, 15);
LocalDate dateOfBirth = LocalDate.of(1980, 1, 20);
Customer customer = new Customer("Sue Smith", now.minusYears(2), dateOfBirth);
BigDecimal discount = calculator.calculateDiscount(customer, now);
Assertions.assertThat(discount).isEqualTo(new BigDecimal("0.2"));
}
Example-based tests
@Test
void shouldCalculateBirthdayDiscount() {
Customer customer = new Customer("Sue Smith", NOW, NOW.withYear(1980));
BigDecimal discount = calculator.calculateDiscount(customer, NOW);
Assertions.assertThat(discount).isEqualTo(new BigDecimal("0.3"));
}
@Test
void shouldCalculateNoDiscount() {
Customer customer = new Customer("Sue Smith", NOW, LocalDate.of(1980, 1, 20));
BigDecimal discount = calculator.calculateDiscount(customer, NOW);
Assertions.assertThat(discount).isEqualTo(new BigDecimal("0"));
}
@Test
void shouldCalculate20percentDiscountFor3Years() {
LocalDate now = LocalDate.of(2019, 9, 15);
LocalDate dateOfBirth = LocalDate.of(1980, 1, 20);
Customer customer = new Customer("Sue Smith", now.minusYears(3), dateOfBirth);
BigDecimal discount = calculator.calculateDiscount(customer, now);
Assertions.assertThat(discount).isEqualTo(new BigDecimal("0.2"));
}
What about properties?
BigDecimal calculateDiscount(Customer customer, LocalDate now);
Example: Calculating discount
Years:
0-1
1-2
2 +
0%
30%
10%
20%
birthday
class Customer {
private final String name;
private final LocalDate joinedAt;
private final LocalDate dateOfBirth;
// (...)
}
Libraries
- First:
- 1999: QuickCheck (Haskell) / Quviq Quickcheck (Erlang)
- Koen Claessen and John Hughes
- 1999: QuickCheck (Haskell) / Quviq Quickcheck (Erlang)
- Later:
-
C, C++, C#, Chicken, Clojure, Common Lisp, D, Elm, Elixir, Erlang, F#, Factor, Go, Io, Java, JavaScript, Julia, Kotlin, Logtalk, Lua, Node.js, Objective-C, OCaml, Perl, Prolog, PHP, Pony, Python, R, Racket, Ruby, Rust, Scala, Scheme, Smalltalk, Standard ML, Swift, Typescript, VB.NET, ...
-
What about Java?
- junit-quickcheck (2013, JUnit 4)
-
QuickTheories (2015, framework-independent, DSL)
-
jqwik (2017, JUnit 5)
@Property
boolean discountShouldNotBeOver30percent(@ForAll("validCustomer") Customer customer, @ForAll("futureDate") LocalDate now) {
BigDecimal discount = calculator.calculateDiscount(customer, now);
Assertions.assertThat(discount).isLessThan(new BigDecimal("0.3"));
}
Property: discount is never larger than 30%
|-------------------jqwik-------------------
tries = 374 | # of calls to property
checks = 374 | # of not rejected calls
generation-mode = RANDOMIZED | parameters are randomly generated
after-failure = PREVIOUS_SEED | use the previous seed
seed = -272292185905571017 | random seed to reproduce generated values
sample =
[Customer{name='AAA', joinedAt=2010-01-01, dateOfBirth=1980-02-29}, 2019-10-06]
original-sample =
[Customer{name='DtQDpPfEx', joinedAt=2010-04-21, dateOfBirth=1980-02-29}, 2185-03-17]
Discovered error
Generators
@ForAll String sentence
@ForAll @StringLength(min = 3, max = 100) String sentence
@ForAll @AlphaChars @NumericChars @Chars({',', '.', ':', ';', '-'}) @Whitespace String sentence
Text
@ForAll Integer number
@ForAll @Positive Integer number
@ForAll @BigRange(min = "1", max = "99.99") @Scale(2) BigDecimal price
Numbers
@ForAll List<@Alpha String> words
@ForAll @Size(10) Set<@Negative Integer> numbers
Collections
@ForAll Optional<String> middleName
@ForAll @WithNull String middleName
Optional
@Property
boolean myProperty(@ForAll("shortStrings") String string, @ForAll("10 to 99") int number) {
String concatenated = string + number;
return concatenated.length() > 2 && concatenated.length() < 11;
}
@Provide
Arbitrary<String> shortStrings() {
return Arbitraries.strings().withCharRange('a', 'z').filter('q')
.ofMinLength(1).ofMaxLength(8);
}
@Provide("10 to 99")
Arbitrary<Integer> numbers() {
return Arbitraries.integers().between(10, 99);
}
Custom generators
@Property
boolean discountShouldNotBeOver30percent(@ForAll("validCustomer") Customer customer) {
BigDecimal discount = calculator.calculateDiscount(customer);
Assertions.assertThat(discount).isLessThan(new BigDecimal("0.3"));
}
@Provide
Arbitrary<Customer> validCustomer() {
Arbitrary<String> names = Arbitraries.strings().alpha().ofMinLength(3).ofMaxLength(21);
Arbitrary<LocalDate> joined = datesGen(BUSINESS_START, LocalDate.now());
Arbitrary<LocalDate> born = datesGen(MIN_DOB, LocalDate.now().minusYears(18));
return Combinators.combine(names, joined, born).as(Customer::new);
}
Arbitrary<LocalDate> datesGen(LocalDate minDate, LocalDate maxDate) {
long range = DAYS.between(minDate, maxDate);
return Arbitraries.longs().between(0, range).map(n -> minDate.plus(n, DAYS));
}
Combining generators
@Property
boolean discountShouldBeProportionalToMembershipYears(@ForAll Customer older, @ForAll Customer newer) {
Assume.that(isNotBirthday(older, now));
Assume.that(isNotBirthday(newer, now));
Assume.that(older.getJoinedAt().isBefore(newer.getJoinedAt()));
BigDecimal discountForOlderCustomer = calculator.calculateDiscount(older);
BigDecimal discountForNewerCustomer = calculator.calculateDiscount(newer);
Assertions.assertThat(discountForOlderCustomer.isGreaterThan(discountForNewerCustomer);
}
Assumptions
@Property
void shouldCalculateDiscount(@ForAll("membershipYears") Short years) {
BigDecimal discount = calculator.prepareDiscount(years);
//(...)
}
@Provide
Arbitrary<Short> membershipYears() {
return Arbitraries.frequency(
Tuple.of(30, 0),
Tuple.of(60, 1),
Tuple.of(7, 2),
Tuple.of(3, 3)
);
}
Frequencies
Challenges
Difficult questions earlier
Challenge #1
Nondeterminism
Challenge #2
Documenting
Challenge #3
Coming up with properties
Challenge #4
Oracle
Alternate wording
Invariants
Symmetry
Strategies
1. Examples first, properties later
2. Properties for key components
Summary
- Higher cost, higher value
- Edge cases and misconceptions
- Coming up with properties
- Balance
More
Questions?
@Property
boolean endOfPresentation(@ForAll Attendee attendee) {
Assume.that(attendee.hasQuestions());
return attendee.questionsAsked > 0;
}
Property-based testing [Java, 45 min]
By Magda Stożek
Property-based testing [Java, 45 min]
- 1,289