Property-based testing
Magda Stożek
Let your testing library work for you
def calculateDiscount(customer: Customer, now: LocalDate): BigDecimal
Example: Calculating discount
Years:
0-1
1-2
2 +
0%
30%
10%
20%
birthday
case class Customer(
name: String,
joinedAt: LocalDate,
dateOfBirth: LocalDate
)
"Discount calculator" should "return 10% discount for a one year member" in {
val dateOfBirth = LocalDate.of(1980, 1, 20)
val customer = Customer("Sue Smith", NOW.minusYears(1), dateOfBirth)
val discount = calculator.calculateDiscount(customer, NOW)
discount should be(BigDecimal(0.1))
}
it should "return 20% discount for a two year member" in {
val dateOfBirth = LocalDate.of(1980, 1, 20)
val customer = Customer("Sue Smith", NOW.minusYears(2), dateOfBirth)
val discount = calculator.calculateDiscount(customer, NOW)
discount should be(BigDecimal(0.2))
}
Example-based tests
it should "return birthday discount" in {
val customer = Customer("Sue Smith", NOW.minusYears(1), NOW.withYear(1980))
val discount = calculator.calculateDiscount(customer, NOW)
discount should be(BigDecimal(0.3))
}
it should "return no discount for a new member" in {
val customer = Customer("Sue Smith", NOW, LocalDate.of(1980, 1, 20))
val discount = calculator.calculateDiscount(customer, NOW)
discount should be(BigDecimal(0))
}
it should "return 20% discount for a three year member" in {
val dateOfBirth = LocalDate.of(1980, 1, 20)
val customer = Customer("Sue Smith", NOW.minusYears(3), dateOfBirth)
val discount = calculator.calculateDiscount(customer, NOW)
discount should be(BigDecimal(0.2))
}
What about properties?
def calculateDiscount(customer: Customer, now: LocalDate): BigDecimal
Example: Calculating discount
Years:
0-1
1-2
2 +
0%
30%
10%
20%
birthday
case class Customer(
name: String,
joinedAt: LocalDate,
dateOfBirth: LocalDate
)
Libraries
- First:
- 1999: QuickCheck (Haskell) / Quviq Quickcheck (Erlang)
- Koen Claessen and John Hughes
- 1999: QuickCheck (Haskell) / Quviq Quickcheck (Erlang)
- Later:
-
C, C++, C#, Chicken, Clojure, Common Lisp, D, Elm, Elixir, Erlang, F#, Factor, Go, Io, Java, JavaScript, Julia, Kotlin, Logtalk, Lua, Node.js, Objective-C, OCaml, Perl, Prolog, PHP, Pony, Python, R, Racket, Ruby, Rust, Scala, Scheme, Smalltalk, Standard ML, Swift, Typescript, VB.NET, ...
-
What about Scala?
Also keep an eye on:
property("Discount should not be over 30 percent") {
forAll { (customer: Customer, now: LocalDate) =>
val discount = calculator.calculateDiscount(customer, now)
discount should be <= BigDecimal(0.3)
}
}
Property: discount is never larger than 30%
ScalaTestFailureLocation: pbt.DiscountCalculatorPropertySpec at (DiscountCalculatorPropertySpec.scala:28)
org.scalatest.exceptions.GeneratorDrivenPropertyCheckFailedException: DateTimeException was thrown during property evaluation.
Message: Invalid date 'February 29' as '2049' is not a leap year
Occurred when passed generated values (
arg0 = Customer(nWtCZLoAQ,2009-04-14,1988-02-29),
arg1 = Customer(bLNkLdYmCYCjVJlIDRWumkWZceQHyoulrCcietBLPyfBCqbyXqub,2015-12-19,1996-12-13),
arg2 = 2049-04-09
)
Discovered error
Generators
Default generators
Gen.choose(10, 99)
Gen.listOfN(5, Gen.alphaUpperChar)
Gen.uuid
Gen.option(Gen.oneOf("Home", "Work", "Other")
// (...)
Custom generators
val customerGen: Gen[Customer] = for {
name <- Gen.alphaStr
dateJoined <- dateFromRangeGen(START_OF_BUSINESS, LocalDate.now)
dateOfBirth <- dateFromRangeGen(OLDEST_DAY_OF_BIRTH, LocalDate.now.minusYears(18))
} yield Customer(name, dateJoined, dateOfBirth)
private def dateFromRangeGen(rangeStart: LocalDate, rangeEnd: LocalDate): Gen[LocalDate] = {
Gen.choose(rangeStart.toEpochDay, rangeEnd.toEpochDay).map(i => LocalDate.ofEpochDay(i))
}
property("Birthday discount should be highest") {
forAll { (customer: Customer, birthdayCustomer: Customer) =>
val now = birthdayCustomer.dateOfBirth.withYear(2021)
whenever(isNotBirthday(customer, now)) {
val ordinaryDiscount = calculator.calculateDiscount(customer, now)
val birthdayDiscount = calculator.calculateDiscount(birthdayCustomer, now)
birthdayDiscount.compare(ordinaryDiscount) should be >= 0
}
}
}
Assumptions
Challenges
Difficult questions earlier
Challenge #1
Coming up with properties
Challenge #2
Oracle
Alternate wording
Invariants
Symmetry
Nondeterminism
Challenge #3
Documenting
Challenge #4
Strategies
1. Examples first, properties later
2. Properties for key components
Summary
- Higher cost, higher value
- Edge cases and misconceptions
- Coming up with properties
- Balance
More
Questions?
property("End of presentation") {
forAll { (attendee: Attendee) =>
whenever(attendee.hasQuestions) {
attendee.questionsAsked should be > 0
}
}
}
https://bit.ly/magda-pbt-scala
Property-based testing [Scala, 45 min]
By Magda Stożek
Property-based testing [Scala, 45 min]
- 932