Property-based testing

Magda Stożek

Let your testing library work for you

def calculateDiscount(customer: Customer, now: LocalDate): BigDecimal

Example: Calculating discount

Years:

0-1

1-2

2 +

0%

30%

10%

20%

birthday

case class Customer(
    name: String, 
    joinedAt: LocalDate, 
    dateOfBirth: LocalDate
)
  "Discount calculator" should "return 10% discount for a one year member" in {
    val dateOfBirth = LocalDate.of(1980, 1, 20)
    val customer = Customer("Sue Smith", NOW.minusYears(1), dateOfBirth)

    val discount = calculator.calculateDiscount(customer, NOW)

    discount should be(BigDecimal(0.1))
  }

  it should "return 20% discount for a two year member" in {
    val dateOfBirth = LocalDate.of(1980, 1, 20)
    val customer = Customer("Sue Smith", NOW.minusYears(2), dateOfBirth)

    val discount = calculator.calculateDiscount(customer, NOW)

    discount should be(BigDecimal(0.2))
  }

Example-based tests

it should "return birthday discount" in {
  val customer = Customer("Sue Smith", NOW.minusYears(1), NOW.withYear(1980))

  val discount = calculator.calculateDiscount(customer, NOW)

  discount should be(BigDecimal(0.3))
}
it should "return no discount for a new member" in {
  val customer = Customer("Sue Smith", NOW, LocalDate.of(1980, 1, 20))

  val discount = calculator.calculateDiscount(customer, NOW)

  discount should be(BigDecimal(0))
}

it should "return 20% discount for a three year member" in {
  val dateOfBirth = LocalDate.of(1980, 1, 20)
  val customer = Customer("Sue Smith", NOW.minusYears(3), dateOfBirth)

  val discount = calculator.calculateDiscount(customer, NOW)

  discount should be(BigDecimal(0.2))
}

What about properties?

def calculateDiscount(customer: Customer, now: LocalDate): BigDecimal

Example: Calculating discount

Years:

0-1

1-2

2 +

0%

30%

10%

20%

birthday

case class Customer(
    name: String, 
    joinedAt: LocalDate, 
    dateOfBirth: LocalDate
)

Libraries

  • First:
  • Later:
    • C, C++, C#, Chicken, Clojure, Common Lisp, D, Elm, Elixir, Erlang, F#, Factor, Go, Io, Java, JavaScript, Julia, Kotlin, Logtalk, Lua, Node.js, Objective-C, OCaml, Perl, Prolog, PHP, Pony, Python, R, Racket, Ruby, Rust, Scala, Scheme, Smalltalk, Standard ML, Swift, Typescript, VB.NET, ...

What about Scala?

Also keep an eye on:

property("Discount should not be over 30 percent") {
  forAll { (customer: Customer, now: LocalDate) =>
    val discount = calculator.calculateDiscount(customer, now)
    discount should be <= BigDecimal(0.3)
  }
}

Property: discount is never larger than 30%

ScalaTestFailureLocation: pbt.DiscountCalculatorPropertySpec at (DiscountCalculatorPropertySpec.scala:28)
org.scalatest.exceptions.GeneratorDrivenPropertyCheckFailedException: DateTimeException was thrown during property evaluation.
  Message: Invalid date 'February 29' as '2049' is not a leap year
  Occurred when passed generated values (
    arg0 = Customer(nWtCZLoAQ,2009-04-14,1988-02-29),
    arg1 = Customer(bLNkLdYmCYCjVJlIDRWumkWZceQHyoulrCcietBLPyfBCqbyXqub,2015-12-19,1996-12-13),
    arg2 = 2049-04-09
  )

Discovered error

Generators

Default generators

Gen.choose(10, 99)

Gen.listOfN(5, Gen.alphaUpperChar)

Gen.uuid

Gen.option(Gen.oneOf("Home", "Work", "Other")

// (...)

Custom generators

val customerGen: Gen[Customer] = for {
    name <- Gen.alphaStr
    dateJoined <- dateFromRangeGen(START_OF_BUSINESS, LocalDate.now)
    dateOfBirth <- dateFromRangeGen(OLDEST_DAY_OF_BIRTH, LocalDate.now.minusYears(18))
  } yield Customer(name, dateJoined, dateOfBirth)
  
private def dateFromRangeGen(rangeStart: LocalDate, rangeEnd: LocalDate): Gen[LocalDate] = {
  Gen.choose(rangeStart.toEpochDay, rangeEnd.toEpochDay).map(i => LocalDate.ofEpochDay(i))
}
property("Birthday discount should be highest") {
  forAll { (customer: Customer, birthdayCustomer: Customer) =>
    val now = birthdayCustomer.dateOfBirth.withYear(2021)
      
    whenever(isNotBirthday(customer, now)) {
      val ordinaryDiscount = calculator.calculateDiscount(customer, now)
      val birthdayDiscount = calculator.calculateDiscount(birthdayCustomer, now)
      birthdayDiscount.compare(ordinaryDiscount) should be >= 0
    }
  }
}

Assumptions

Challenges

Difficult questions earlier

Challenge #1

Coming up with properties

Challenge #2

Oracle

Alternate wording

Invariants

Symmetry

Nondeterminism

Challenge #3

Documenting

Challenge #4

Strategies

1. Examples first, properties later

2. Properties for key components

Summary

  1. Higher cost, higher value
  2. Edge cases and misconceptions
  3. Coming up with properties
  4. Balance

More

Questions?

property("End of presentation") {
  forAll { (attendee: Attendee) =>
    whenever(attendee.hasQuestions) {
      attendee.questionsAsked should be > 0
    }
  }
}
http://bit.ly/magda-scalalove

https://bit.ly/magda-pbt-scala

Property-based testing [Scala, 45 min]

By Magda Stożek

Property-based testing [Scala, 45 min]

  • 937