Property Based Testing in Scala

with

ScalaCheck

Marc Saegesser (@marcsaegesser)

AMI Entertainment Network, Inc.

Agenda

  • What is property based testing
  • Properties
  • Generating test data
  • Running tests
  • Designing properties
  • Example
  • Testing stateful code
  • Commands example

What is Property Based Testing?

Properties

  • A  Specification of the software

  • ​Facts about the software that are always true

Automatic test generation

  • "Don't write tests, generate them" -- John Hughes

  • Evaluate properties with  intelligently random input

  • Attempt to invalidate properties

Failure minimalization

  • Find simplest input that invalidates a property
  • Simplify input until a failing property succeeds

A specification of Math.max:  

Given two integers return the larger of the two.

In ScalaCheck this can be written as

import org.scalacheck.Prop.forAll

forAll { (x: Int, y: Int) =>
  val z = Math.max(x, y)
  (z == x || z == y) && (z >= x && z >= y)
}

Source:   ScalaCheck: The Definitive Guide

Properties in ScalaCheck

Generating Data

Generators (org.scalacheck.Gen) create test data.

ScalaCheck provides dozens of built-in generators and combinators.

This property asserts that the sum of two positive numbers is also positive.

forAll(Gen.posNum[Int], Gen.posNum[Int]) { (x, y) => 
  x + y >= 0
}

Question:  Does this property hold?

Built-in Generators

def alphaChar: Gen[Char]
def alphaLowerChar: Gen[Char]
def alphaNumChar: Gen[Char]
def alphaStr: Gen[String]
def alphaUpperChar: Gen[Char]
def choose[T](min: T, max: T)(implicit c: Choose[T]): Gen[T]
def const[T](x: T): Gen[T]
def identifier: Gen[String]
def listOf[T](g: ⇒ Gen[T]): Gen[List[T]]
def listOfN[T](n: Int, g: Gen[T]): Gen[List[T]]
def negNum[T](implicit num: Numeric[T], c: Choose[T]): Gen[T]
def nonEmptyListOf[T](g: ⇒ Gen[T]): Gen[List[T]]
def numChar: Gen[Char]
def numStr: Gen[String]
def oneOf[T](g0: Gen[T], g1: Gen[T], gn: Gen[T]*): Gen[T]
def option[T](g: Gen[T]): Gen[Option[T]]
def posNum[T](implicit num: Numeric[T], c: Choose[T]): Gen[T]
def someOf[T](g1: Gen[T], g2: Gen[T], gs: Gen[T]*): Gen[Seq[T]]
lazy val uuid: Gen[UUID]

A sample of some provided Generators

Custom Generators

Consider the following classes

case class Person(hon: String, first: String, last: String, age: Int, phone: PhoneNumber)

case class PhoneNumber(npa: String, nxx: String, xxxx: String) {
  require(npa.length  == 3 && !npa.exists(!_.isDigit) && npa(0) != '0', "Invalid npa")
  require(nxx.length  == 3 && !npa.exists(!_.isDigit) && nxx(0) != '0', "Invalid nxx")
  require(xxxx.length == 4 && !npa.exists(!_.isDigit), "Invalid xxxx")

  override def toString = s"($npa)$nxx-$xxxx"
}

Custom Generators

def genDigitsNoLeadingZero(n: Int): Gen[String] = 
  Gen.listOfN(n, Gen.numChar) suchThat (_(0) != '0') map (_.mkString)

def genDigits(n: Int): Gen[String] = 
  Gen.listOfN(n, Gen.numChar) map(_.mkString)

val genPhoneNumber: Gen[PhoneNumber] =
  for {
    npa <- genDigitsNoLeadingZero(3)
    nxx <- genDigitsNoLeadingZero(3)
    xxxx <- genDigits(4)
  } yield PhoneNumber(npa, nxx, xxxx)

val genPerson: Gen[Person] =
  for {
    h <- Gen.oneOf("Mr.", "Mrs.", "Ms.", "Master", "Miss")
    f <- arbitrary[String]
    l <- arbitrary[String]
    a <- Gen.choose(1, 100)
    p <- genPhoneNumber
  } yield Person(h, f, l, a, p)

Custom Generators

scala> genPerson.sample
res69: Option[Person] = Some(Person(Master,룈鉮죒཮喇땐亩뽾ꎐﯸ湇⭩㈦䌑퀥촕쭫쾀꺋
뺋隷䓯㠩顡農ᦚ碶蘭螝ꇂꎃ塖쥠袙ሖ࢛皷ᑁ⛣ㄤ䏿眄苸ᬔ낶ቕ纭般⻁ᕥ⊲ﵷᜉ縞犨읮餹ꦒ蔊,坜噶ฝ굺੆鎮
汃,66,(820)741-1472))

scala> genPerson.sample
res70: Option[Person] = Some(Person(Miss,⛃桕ꬕ戹䑹娕摒ን뎱♗ঘ虘됻癮뮎띘֥鰄질,,
78,(722)586-1474))

scala> genPerson.sample
res71: Option[Person] = Some(Person(Master,莹홆꼽㋎ੁ᐀諬하贯凧⢙蠪彵쨋嘆⡒ౌ䤮
雟秱ﺥ뽪쁝ﺭ輠ㆭ력唅폃洲ĭ㠥ๆ햑⺑ᔧꊉƹ쯇륲ꈭ,,68,(195)132-3212))

scala> genPerson.sample
res72: Option[Person] = None

You can test generators from the REPL

Running Tests

object MathSpec extends Properties("Math") {
  property("max") = forAll { (x: Int, y: Int) =>
    val z = Math.max(x, y)
    (z == x || z == y) && (z >= x && z >= y)
  }
  property("Positive plus Positive is Positive") = 
    forAll(Gen.posNum[Int], Gen.posNum[Int]) { (x, y) =>
      x + y >= 0
    }
  property("Pos + Pos is Pos") = 
    forAll { (x: Int, y: Int) =>
      (x >= 0 && y >= 0) ==> (x + y) >= 0
    }
  property("Pos(BigInt) + Pos(BigInt) is Positive") = 
    forAll { (x: BigInt, y: BigInt) =>
      (x >= 0 && y >= 0) ==> (x + y) >= 0
    }
}

> test-only *MathSpec
[info] + Math.Positive plus Positive is Positive: OK, passed 100 tests.
[info] + Math.max: OK, passed 100 tests.
[info] + Math.Pos(BigInt) + Pos(BigInt) is Positive: OK, passed 100 tests.
[info] ! Math.Pos + Pos is Pos: Falsified after 2 passed tests.
[info] > ARG_0: 194249254
[info] > ARG_0_ORIGINAL: 401139303
[info] > ARG_1: 2147483647
[info] Failed: Total 4, Failed 1, Errors 0, Passed 3
[error] Failed tests:
[error] 	demo.MathSpec
[error] (test:testOnly) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 1 s, completed Feb 14, 2016 9:31:02 PM

Designing Properties

Start simple

  • Look for situations that should never occur
  • Avoid duplicating application code in properties
  • Favor more simple properties over fewer complex ones

Comparison tests

  • When there is a relation between inputs and outputs 
  • Compare multiple outputs with each other rather than checking exact output values

Round-trip tests

  • Use when you have inverse relationships
  • Very useful for encoder/decoders, parsers, ...

Designing Properties

Be creative

  • Think deeply about what you are testing
  • Test expected failures not just expect success
  • Performance properties

Start slowly

  • Use scenario based unit testing along side property based tests
  • Start by adding a few properties for code already being tested by unit tests

Model based

  • Model your system with a simple, known correct data structure
  • For example, a database might be modeled by a Map

Example:  Scala-DBus

object DBusMarshalSpecification extends Properties("Marshal") {
  import Gen._, DBus._, Arbitrary.arbitrary

  val genFieldBoolean: Gen[FieldBoolean] = arbitrary[Boolean] map (FieldBoolean)
  val genFieldWord8:   Gen[FieldWord8]   = arbitrary[Byte]    map (FieldWord8)
  val genFieldInt16:   Gen[FieldInt16]   = arbitrary[Short]   map (FieldInt16)
  val genFieldInt32:   Gen[FieldInt32]   = arbitrary[Int]     map (FieldInt32)
  val genFieldString:  Gen[FieldString]  = arbitrary[String]  map (FieldString)


























}

Example:  Scala-DBus

object DBusMarshalSpecification extends Properties("Marshal") {
  import Gen._, DBus._, Arbitrary.arbitrary

  val genFieldBoolean: Gen[FieldBoolean] = arbitrary[Boolean] map (FieldBoolean)
  val genFieldWord8:   Gen[FieldWord8]   = arbitrary[Byte]    map (FieldWord8)
  val genFieldInt16:   Gen[FieldInt16]   = arbitrary[Short]   map (FieldInt16)
  val genFieldInt32:   Gen[FieldInt32]   = arbitrary[Int]     map (FieldInt32)
  val genFieldString:  Gen[FieldString]  = arbitrary[String]  map (FieldString)

  val alphaNumStr = nonEmptyListOf(alphaNumChar) map (_.mkString)
  val genObjectPath = listOf(alphaNumStr) map (_.mkString("/", "/", "").toObjectPath_)
  val genFieldObjectPath = genObjectPath map (FieldObjectPath)






















}

Example:  Scala-DBus

object DBusMarshalSpecification extends Properties("Marshal") {
  import Gen._, DBus._, Arbitrary.arbitrary

  val genFieldBoolean: Gen[FieldBoolean] = arbitrary[Boolean] map (FieldBoolean)
  val genFieldWord8:   Gen[FieldWord8]   = arbitrary[Byte]    map (FieldWord8)
  val genFieldInt16:   Gen[FieldInt16]   = arbitrary[Short]   map (FieldInt16)
  val genFieldInt32:   Gen[FieldInt32]   = arbitrary[Int]     map (FieldInt32)
  val genFieldString:  Gen[FieldString]  = arbitrary[String]  map (FieldString)

  val alphaNumStr = nonEmptyListOf(alphaNumChar) map (_.mkString)
  val genObjectPath = listOf(alphaNumStr) map (_.mkString("/", "/", "").toObjectPath_)
  val genFieldObjectPath = genObjectPath map (FieldObjectPath)

  val genAtomic = oneOf("b", "y", "q", "u", "t", "n", "i", "x", "d", "h", "s", "g", "o")
  val genSig = nonEmptyListOf(genAtomic) suchThat (_.length <= 255) map (_.mkString.toSignature_)
  val genFieldSignature: Gen[FieldSignature] = genSig map (FieldSignature)


















}

Example:  Scala-DBus

object DBusMarshalSpecification extends Properties("Marshal") {
  import Gen._, DBus._, Arbitrary.arbitrary

  val genFieldBoolean: Gen[FieldBoolean] = arbitrary[Boolean] map (FieldBoolean)
  val genFieldWord8:   Gen[FieldWord8]   = arbitrary[Byte]    map (FieldWord8)
  val genFieldInt16:   Gen[FieldInt16]   = arbitrary[Short]   map (FieldInt16)
  val genFieldInt32:   Gen[FieldInt32]   = arbitrary[Int]     map (FieldInt32)
  val genFieldString:  Gen[FieldString]  = arbitrary[String]  map (FieldString)

  val alphaNumStr = nonEmptyListOf(alphaNumChar) map (_.mkString)
  val genObjectPath = listOf(alphaNumStr) map (_.mkString("/", "/", "").toObjectPath_)
  val genFieldObjectPath = genObjectPath map (FieldObjectPath)

  val genAtomic = oneOf("b", "y", "q", "u", "t", "n", "i", "x", "d", "h", "s", "g", "o")
  val genSig = nonEmptyListOf(genAtomic) suchThat (_.length <= 255) map (_.mkString.toSignature_)
  val genFieldSignature: Gen[FieldSignature] = genSig map (FieldSignature)

  val genField: Gen[Field] = frequency((10, genFieldBoolean),(10, genFieldWord8),(10, genFieldInt16),
    (10, genFieldInt32), (10, genFieldString), (10, genFieldObjectPath), (10, genFieldSignature), 
    (10, genVariant), (1,  genStructure)
  )













}

Example:  Scala-DBus

object DBusMarshalSpecification extends Properties("Marshal") {
  import Gen._, DBus._, Arbitrary.arbitrary

  val genFieldBoolean: Gen[FieldBoolean] = arbitrary[Boolean] map (FieldBoolean)
  val genFieldWord8:   Gen[FieldWord8]   = arbitrary[Byte]    map (FieldWord8)
  val genFieldInt16:   Gen[FieldInt16]   = arbitrary[Short]   map (FieldInt16)
  val genFieldInt32:   Gen[FieldInt32]   = arbitrary[Int]     map (FieldInt32)
  val genFieldString:  Gen[FieldString]  = arbitrary[String]  map (FieldString)

  val alphaNumStr = nonEmptyListOf(alphaNumChar) map (_.mkString)
  val genObjectPath = listOf(alphaNumStr) map (_.mkString("/", "/", "").toObjectPath_)
  val genFieldObjectPath = genObjectPath map (FieldObjectPath)

  val genAtomic = oneOf("b", "y", "q", "u", "t", "n", "i", "x", "d", "h", "s", "g", "o")
  val genSig = nonEmptyListOf(genAtomic) suchThat (_.length <= 255) map (_.mkString.toSignature_)
  val genFieldSignature: Gen[FieldSignature] = genSig map (FieldSignature)

  val genField: Gen[Field] = frequency((10, genFieldBoolean),(10, genFieldWord8),(10, genFieldInt16),
    (10, genFieldInt32), (10, genFieldString), (10, genFieldObjectPath), (10, genFieldSignature), 
    (10, genVariant), (1,  genStructure)
  )

  def genVariant: Gen[FieldVariant] = lzy(genField map (FieldVariant))
  def genStructure: Gen[FieldStructure] = lzy(genMessage map (m => FieldStructure(messageSignature_(m), m)))

  def genMessage: Gen[Vector[Field]] = 
    lzy(nonEmptyContainerOf[Vector, Field](genField) suchThat(m => messageSignature(m).isRight))

  implicit lazy val arbMessage = Arbitrary(genMessage)





}

Example:  Scala-DBus

object DBusMarshalSpecification extends Properties("Marshal") {
  import Gen._, DBus._, Arbitrary.arbitrary

  val genFieldBoolean: Gen[FieldBoolean] = arbitrary[Boolean] map (FieldBoolean)
  val genFieldWord8:   Gen[FieldWord8]   = arbitrary[Byte]    map (FieldWord8)
  val genFieldInt16:   Gen[FieldInt16]   = arbitrary[Short]   map (FieldInt16)
  val genFieldInt32:   Gen[FieldInt32]   = arbitrary[Int]     map (FieldInt32)
  val genFieldString:  Gen[FieldString]  = arbitrary[String]  map (FieldString)

  val alphaNumStr = nonEmptyListOf(alphaNumChar) map (_.mkString)
  val genObjectPath = listOf(alphaNumStr) map (_.mkString("/", "/", "").toObjectPath_)
  val genFieldObjectPath = genObjectPath map (FieldObjectPath)

  val genAtomic = oneOf("b", "y", "q", "u", "t", "n", "i", "x", "d", "h", "s", "g", "o")
  val genSig = nonEmptyListOf(genAtomic) suchThat (_.length <= 255) map (_.mkString.toSignature_)
  val genFieldSignature: Gen[FieldSignature] = genSig map (FieldSignature)

  val genField: Gen[Field] = frequency((10, genFieldBoolean),(10, genFieldWord8),(10, genFieldInt16),
    (10, genFieldInt32), (10, genFieldString), (10, genFieldObjectPath), (10, genFieldSignature), 
    (10, genVariant), (1,  genStructure)
  )

  def genVariant: Gen[FieldVariant] = lzy(genField map (FieldVariant))
  def genStructure: Gen[FieldStructure] = lzy(genMessage map (m => FieldStructure(messageSignature_(m), m)))

  def genMessage: Gen[Vector[Field]] = 
    lzy(nonEmptyContainerOf[Vector, Field](genField) suchThat(m => messageSignature(m).isRight))

  implicit lazy val arbMessage = Arbitrary(genMessage)

  property("roundTrip") = forAll { m: Vector[Field] =>
    val s = messageSignature_(m)
    unmarshal_(s, marshal_(m)) == m
  }
}

Testing Stateful Systems

ScalaCheck Commands is a way to test larger systems that contain internal state.

Basic concepts

  • System to be tested
  • Model of the system
  • Sequence of commands
  • Run the command sequence against the system under test
  • Run the same sequence against the model and verify the actual results match the model results

Testing Stateful Systems

trait Commands {
  type State
  type Sut

















}

Testing Stateful Systems

trait Commands {
  type State
  type Sut

  def canCreateNewSut(newState: State, 
                      initSuts: Traversable[State], 
                      runningSuts: Traversable[Sut]): Boolean
  def newSut(state: State): Sut
  def destroySut(sut: Sut): Unit
  def initialPreCondition(state: State): Boolean
  def genInitialState: Gen[State]
  def genCommand(state: State): Gen[Command]








}

Testing Stateful Systems

trait Commands {
  type State
  type Sut

  def canCreateNewSut(newState: State, 
                      initSuts: Traversable[State], 
                      runningSuts: Traversable[Sut]): Boolean
  def newSut(state: State): Sut
  def destroySut(sut: Sut): Unit
  def initialPreCondition(state: State): Boolean
  def genInitialState: Gen[State]
  def genCommand(state: State): Gen[Command]

  trait Command {
    type Result
    def run(sut: Sut): Result
    def nextState(state: State): State
    def preCondition(state: State): Boolean
    def postCondition(state: State, result: Try[Result]): Prop
  }
}

Testing ScalaZ Dequeue

Dequeue is a double-ended queue

  • O(1) access and add at both ends
  • Amortized O(1) delete at both ends
  • Internally uses two Lists
  • May need to move items between lists when an item is removed
abstract class Dequeue[A] extends AnyRef {
  def cons(a: A): Dequeue[A]
  def snoc(a: A): Dequeue[A]
  def uncons: Maybe[(A, Dequeue[A])]
  def unsnoc: Maybe[(A, Dequeue[A])]
  // ...
}

Testing ScalaZ Dequeue

class QueueContainer[T] {
  val queue = Ref(Dequeue.empty[T]).single

  private def swap(x: Maybe[(T, Dequeue[T])]) = x map { x => (x._2, Maybe.just(x._1)) } getOrElse {(Dequeue.empty[T], Maybe.empty[T])}

  def cons(t: T): Unit   = queue transform { _.cons(t) }
  def snoc(t: T): Unit   = queue transform { _.snoc(t) }
  def uncons(): Maybe[T] = queue transformAndExtract { q => swap(q.uncons) }
  def unsnoc(): Maybe[T] = queue transformAndExtract { q => swap(q.unsnoc) }
  def toList: List[T]    = queue().toList
}

object DequeueSpec extends Properties("Dequeue") {
  property("commands") = DequeueCommands.property(threadCount = 1)
}

object DequeueCommands extends Commands {
  type Sut = QueueContainer[Int]
  type State = Vector[Int]

  def canCreateNewSut(...): Boolean = true
  def initialPreCondition(state: State): Boolean = state.isEmpty
  def newSut(state: State): Sut = new QueueContainer()
  def destroySut(sut: Sut): Unit = {}
  def genInitialState: Gen[State] = Vector.empty[Int]
  def genCommand(state: State): Gen[Command] =
    Gen.oneOf(Gen.resultOf(Cons), Gen.resultOf(Snoc), Gen.const(Uncons), 
              Gen.const(Unsnoc), Gen.const(List))
  // ...
}

Testing ScalaZ Dequeue

object DequeueCommands extends Commands {
  // ...
  case class Cons(i: Int) extends UnitCommand {
    def run(sut: Sut): Unit = sut.cons(i)
    def nextState(state: State): State = i +: state
    def preCondition(state: State): Boolean = true
    def postCondition(state: State, success: Boolean): Prop = success
  }

  case object Uncons extends Command {
    type Result = Maybe[Int]
    def run(sut: Sut): Maybe[Int] = sut.uncons()
    def nextState(state: State): State =
      if(state.isEmpty) state
      else state.tail
    def preCondition(state: State): Boolean = true
    def postCondition(state: State, result: Try[Maybe[Int]]): Prop =
      result == Success(Maybe.fromOption(state.headOption))
  }

  case object ToList extends Command {
    type Result = List[Int]
    def run(sut: Sut): List[Int] = sut.toList
    def nextState(state: State): State = state
    def preCondition(state: State): Boolean = true
    def postCondition(state: State, result: Try[List[Int]]): Prop =
      result == Success(state.toList)
  }
  // ...
}

Testing ScalaZ Dequeue

> test-only *DequeueSpec
[info] + Dequeue.commands: OK, passed 100 tests.
[info] Passed: Total 1, Failed 0, Errors 0, Passed 1
[success] Total time: 1 s, completed Feb 16, 2016 8:38:25 PM

Thinking Bigger

A couple interesting places to explore next...

  • Benjamin Pierce -  A Deep Specification for Dropbox
    • Focus on software specification
    • QuickCheck to derive a model for Dropbox
    • Started with a simple model and grew until it matched the actual functionality of Dropbox

Discussion