Meet the new kid
testing framework on the block

Petra Bierleutgeb

@pbvie

MUnit

presented at ScalaLove ❤️ Conf by

Great! ...but do we need one?

A new testing framework

ScalaTest

MUnit

scala-verify

uTest

minitest

🐲 The Dark Age without
Scala Testing Frameworks

Scala Testing Frameworks
through the ages

specs2

# Quick Facts

  • started by Ólafur Páll Geirsson - @olafurpg
  • first release in 2020
  • used in several Scalameta projects
  • member of the ScalaMeta family

# Goals and Features I

  • tests as values
  • minimal and developer-friendly API
  • extensive and dynamic filtering
  • transformations
  • actionable errors

# Goals and Features II

  • support for the whole Scala ecosystem
  • good IDE support, leverage existing tooling
  • visibility and reporting
  • support for ScalaCheck
    (contributed by Gabriele Petronella - @gabro27)

Getting Started with MUnit

Hello World

# Setup

libraryDependencies += "org.scalameta" %% "munit" % "0.7.2" % Test

testFrameworks += new TestFramework("munit.Framework")

build.sbt

# Writing tests

class HelloMUnitSuite extends munit.FunSuite {

  test("hello world") {
    val greeting = "hello world"
    assertEquals(greeting, "hello world")
  }
  
}

HelloMUnitSuite.sbt

  • test suites must be classes
  • extend munit.Suite
    • use munit.FunSuite for convenience...or roll your own

# FunSuite

clean and (quite) minimal testing suite that can be used without learning about the underlying implementation

comes with batteries included and should cover most of your
day-to-day testing needs :)

# FunSuite

  • best way to get started with MUnit
  • provides `test` method to define tests that automatically work with sync/async values
class HelloFunSuite extends munit.FunSuite {

  def doSomething(): Int = ???
  def doSomethingAsync(): Future[Int] ???

  test("sync") {
    doSomething()
  }
  
  test("async") {
    doSomethingAsync()
  }
  
}

# Features as traits

  • assertions
  • fixtures, before/after
  • transformations for suites, tests and values

automatically available when using FunSuite - can be mixed into your own suites

Tests as Values

# Tests as Values

abstract class Suite extends PlatformSuite {

  // the value produced by test bodies
  type TestValue
  final type Test = GenericTest[TestValue]

  // a suite consists of a sequence of tests
  def munitTests(): Seq[Test]

  ...

a test is an instance of GenericTest[TestValue]

knowing about GenericTest[TestValue] is helpful to tweak/customize your tests but it's not required

using FunSuite will take care of most things

# Tests as Values - FunSuite

// simplified
abstract class FunSuite
    extends Suite
    with Assertions
    with ... {

  final type TestValue = Future[Any]

  final val munitTestsBuffer: mutable.ListBuffer[Test] =
    mutable.ListBuffer.empty[Test]
    
  def munitTests(): Seq[Test] = {
    munitSuiteTransform(munitTestsBuffer.toList)
  }

  def test(...) {
    // create new test and add it to the test buffer
  }

# GenericTest[TestValue]

class GenericTest[T](
    val name: String,
    val body: () => T,
    val tags: Set[Tag], // set of tags for filtering/transformations
    val location: Location // enables "jump to test/error"
)

that encourages customization

Developer-friendly API

# Customizations

treating tests as values gives developers a lot of freedom

many customizations can be achieved through tags, filters and  transformations

# munit.Suite - the minimalist

class CustomSuite extends munit.Suite {

  override type TestValue = Future[String]

  override def munitTests() = List(
    new Test(
      "my custom test",
      // won't compile if test body is not a Future[String]
      body = () => Future.successful("hello")
      tags = Set.empty[Tag],
      location = implicitly[Location]
    )
  )

}

Filtering and Transformation

with Tags

# How to define tags

class TagSuite extends munit.FunSuite {

  val takesForever = new munit.Tag("takesforever")
  val db           = new munit.Tag("db")

  test("a very slow test".tag(takesForever).tag(db)) {
    // ...
  }
  
  // ...

# Filtering - Suites

// always ignore suite
@munit.IgnoreSuite
class MySuite extends munit.FunSuite { ...

// ignore suite on dynamic condition
class MyWindowsOnlySuite extends munit.FunSuite {
  override def munitIgnore: Boolean = // condition

# Filtering - Tests

// in your test suite
class TagSuite extends munit.FunSuite {

  val db = new munit.Tag("db")

  override def munitTests(): Seq[Test] = {
    // useful for filtering based on dynamic conditions
    val unfiltered = super.munitTests()
    if (myCondition) unfiltered.filter(_.tags.contains(db))
    else unfiltered
  }
  
  test("slow test".tag(db)) { ... }
// in sbt
testOnly -- --exclude-tags=db

sbt shell

TagSuite.scala

or

# Transformations

used to transform suites/tests/values based on
static or dynamic conditions

# Transformations

SuiteTransforms

TestTransforms

ValueTransforms

# Example: Value Transformation

  • adding support for monix Task
import monix.eval.Task
// ...

class TaskSuite extends munit.FunSuite {

  def getFromDbTask: Task[Int] = Task {
    // pretend to get value from db
    throw new RuntimeException("boom")
  }

  /* problem: Task is lazy, so nothing will actually run 
     and the test will not fail */
  test("hello monix task") {
    getFromDbTask.map { result =>
      // ...
    }
  }

# Example: Value Transformation

  • one way to fix it
class TaskSuite extends munit.FunSuite {

  def getFromDbTask: Task[Int] = Task {
    // pretend to get value from db
    throw new RuntimeException("boom")
  }

  /* problem: Task is lazy, so nothing will actually run 
     and the test will not fail */
  test("hello monix task") {
    getFromDbTask.map { result =>
      // ...
    }.runToFuture
  }

# Example: Value Transformation

  • but it's not nice: we have to repeat it in every single test
  • even worse: if we forget it, the test will still pass
  • a better solution: automatically transform the Task to a Future by using a Value Transformation

# Example: Value Transformation

class TaskSuite extends munit.FunSuite {

  override def munitValueTransforms = 
    super.munitValueTransforms ++ List(
      new ValueTransform("Task", {
        case t: Task[_] => t.runToFuture
      })
    )

  def getFromDbTask: Task[Int] = Task {
    // pretend to get value from db
    throw new RuntimeException("boom")
  }

  test("hello monix task") {
    getFromDbTask.map { result =>
      // ...
    } // no need to call .runToFuture in every test anymore
  }

and helpful diffs

Actionable Test Results

# Actionable Test Results

provide useful information to debug test failures right from the terminal/test output

# Actionable Test Results

  • jump to location
  • clues
  • stracktrace highlighting
  • nice diffs when using `assertEquals`

# Actionable Test Results

final case class Cat(name: String, color: String)

test("diff case classes") {
  val realGarfield = Cat("garfield", "ginger")
  val imposterGarfield = Cat("garfield", "grey")
  assertEquals(imposterGarfield, realGarfield)
}

# Actionable Test Results

...and more

# Features/Content that didn't make it
into the presentation

  • fixtures
  • special treatment of flaky tests
  • sbt plugin to upload reports to Google Cloud Storage and generate reports with mdoc
  • more examples

and further reading

Credits

# MUnit is inspired by

  • ScalaTest
  • uTest
  • JUnit
  • ava (JavaScript test framework)

# Reading Material and references

Hope you enjoyed it.

That's it for today!

Made with Slides.com