Beware mutants

Testing Your Tests

Ian Littman / @iansltx

php[tek] 2016


100% code coverage

It's probably a lie.

  • Line
  • Function/method
  • Class/trait
  • Opcode
  • Branch
  • Path

100% executed != 100% tested

tests are code

  • Failures on boundary conditions
  • Incomplete specifications
  • Plain old bugs

Where do we go from here?

Tests, what are they good for?

  • Ensure functionality when written
    • Common cases
    • Edge cases
    • Corner cases
  • Catch regressions when modified

What if...

Mutation Testing!

  • Tweak code (one regression at a time)
  • Run tests
  • See what happens!

Tweak The Code

  • Flip booleans
  • Off-by-one
  • Modify equalities
  • Negate expressions
  • Change return values
  • Change math operators
  • Change logical operators
  • etc.

Run The Tests

  • Line coverage: you don't know what you don't know
  • Release mutants in a secured location
  • This will take awhile

View the results

  • Objective: catch mutants at test time
  • Good
    • Test fails
    • Test times out
    • Test results in a fatal
  • Bad: test suite passes

Use it in PHP

  • PiTest Mutagenesis MutateMe Humbug

  • from the maker of Mockery (Pádraic Brady)

  • Requires xdebug*

  • Ties in with PHPUnit

  • JSON config

    • Override/exclude code or test dirs

    • Set timeouts

  • Plenty of mutators

  • Performance optimizations

  • Incremental Testing (unstable)

setting it up

  • composer global require 'humbug/humbug=~1.0@dev'
  • Make sure ~/.composer/vendor/bin is in $PATH

  • humbug configure
  • humbug

You've Got Mutants! (probably)

  • Summary
    • # killed/escaped/timed out/errored/not covered
    • % mutants killed (Mutation Score Indicator)
    • % mutants in covered code
    • % mutants killed in covered code
  • For each mutant...
    • Code diff
    • Mutation type

You've GOt Mutants! (JSON Edition)

  • Tests run
  • For each test...
    • Class
    • Method
    • Line
    • Test output

Demo Time!

System under test: BusinessDays

Thanks! Questions?

Beware Mutants: Testing Your Tests - php[tek] 2016

By Ian Littman

Beware Mutants: Testing Your Tests - php[tek] 2016

When building a test suite, it is easy to forget that tests are code. Therefore, tests can have bugs and uncaught edge cases, just like normal code...even with 100% line code coverage. The solution: a test to test your tests, warping your code to try to make your tests fail. I will guide you through the ideas behind this so-called mutation testing, including tools to perform the tests and enhancements to testing tools to catch when "100%" code coverage does not cut it.

  • 2,522