Beware Mutants

Testing Your Tests

Ian Littman / @iansltx

AustinPHP / February 11, 2016


  • I write web apps, primarily APIs, for a few companies
  • I like conferences
    • Attended SunshinePHP 2016 last weekend
    • Attending Lone Star PHP 2016 in April
    • Speaking at php[tek] 2016 in May
  • (@)iansltx everywhere

So, what's this then?

  • The problem with code coverage (why mutation testing?)
  • What is mutation testing?
  • How do you mutation test PHP code? (demo!)

Who watches the watchers?

  • Tests are code
    • Failures on boundary conditions
    • Outright bugs
    • Incomplete
  • Unit tests should catch common and edge cases during original dev, and catch regressions later
  • Code coverage indicates how much of a code base is executed when running a test suite...
  • ...but it can be misleading

100% code coverage

The solution: mutation testing

  • Tweak your code (simulate regressions/typos)
  • Run your test suite
  • See what happens!

tweak your code

  • Flip booleans
  • Off-by-one
  • Modify equalities
  • Negate expressions
  • Change return values
  • Change math operators
  • Change logical operators
  • ...and more!

Run your test suite

  • Line coverage is a starting point
    • You're probably going to miss a mutation if it's not in covered code
    • High coverage -> more chances to catch mutants...
    • ...or watch them escape
  • Run these tests in a secure location!
  • Mutation testing takes awhile
    • Runs all of, or a part of, your test suite for each mutation

see what happens

  • Tests fail (good!) <- mutant killed
  • Tests time out (infinite loop; okay!)
  • Tests fatal (probably fine)
  • Test suite passes (probably bad) <- mutant escaped
  • "Probably" due to false positives
  • Fatals and timeouts are fine because you'll catch those sorts of regressions by running your automated test suite, rather than in production

How do we use this?

  • PiTest Mutagenesis MutateMe Humbug
  • from the maker of Mockery (Pádraic Brady)
  • Ties in with PHPUnit
    • Current version of 4.8 fatals
    • Some versions of 4.x work fine
    • Current version of 5.x (5.2.4) works fine
  • JSON config
    • Override/exclude code or test dirs
    • Set timeouts

How do we use Humbug?

  • PHP 5.4+

  • composer global require 'humbug/humbug=~1.0@dev'
  • Add ~/.composer/vendor/bin to $PATH if you haven't already

  • humbug configure
  • humbug
  • Wait for results (you'll see mutants get killed/escape/time out/fatal as test suites are run)

We've got results!

  • Summary
    • # killed/escaped/timed out/errored/not covered
    • % of mutants killed (Mutation Score Indicator)
    • % of mutants in covered code
    • % of mutants killed in covered code
  • Text log
    • Summary data, plus...
    • For each escaped mutant
      • Location of code diff
      • Code diff
      • Mutation type (e.g. conditional boundary change)
  • JSON log (more detailed than text log)
    • Tests executed
    • Class/method/line/test output

Demo Time!


Beware Mutants: Testing Your Tests - AustinPHP February 2016

By Ian Littman

Beware Mutants: Testing Your Tests - AustinPHP February 2016

  • 1,206
Loading comments...

More from Ian Littman