Testing Your Tests
Ian Littman / @iansltx
AustinPHP / February 11, 2016
- I write web apps, primarily APIs, for a few companies
- I like conferences
- Attended SunshinePHP 2016 last weekend
- Attending Lone Star PHP 2016 in April
- Speaking at php[tek] 2016 in May
- (@)iansltx everywhere
So, what's this then?
- The problem with code coverage (why mutation testing?)
- What is mutation testing?
- How do you mutation test PHP code? (demo!)
Who watches the watchers?
- Tests are code
- Failures on boundary conditions
- Outright bugs
- Unit tests should catch common and edge cases during original dev, and catch regressions later
- Code coverage indicates how much of a code base is executed when running a test suite...
- ...but it can be misleading
100% code coverage
The solution: mutation testing
- Tweak your code (simulate regressions/typos)
- Run your test suite
- See what happens!
tweak your code
- Flip booleans
- Modify equalities
- Negate expressions
- Change return values
- Change math operators
- Change logical operators
- ...and more!
Run your test suite
- Line coverage is a starting point
- You're probably going to miss a mutation if it's not in covered code
- High coverage -> more chances to catch mutants...
- ...or watch them escape
- Run these tests in a secure location!
- Mutation testing takes awhile
- Runs all of, or a part of, your test suite for each mutation
see what happens
- Tests fail (good!) <- mutant killed
- Tests time out (infinite loop; okay!)
- Tests fatal (probably fine)
- Test suite passes (probably bad) <- mutant escaped
- "Probably" due to false positives
- Fatals and timeouts are fine because you'll catch those sorts of regressions by running your automated test suite, rather than in production
How do we use this?
How do we use Humbug?
composer global require 'humbug/humbug=~1.0@dev'
Add ~/.composer/vendor/bin to $PATH if you haven't already
- Wait for results (you'll see mutants get killed/escape/time out/fatal as test suites are run)
We've got results!
- # killed/escaped/timed out/errored/not covered
- % of mutants killed (Mutation Score Indicator)
- % of mutants in covered code
- % of mutants killed in covered code
- Text log
- Summary data, plus...
- For each escaped mutant
- Location of code diff
- Code diff
- Mutation type (e.g. conditional boundary change)
- JSON log (more detailed than text log)
- Tests executed
- Class/method/line/test output
Beware Mutants: Testing Your Tests - AustinPHP February 2016
By Ian Littman