Your tests suck. Try this one weird trick!

Tom Newby

@tomnewbyau

I love testing

• So should you
• It's good for you
• Tests are how you prove your code works, today and forever

Code coverage ain't everything

• write tests that cover every line, but make few assertions?
• then your tests might suck

"If you change the logic and none of your tests fail, were you even really testing it?"

~ the groundhog from Groundhog Day, probably

Enter mutation testing

Philosophy

If I mutate your code ever so slightly, your tests should fail because we've changed the implementation

Definitions:

mutant - a single code modification

uncovered mutants - tests didn't even cover this line

escaped mutants - your tests covered this line, but didn't fail with the mutation

In practice

☢️ infection.github.io ☢️

• Runs your PHPUnit tests with xdebug code coverage on
• Automatically mutates your code
• For each mutation, run ONLY the tests that covered that line

MSI = % killed mutants

Plus many more

Example mutation log

Considerations

• Speed
  • xdebug for initial pass, but not for mutations
• Aim for 100% MSI on a reduce scope of files
• Limit scope with config of source.directories
  • limit namespaces/paths
  • target it on "core" code
  • not on generic controllers
• PR reviews "complex, pls move to core"
• Diff only

Questions?

Hiring engineers for 2022