Positive Mutation
how mutation tests will improve your work
ARKADIUSZ KONDAS
Data Scientist
@ Buddy.Works
Code Craftsman
Blogger
Ultra Runner
@ ArkadiuszKondas
github.com/akondas
arkadiuszkondas.com
Agenda:
- Story
- Theory
- Demo
Story
Coders Guild
Systems
Testers Guild
Coverage Report
Who will guard the guards themselves?
Introduction of mutations
Eternal prosperity
Theory
Code Coverage
Measure how thoroughly tests exercise application
Code Coverage
Line Coverage / Statement coverage
Lines: 4/5 Coverage: 80%
Code Coverage
Line Coverage / Statement coverage
Code Coverage
Function and Method Coverage
Code Coverage
Class and Trait Coverage
Code Coverage
Branch/patch coverage
class AgeVerification
{
public function passes(int $age): bool
{
if($age <= 0) {
throw new \InvalidArgumentException('Age cannot be negative.');
}
return $age >= 18 && $age <= 99;
}
}
Code Coverage
Branch/patch coverage
Code Coverage
Change Risk Anti-Patterns (CRAP) Index
The Change Risk Anti-Patterns (CRAP) Index is calculated based on the cyclomatic complexity and code coverage of a unit of code. Code that is not too complex and has an adequate test coverage will have a low CRAP index.
The CRAP index can be lowered by writing tests and by refactoring the code to lower its complexity.
Code Coverage
Change Risk Anti-Patterns (CRAP) Index
CRAP1(m) = comp(m)^2 * (1 – cov(m)/100)^3 + comp(m)
Code Coverage
Change Risk Anti-Patterns (CRAP) Index
Code Coverage
Metrics
Code Coverage
<?php
declare (strict_types = 1);
class GoldCustomer implements CustomerSpecification
{
/**
* @param Customer $candidate
*
* @return bool
*/
public function isSatisfiedBy(Customer $candidate): bool
{
return $candidate->getTotalPurchases() >= 100;
}
}
Code Coverage
<?php
declare (strict_types = 1);
class GoldCustomerTest extends \PHPUnit_Framework_TestCase
{
public function testIsSatisfiedByCustomer()
{
$customer = new Customer();
$goldCustomer = new GoldCustomer();
$this->assertFalse($goldCustomer->isSatisfiedBy($customer));
}
}
100% Code Coverage, but ...
Code Coverage
/**
* @param int $totalPurchases
* @param bool $expected
*
* @dataProvider customerProvider
*/
public function testIsSatisfiedReturnsCorrectResult(int $totalPurchases, bool $expected)
{
$customer = new Customer($totalPurchases);
$goldCustomer = new GoldCustomer();
$this->assertEquals($expected, $goldCustomer->isSatisfiedBy($customer));
}
/**
* @return array
*/
public function customerProvider()
{
return [
[1, false],
[99, false],
[100, true],
[200, true]
];
}
Code Coverage
Mutation Testing
to the rescue
Mutation testing
A tool to provide insight in stability of your code
Mutation testing
- run test siut and check if pass
- generate all possible mutations
- for each mutation run testsiut and check if fails
- evaluate results
Mutation testing
example
public function isInRetirementAge() {
return $this->age > 67;
}
function testIsInRetirementAge() {
$this->assertFalse((new Applicant('John', 30))->isInRetirementAge());
$this->assertTrue((new Applicant('John', 70))->isInRetirementAge());
}
public function isInRetirementAge() {
- return $this->age > 67;
+ return $this->age >= 67;
}
The mutant is alive == has escaped
Mutation testing
example
function testIsInRetirementAge() {
- $this->assertFalse((new Applicant('John', 30))->isInRetirementAge());
+ $this->assertFalse((new Applicant('John', 67))->isInRetirementAge());
$this->assertTrue((new Applicant('John', 70))->isInRetirementAge());
}
public function isInRetirementAge() {
- return $this->age > 67;
+ return $this->age >= 67;
}
The mutant is killed
Mutation
- killed if at least 1 test fails
- escaped if at all test pass
-
uncovered mutant is not covered
by a test - fatal mutant produces a fatal error
- timeout unit tests exceed allowed timeout
- skipped could not be tested
a piece of code that has been mutated by mutator
Mutators
Binary Arithmetic
Binary Arithmetic
26) \Humbug\Mutator\Arithmetic\PlusEqual
--- Original
+++ New
@@ @@
for ($n = 0; $n < $this->dimension; ++$n) {
- $centroid->coordinates[$n] += $point->coordinates[$n];
+ $centroid->coordinates[$n] -= $point->coordinates[$n];
}
}
for ($n = 0; $n < $this->dimension; ++$n) {
$this->coordinates[$n] = $centroid->coordinates[$n] / $count;
}
Mutators
Metrics
-
Mutation Score Indicator (MSI):
percentage of mutants covered & killed by tests
-
Mutation Code Coverage:
percentage of mutants covered by tests
-
Covered Code MSI:
percentage of killed mutants that werecoverd by tests
Metrics Example
https://github.com/munusphp/munus
506 mutations were generated:
401 mutants were killed
47 mutants were not covered by tests
45 covered mutants were not detected
4 errors were encountered
9 time outs were encountered
0 mutants required more time than configured
Workflow
Source: Mutation Testing Better Code by Making Bugs Filip van Laenen
CHANGED_FILES=$(git diff origin/master --diff-filter=AM --name-only | grep src/ | paste -sd "," -);
INFECTION_FILTER="--filter=${CHANGED_FILES} --ignore-msi-with-no-mutations";
infection --threads=4 $INFECTION_FILTER
Run mutation testing on changesets
Implementation
- The filesystem
- The Abstract Syntax Tree (AST)
- The bytecode / opcode
Demo
source: https://newsmeter.in/teslas-cybertruck-armour-glass-demo-goes-wrong/
Infection
Eats Code Coverage for breakfast
Infection
- Running all tests for each mutation is inefficient
- Before running tests, gather coverage data from phpunit
- Next only run the tests that cover the mutated code
- Stop testing a mutation as soon as at least 1 test fails
- Sort the test on their execution time
Let's run CLI ...
Summary
Summary
- Write tests
- Separate fast unit tests from slow integration tests
- False positive will show up
- Mutation testing will improve the quality of your tests
Other languages
- Java: Pitest (http://pitest.org/)
- Ruby: Mutant (https://github.com/mbj/mutant)
- C#: NinjaTurtles* (https://ninjaturtles.codeplex.com/)
- JavaScript: Stryker (https://github.com/stryker-mutator/stryker)
-
Python: mutmut
(https://github.com/boxed/mutmut)
Q&A
Resources/attributions
- https://infection.github.io/guide/
- https://doug.codes/php-code-coverage
- https://phpunit.readthedocs.io/en/9.3/code-coverage-analysis.html
- http://crestweb.cs.ucl.ac.uk/resources/mutation_testing_repository/
- https://github.com/mariuszgil/aggregates-by-example
- https://pixabay.com/
- Illustration by Freepik Stories: https://stories.freepik.com/web
Thanks for listening
@ ArkadiuszKondas
github.com/akondas
Pozytywna Mutacja: jak testy mutacyjne usprawnią Twoją pracę
By Arkadiusz Kondas
Pozytywna Mutacja: jak testy mutacyjne usprawnią Twoją pracę
Testowanie mutacyjne to technika pozwalając na pomiar jakości testów. Polega ona na celowym wprowadzaniu małych zmian (mutacji) w kodzie, a następnie sprawdzeniu czy przynajmniej jeden test nie przechodzi. Podczas prezentacji przedstawię koncepcję testów mutacyjnych wraz z praktycznym wdrożeniem na podstawie biblioteki infection oraz interpretacją wyników.
- 1,127