Positive Mutation
how mutation tests will improve your work
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/7851244/SM_Logo_Dark.png)
ARKADIUSZ KONDAS
Data Scientist
@ Buddy.Works
Code Craftsman
Blogger
Ultra Runner
@ ArkadiuszKondas
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/1920070/twitter-xxl.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/1920092/5328e4296f63cd592700018f_globe-icon.png)
github.com/akondas
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/7851210/photo_2020-07-13_10-22-57.jpg)
arkadiuszkondas.com
Agenda:
- Story
- Theory
- Demo
Story
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/7851392/Pair_programming-amico.png)
Coders Guild
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/7851454/Golden_gate_bridge-amico__1_.png)
Systems
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/7851453/Team-amico.png)
Testers Guild
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/7851458/Data_report-amico.png)
Coverage Report
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/7851461/Questions-amico.png)
Who will guard the guards themselves?
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/7851468/Version_control-amico.png)
Introduction of mutations
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/7851475/Celebration-amico.png)
Eternal prosperity
Theory
Code Coverage
Measure how thoroughly tests exercise application
Code Coverage
Line Coverage / Statement coverage
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/3031539/screenshot-localhost_63342_2016-09-21_21-10-18.png)
Lines: 4/5 Coverage: 80%
Code Coverage
Line Coverage / Statement coverage
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/3031552/Screenshot_from_2016-09-21_21-12-28.png)
Code Coverage
Function and Method Coverage
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/3031716/screenshot-localhost_63342_2016-09-21_21-20-37.png)
Code Coverage
Class and Trait Coverage
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/3031761/screenshot-localhost_63342_2016-09-21_21-56-10.png)
Code Coverage
Branch/patch coverage
class AgeVerification
{
public function passes(int $age): bool
{
if($age <= 0) {
throw new \InvalidArgumentException('Age cannot be negative.');
}
return $age >= 18 && $age <= 99;
}
}
Code Coverage
Branch/patch coverage
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/7850532/Screenshot_from_2020-10-21_19-28-02.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/7850538/Screenshot_from_2020-10-21_19-44-50.png)
Code Coverage
Change Risk Anti-Patterns (CRAP) Index
The Change Risk Anti-Patterns (CRAP) Index is calculated based on the cyclomatic complexity and code coverage of a unit of code. Code that is not too complex and has an adequate test coverage will have a low CRAP index.
The CRAP index can be lowered by writing tests and by refactoring the code to lower its complexity.
Code Coverage
Change Risk Anti-Patterns (CRAP) Index
CRAP1(m) = comp(m)^2 * (1 – cov(m)/100)^3 + comp(m)
Code Coverage
Change Risk Anti-Patterns (CRAP) Index
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/3031797/screenshot-localhost_63342_2016-09-21_22-03-48.png)
Code Coverage
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/3031811/screenshot-twitter.com_2016-09-21_22-06-35.png)
Metrics
Code Coverage
<?php
declare (strict_types = 1);
class GoldCustomer implements CustomerSpecification
{
/**
* @param Customer $candidate
*
* @return bool
*/
public function isSatisfiedBy(Customer $candidate): bool
{
return $candidate->getTotalPurchases() >= 100;
}
}
Code Coverage
<?php
declare (strict_types = 1);
class GoldCustomerTest extends \PHPUnit_Framework_TestCase
{
public function testIsSatisfiedByCustomer()
{
$customer = new Customer();
$goldCustomer = new GoldCustomer();
$this->assertFalse($goldCustomer->isSatisfiedBy($customer));
}
}
100% Code Coverage, but ...
Code Coverage
/**
* @param int $totalPurchases
* @param bool $expected
*
* @dataProvider customerProvider
*/
public function testIsSatisfiedReturnsCorrectResult(int $totalPurchases, bool $expected)
{
$customer = new Customer($totalPurchases);
$goldCustomer = new GoldCustomer();
$this->assertEquals($expected, $goldCustomer->isSatisfiedBy($customer));
}
/**
* @return array
*/
public function customerProvider()
{
return [
[1, false],
[99, false],
[100, true],
[200, true]
];
}
Code Coverage
Mutation Testing
to the rescue
Mutation testing
A tool to provide insight in stability of your code
Mutation testing
- run test siut and check if pass
- generate all possible mutations
- for each mutation run testsiut and check if fails
- evaluate results
Mutation testing
example
public function isInRetirementAge() {
return $this->age > 67;
}
function testIsInRetirementAge() {
$this->assertFalse((new Applicant('John', 30))->isInRetirementAge());
$this->assertTrue((new Applicant('John', 70))->isInRetirementAge());
}
public function isInRetirementAge() {
- return $this->age > 67;
+ return $this->age >= 67;
}
The mutant is alive == has escaped
Mutation testing
example
function testIsInRetirementAge() {
- $this->assertFalse((new Applicant('John', 30))->isInRetirementAge());
+ $this->assertFalse((new Applicant('John', 67))->isInRetirementAge());
$this->assertTrue((new Applicant('John', 70))->isInRetirementAge());
}
public function isInRetirementAge() {
- return $this->age > 67;
+ return $this->age >= 67;
}
The mutant is killed
Mutation
- killed if at least 1 test fails
- escaped if at all test pass
-
uncovered mutant is not covered
by a test - fatal mutant produces a fatal error
- timeout unit tests exceed allowed timeout
- skipped could not be tested
a piece of code that has been mutated by mutator
Mutators
Binary Arithmetic
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/7851545/Screenshot_from_2020-10-22_00-03-20.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/7851548/Screenshot_from_2020-10-22_00-03-46.png)
Binary Arithmetic
26) \Humbug\Mutator\Arithmetic\PlusEqual
--- Original
+++ New
@@ @@
for ($n = 0; $n < $this->dimension; ++$n) {
- $centroid->coordinates[$n] += $point->coordinates[$n];
+ $centroid->coordinates[$n] -= $point->coordinates[$n];
}
}
for ($n = 0; $n < $this->dimension; ++$n) {
$this->coordinates[$n] = $centroid->coordinates[$n] / $count;
}
Mutators
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/7851566/Screenshot_from_2020-10-22_00-07-17.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/7851571/Screenshot_from_2020-10-22_00-08-01.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/7851573/Screenshot_from_2020-10-22_00-08-38.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/7851574/Screenshot_from_2020-10-22_00-09-03.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/7851576/Screenshot_from_2020-10-22_00-09-32.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/7851579/Screenshot_from_2020-10-22_00-10-03.png)
Metrics
-
Mutation Score Indicator (MSI):
percentage of mutants covered & killed by tests
-
Mutation Code Coverage:
percentage of mutants covered by tests
-
Covered Code MSI:
percentage of killed mutants that werecoverd by tests
Metrics Example
https://github.com/munusphp/munus
506 mutations were generated:
401 mutants were killed
47 mutants were not covered by tests
45 covered mutants were not detected
4 errors were encountered
9 time outs were encountered
0 mutants required more time than configured
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/7851650/Screenshot_from_2020-10-22_00-29-39.png)
Workflow
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/2783125/flow.png)
Source: Mutation Testing Better Code by Making Bugs Filip van Laenen
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/7852570/keep-calm-and-dont-boil-the-ocean.png)
CHANGED_FILES=$(git diff origin/master --diff-filter=AM --name-only | grep src/ | paste -sd "," -);
INFECTION_FILTER="--filter=${CHANGED_FILES} --ignore-msi-with-no-mutations";
infection --threads=4 $INFECTION_FILTER
Run mutation testing on changesets
Implementation
- The filesystem
- The Abstract Syntax Tree (AST)
- The bytecode / opcode
Demo
source: https://newsmeter.in/teslas-cybertruck-armour-glass-demo-goes-wrong/
Infection
Eats Code Coverage for breakfast
Infection
- Running all tests for each mutation is inefficient
- Before running tests, gather coverage data from phpunit
- Next only run the tests that cover the mutated code
- Stop testing a mutation as soon as at least 1 test fails
- Sort the test on their execution time
Let's run CLI ...
Summary
Summary
- Write tests
- Separate fast unit tests from slow integration tests
- False positive will show up
- Mutation testing will improve the quality of your tests
Other languages
- Java: Pitest (http://pitest.org/)
- Ruby: Mutant (https://github.com/mbj/mutant)
- C#: NinjaTurtles* (https://ninjaturtles.codeplex.com/)
- JavaScript: Stryker (https://github.com/stryker-mutator/stryker)
-
Python: mutmut
(https://github.com/boxed/mutmut)
Q&A
Resources/attributions
- https://infection.github.io/guide/
- https://doug.codes/php-code-coverage
- https://phpunit.readthedocs.io/en/9.3/code-coverage-analysis.html
- http://crestweb.cs.ucl.ac.uk/resources/mutation_testing_repository/
- https://github.com/mariuszgil/aggregates-by-example
- https://pixabay.com/
- Illustration by Freepik Stories: https://stories.freepik.com/web
Thanks for listening
@ ArkadiuszKondas
github.com/akondas
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/1920070/twitter-xxl.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/1920092/5328e4296f63cd592700018f_globe-icon.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/1920105/dziekuje-za-uwage.png)
Pozytywna Mutacja: jak testy mutacyjne usprawnią Twoją pracę
By Arkadiusz Kondas
Pozytywna Mutacja: jak testy mutacyjne usprawnią Twoją pracę
Testowanie mutacyjne to technika pozwalając na pomiar jakości testów. Polega ona na celowym wprowadzaniu małych zmian (mutacji) w kodzie, a następnie sprawdzeniu czy przynajmniej jeden test nie przechodzi. Podczas prezentacji przedstawię koncepcję testów mutacyjnych wraz z praktycznym wdrożeniem na podstawie biblioteki infection oraz interpretacją wyników.
- 1,021