mutation testing for beginners
ARKADIUSZ KONDAS
Backend Team Leader
@ Da Vinci Studio
Zend Certified Engineer
Code Craftsman
Blogger
Ultra Runner
@ ArkadiuszKondas
itcraftsman.pl
Zend Certified Architect
Rasmus Lerdorf
PHP
1994
PHP 7.0
206,128 instances classified in 30 seconds
(6,871 per second)
https://github.com/syntheticminds/php7-ml-comparison
Python 2.7
106,879 instances classified in 30 seconds
(3,562 per second)
NodeJS v5.11.1
245,227 instances classified in 30 seconds
(8,174 per second)
Java 8
16,809,048 instances classified in 30 seconds
(560,301 per second)
For PHP:
Measure how thoroughly tests exercise application
sebastianbergmann/php-code-coverage
Line Coverage / Statement coverage
The Line Coverage software metric measures whether each executable line was executed.
Line Coverage / Statement coverage
Lines: 4/5 Coverage: 80%
Line Coverage / Statement coverage
Function and Method Coverage
The Function and Method Coverage software metric measures whether each function or method has been invoked.
PHP_CodeCoverage only considers a function or method as covered when all of its executable lines are covered.
Function and Method Coverage
Class and Trait Coverage
The Class and Trait Coverage software metric measures whether each method of a class or trait is covered.
PHP_CodeCoverage only considers a class or trait as covered when all of its methods are covered.
Class and Trait Coverage
Change Risk Anti-Patterns (CRAP) Index
The Change Risk Anti-Patterns (CRAP) Index is calculated based on the cyclomatic complexity and code coverage of a unit of code. Code that is not too complex and has an adequate test coverage will have a low CRAP index.
The CRAP index can be lowered by writing tests and by refactoring the code to lower its complexity.
Change Risk Anti-Patterns (CRAP) Index
CRAP1(m) = comp(m)^2 * (1 – cov(m)/100)^3 + comp(m)
Change Risk Anti-Patterns (CRAP) Index
Metrics
<?php
declare (strict_types = 1);
class GoldCustomer implements CustomerSpecification
{
/**
* @param Customer $candidate
*
* @return bool
*/
public function isSatisfiedBy(Customer $candidate): bool
{
return $candidate->getTotalPurchases() >= 100;
}
}
<?php
declare (strict_types = 1);
class GoldCustomerTest extends \PHPUnit_Framework_TestCase
{
public function testIsSatisfiedByCustomer()
{
$customer = new Customer();
$goldCustomer = new GoldCustomer();
$this->assertFalse($goldCustomer->isSatisfiedBy($customer));
}
}
100% Code Coverage, but ...
/**
* @param int $totalPurchases
* @param bool $expected
*
* @dataProvider customerProvider
*/
public function testIsSatisfiedReturnsCorrectResult(int $totalPurchases, bool $expected)
{
$customer = new Customer($totalPurchases);
$goldCustomer = new GoldCustomer();
$this->assertEquals($expected, $goldCustomer->isSatisfiedBy($customer));
}
/**
* @return array
*/
public function customerProvider()
{
return [
[1, false],
[99, false],
[100, true],
[200, true]
];
}
Mutation Testing
to the rescue
A tool to provide insight in stability of your code
public function isInRetirementAge() {
return $this->age > 67;
}
function testIsInRetirementAge() {
$this->assertFalse((new Applicant('John', 30))->isInRetirementAge());
$this->assertTrue((new Applicant('John', 70))->isInRetirementAge());
}
public function isInRetirementAge() {
- return $this->age > 67;
+ return $this->age >= 67;
}
The mutant is alive == has escaped
function testIsInRetirementAge() {
- $this->assertFalse((new Applicant('John', 30))->isInRetirementAge());
+ $this->assertFalse((new Applicant('John', 67))->isInRetirementAge());
$this->assertTrue((new Applicant('John', 70))->isInRetirementAge());
}
public function isInRetirementAge() {
- return $this->age > 67;
+ return $this->age >= 67;
}
The mutant is killed
a piece of code that has been mutated by mutator
Source | Mutation |
---|---|
+ | - |
- | + |
* | / |
/ | * |
% | * |
26) \Humbug\Mutator\Arithmetic\PlusEqual
--- Original
+++ New
@@ @@
for ($n = 0; $n < $this->dimension; ++$n) {
- $centroid->coordinates[$n] += $point->coordinates[$n];
+ $centroid->coordinates[$n] -= $point->coordinates[$n];
}
}
for ($n = 0; $n < $this->dimension; ++$n) {
$this->coordinates[$n] = $centroid->coordinates[$n] / $count;
}
Source
true
false
&&
||
and
or
!
Mutation
false
true
||
&&
or
and
19) \Humbug\Mutator\Boolean\TrueValue
--- Original
+++ New
@@ @@
}
- $visited[$index] = true;
+ $visited[$index] = false;
$regionSamples = $this->getSamplesInRegion($sample, $samples);
if (count($regionSamples) >= $this->minSamples) {
$clusters[] = $this->expandCluster($regionSamples, $visited);
}
}
Source
>
<
>=
<=
Mutation >=
<=
>
<
50) \Humbug\Mutator\ConditionalBoundary\GreaterThan
--- Original
+++ New
@@ @@
foreach ($this as $point) {
- if (($sum -= $distances[$point]) > 0) {
+ if (($sum -= $distances[$point]) >= 0) {
continue;
}
$clusters[] = new Cluster($this, $point->getCoordinates());
break;
}
Source
return true;
return 1.0;
return $this;
return (stmt);
return 1.0;
Mutation
return false;
return -( + 1);
return null;
(stmt); return null;
return 0.0;
181) \Humbug\Mutator\ReturnValue\FunctionCall
--- Original
+++ New
@@ @@
{
- return implode(static::$separator, $words);
+ implode(static::$separator, $words); return null;
}
protected static function strlen($text)
{
return function_exists('mb_strlen') ? mb_strlen($text, 'UTF-8') : strlen($text);
}
Negated Conditionals
!== ===
Increments
++ --
Literal Numbers
changes literal int and float values
Literal Numbers
--- Original
+++ New
@@ @@
{
- if ($clustersNumber <= 0) {
+ if ($clustersNumber <= 1) {
throw InvalidArgumentException::invalidClustersNumber();
}
$this->clustersNumber = $clustersNumber;
$this->initialization = $initialization;
}
Humbug = filesystem
Mutation Score Indicator (MSI): 83%
Mutation Code Coverage: 98%
Covered Code MSI: 85%
https://github.com/php-ai/php-ml
651 mutations were generated:
522 mutants were killed
10 mutants were not covered by tests
99 covered mutants were not detected
0 fatal errors were encountered
20 time outs were encountered
Source: Mutation Testing Better Code by Making Bugs Filip van Laenen
Eats Code Coverage for breakfast
composer global require 'humbug/humbug=~1.0@dev'
export PATH=~/.composer/vendor/bin:$PATH
humbug configure
// humbug.json.dist
{
"timeout": 10,
"source": {
"directories": [
"src"
]
},
"logs": {
"text": "humbuglog.txt",
"json": "humbuglog.json"
}
}
// all
humbug
// custom timeput
humbug --timeout=10
// selected file
humbug --file=PrimeFactor.php
// files mask
humbug --file=*Driver.php
Humbug running test suite to generate logs and code coverage data...
Humbug has completed the initial test run successfully.
Tests: 105 Line Coverage: 95.48%
Humbug is analysing source files...
Mutation Testing is commencing on 61 files...
(.: killed, M: escaped, S: uncovered, E: fatal error, T: timed out)
............................................M............... | 60 ( 7/61)
...................................M.....M.........M....MMMM | 120 (11/61)
M.................................M.................M....... | 180 (11/61)
...M.........M....M...............M......S....S.SM.MM.M..M.. | 240 (19/61)
M..M.MM.M.MM.M.MSM.M....S......MM...MMMM..M.M.MMT...M...TMMM | 300 (21/61)
M.....TMMM.MMMMM.M.MT.M.MT.M......M....M......M.....M....... | 360 (30/61)
......M.M...............MM........M.MM...MM...M.MMM......... | 420 (49/61)
....T.........M....M.SSS......MM.....
457 mutations were generated:
363 mutants were killed
8 mutants were not covered by tests
80 covered mutants were not detected
0 fatal errors were encountered
6 time outs were encountered
Metrics:
Mutation Score Indicator (MSI): 81%
Mutation Code Coverage: 98%
Covered Code MSI: 82%
Remember that some mutants will inevitably be harmless (i.e. false positives).
Time: 3.65 minutes Memory: 10.00MB
Humbug results are being logged as TEXT to: humbuglog.txt
------
Escapes
------
1) \Humbug\Mutator\Number\IntegerValue
Diff on \Phpml\Math\Statistic\StandardDeviation::population() in /var/www/php-ml/src/Phpml/Math/Statistic/StandardDeviation.php:
--- Original
+++ New
@@ @@
if ($sample && $n === 1) {
- throw InvalidArgumentException::arraySizeToSmall(2);
+ throw InvalidArgumentException::arraySizeToSmall(3);
}
$mean = Mean::arithmetic($a);
$carry = 0.0;
foreach ($a as $val) {
$d = $val - $mean;
29) \Humbug\Mutator\Arithmetic\Division
Diff on \Phpml\Clustering\KMeans\Cluster::updateCentroid()
in /var/www/php-ml/src/Phpml/Clustering/KMeans/Cluster.php:
--- Original
+++ New
@@ @@
for ($n = 0; $n < $this->dimension; ++$n) {
- $this->coordinates[$n] = $centroid->coordinates[$n] / $count;
+ $this->coordinates[$n] = $centroid->coordinates[$n] * $count;
}
}
/**
* @return Point[]|SplObjectStorage
*/
30) \Humbug\Mutator\ConditionalBoundary\LessThan
Diff on \Phpml\Clustering\KMeans\Space::__construct() in /var/www/php-ml/src/Phpml/Clustering/KMeans/Space.php:
--- Original
+++ New
@@ @@
{
- if ($dimension < 1) {
+ if ($dimension <= 1) {
throw new LogicException('a space dimension cannot be null or negative');
}
$this->dimension = $dimension;
}
{
"summary": {
"total": 457,
"kills": 364,
"escapes": 79,
"errors": 0,
"timeouts": 6,
"notests": 8,
"covered_score": 82,
"combined_score": 81,
"mutation_coverage": 98
},
"escaped": [
{
"file": "src\/Phpml\/Math\/Statistic\/StandardDeviation.php",
"mutator": "\\Humbug\\Mutator\\Number\\IntegerValue",
"class": "\\Phpml\\Math\\Statistic\\StandardDeviation",
"method": "population",
"line": 28,
"diff": "--- Original\n+++ New\n@@ @@\n if ($sample && $n === 1) {\n- throw InvalidArgumentException::arraySizeToSmall(2);\n+ throw InvalidArgumentException::arraySizeToSmall(3);\n }\n \n $mean = Mean::arithmetic($a);\n $carry = 0.0;\n foreach ($a as $val) {\n $d = $val - $mean;",
"tests": [
"test\\Phpml\\Math\\StandardDeviation\\StandardDeviationTest::testThrowExceptionOnToSmallArray"
],
"stderr": "",
"stdout": "TAP version 13"
},
{
"file": "src\/Phpml\/Math\/Matrix.php"
Project | Coverage | MSI | MCC | CCM | Time |
---|---|---|---|---|---|
php-ai/php-ml | 95,7% | 83% | 98% | 85% | 8m |
twigphp/Twig | 82% | 68% | 86% | 79% | 26m |
mathiasverraes/money | 96% | 92% | 100% | 92% | 25s |
symfony/event-dispatcher | 85% | 54% | 69% | 78% | 40s |
https://slides.com/arkadiuszkondas
https://github.com/itcraftsmanpl/phpcon-2016