Mutated PHP
mutation testing for beginners
ARKADIUSZ KONDAS
Backend Team Leader
@ Da Vinci Studio
Zend Certified Engineer
Code Craftsman
Blogger
Ultra Runner
@ ArkadiuszKondas
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/1920070/twitter-xxl.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/1920092/5328e4296f63cd592700018f_globe-icon.png)
itcraftsman.pl
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/2035688/zce-php5-3-logo.gif)
Zend Certified Architect
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/2035692/zf-zce-logo.gif)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/3031483/USC037.jpg)
Agenda:
- Code Coverage
- Mutation testing
- Humbug
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/3068748/rl.jpg)
Rasmus Lerdorf
PHP
1994
PHP 7.0
206,128 instances classified in 30 seconds
(6,871 per second)
https://github.com/syntheticminds/php7-ml-comparison
Python 2.7
106,879 instances classified in 30 seconds
(3,562 per second)
NodeJS v5.11.1
245,227 instances classified in 30 seconds
(8,174 per second)
Java 8
16,809,048 instances classified in 30 seconds
(560,301 per second)
Code Quality
Static Code Analysis
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/3064632/screenshot-codeclimate.com_2016-09-30_15-36-45.png)
Static Code Analysis
- Code Climate
- Scrutinizer
- SensioLabsInsight
For PHP:
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/3064664/insight.jpg)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/3064666/2988888.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/3064668/code-climate-stickers.jpg)
Tests
- PHPUnit
- phpspec
- Codeception
- Behat
- ...
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/3065812/phpunit.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/3065822/68747470733a2f2f646c2e64726f70626f7875736572636f6e74656e742e636f6d2f752f3238323739372f62656861742f62656861742e706e67.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/3065823/phpspec.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/3065826/codeception.png)
Code Coverage
Measure how thoroughly tests exercise application
Code Coverage
sebastianbergmann/php-code-coverage
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/3031740/screenshot-github.com_2016-09-21_21-50-28.png)
Code Coverage
Line Coverage / Statement coverage
The Line Coverage software metric measures whether each executable line was executed.
Code Coverage
Line Coverage / Statement coverage
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/3031539/screenshot-localhost_63342_2016-09-21_21-10-18.png)
Lines: 4/5 Coverage: 80%
Code Coverage
Line Coverage / Statement coverage
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/3031552/Screenshot_from_2016-09-21_21-12-28.png)
Code Coverage
Function and Method Coverage
The Function and Method Coverage software metric measures whether each function or method has been invoked.
PHP_CodeCoverage only considers a function or method as covered when all of its executable lines are covered.
Code Coverage
Function and Method Coverage
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/3031716/screenshot-localhost_63342_2016-09-21_21-20-37.png)
Code Coverage
Class and Trait Coverage
The Class and Trait Coverage software metric measures whether each method of a class or trait is covered.
PHP_CodeCoverage only considers a class or trait as covered when all of its methods are covered.
Code Coverage
Class and Trait Coverage
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/3031761/screenshot-localhost_63342_2016-09-21_21-56-10.png)
Code Coverage
Change Risk Anti-Patterns (CRAP) Index
The Change Risk Anti-Patterns (CRAP) Index is calculated based on the cyclomatic complexity and code coverage of a unit of code. Code that is not too complex and has an adequate test coverage will have a low CRAP index.
The CRAP index can be lowered by writing tests and by refactoring the code to lower its complexity.
Code Coverage
Change Risk Anti-Patterns (CRAP) Index
CRAP1(m) = comp(m)^2 * (1 – cov(m)/100)^3 + comp(m)
Code Coverage
Change Risk Anti-Patterns (CRAP) Index
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/3031797/screenshot-localhost_63342_2016-09-21_22-03-48.png)
Code Coverage
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/3031811/screenshot-twitter.com_2016-09-21_22-06-35.png)
Metrics
Code Coverage
<?php
declare (strict_types = 1);
class GoldCustomer implements CustomerSpecification
{
/**
* @param Customer $candidate
*
* @return bool
*/
public function isSatisfiedBy(Customer $candidate): bool
{
return $candidate->getTotalPurchases() >= 100;
}
}
Code Coverage
<?php
declare (strict_types = 1);
class GoldCustomerTest extends \PHPUnit_Framework_TestCase
{
public function testIsSatisfiedByCustomer()
{
$customer = new Customer();
$goldCustomer = new GoldCustomer();
$this->assertFalse($goldCustomer->isSatisfiedBy($customer));
}
}
100% Code Coverage, but ...
Code Coverage
/**
* @param int $totalPurchases
* @param bool $expected
*
* @dataProvider customerProvider
*/
public function testIsSatisfiedReturnsCorrectResult(int $totalPurchases, bool $expected)
{
$customer = new Customer($totalPurchases);
$goldCustomer = new GoldCustomer();
$this->assertEquals($expected, $goldCustomer->isSatisfiedBy($customer));
}
/**
* @return array
*/
public function customerProvider()
{
return [
[1, false],
[99, false],
[100, true],
[200, true]
];
}
Code Coverage
Mutation Testing
to the rescue
Mutation testing
A tool to provide insight in stability of your code
Mutation testing
- run test
siut and check ifpass - generate all possible mutations
- for each mutation run test
siut and check if fail* - evaluate results
Mutation testing
example
public function isInRetirementAge() {
return $this->age > 67;
}
function testIsInRetirementAge() {
$this->assertFalse((new Applicant('John', 30))->isInRetirementAge());
$this->assertTrue((new Applicant('John', 70))->isInRetirementAge());
}
public function isInRetirementAge() {
- return $this->age > 67;
+ return $this->age >= 67;
}
The mutant is alive == has escaped
Mutation testing
example
function testIsInRetirementAge() {
- $this->assertFalse((new Applicant('John', 30))->isInRetirementAge());
+ $this->assertFalse((new Applicant('John', 67))->isInRetirementAge());
$this->assertTrue((new Applicant('John', 70))->isInRetirementAge());
}
public function isInRetirementAge() {
- return $this->age > 67;
+ return $this->age >= 67;
}
The mutant is killed
Mutation
- killed if at least 1 test fails
- escaped if at all test pass
-
uncovered mutant is not covered
by a test - fatal mutant produces a fatal error
-
timout unit tests exceed allowed timeout
a piece of code that has been mutated by mutator
Mutators
- Binary Arithmetic
- Boolean Substitution
- Conditional Boundaries
- Return values
- Negated Conditionals
- Increments
- Literal Numbers
Binary Arithmetic
Source | Mutation |
---|---|
+ | - |
- | + |
* | / |
/ | * |
% | * |
Binary Arithmetic
26) \Humbug\Mutator\Arithmetic\PlusEqual
--- Original
+++ New
@@ @@
for ($n = 0; $n < $this->dimension; ++$n) {
- $centroid->coordinates[$n] += $point->coordinates[$n];
+ $centroid->coordinates[$n] -= $point->coordinates[$n];
}
}
for ($n = 0; $n < $this->dimension; ++$n) {
$this->coordinates[$n] = $centroid->coordinates[$n] / $count;
}
Boolean Substitution
Source
true
false
&&
||
and
or
!
Mutation
false
true
||
&&
or
and
Boolean Substitution
19) \Humbug\Mutator\Boolean\TrueValue
--- Original
+++ New
@@ @@
}
- $visited[$index] = true;
+ $visited[$index] = false;
$regionSamples = $this->getSamplesInRegion($sample, $samples);
if (count($regionSamples) >= $this->minSamples) {
$clusters[] = $this->expandCluster($regionSamples, $visited);
}
}
Conditional Boundaries
Source
>
<
>=
<=
Mutation >=
<=
>
<
Conditional Boundaries
50) \Humbug\Mutator\ConditionalBoundary\GreaterThan
--- Original
+++ New
@@ @@
foreach ($this as $point) {
- if (($sum -= $distances[$point]) > 0) {
+ if (($sum -= $distances[$point]) >= 0) {
continue;
}
$clusters[] = new Cluster($this, $point->getCoordinates());
break;
}
Return values
Source
return true;
return 1.0;
return $this;
return (stmt);
return 1.0;
Mutation
return false;
return -( + 1);
return null;
(stmt); return null;
return 0.0;
Return values
181) \Humbug\Mutator\ReturnValue\FunctionCall
--- Original
+++ New
@@ @@
{
- return implode(static::$separator, $words);
+ implode(static::$separator, $words); return null;
}
protected static function strlen($text)
{
return function_exists('mb_strlen') ? mb_strlen($text, 'UTF-8') : strlen($text);
}
Other types of operators
Negated Conditionals
!== ===
Increments
++ --
Literal Numbers
changes literal int and float values
Other types of operators
Literal Numbers
--- Original
+++ New
@@ @@
{
- if ($clustersNumber <= 0) {
+ if ($clustersNumber <= 1) {
throw InvalidArgumentException::invalidClustersNumber();
}
$this->clustersNumber = $clustersNumber;
$this->initialization = $initialization;
}
Implementation
- The filesystem
- The Abstract Syntax Tree (AST)
- The bytecode / opcode
Humbug = filesystem
Metrics
-
Mutation Score Indicator (MSI):
percentage of mutants covered & killed by tests
-
Mutation Code Coverage:
percentage of mutants covered by tests
-
Covered Code MSI:
percentage of killed mutants that werecoverd by tests
Metrics Example
Mutation Score Indicator (MSI): 83%
Mutation Code Coverage: 98%
Covered Code MSI: 85%
https://github.com/php-ai/php-ml
651 mutations were generated:
522 mutants were killed
10 mutants were not covered by tests
99 covered mutants were not detected
0 fatal errors were encountered
20 time outs were encountered
TDD Flow
![](https://s3.amazonaws.com/media-p.slid.es/uploads/341527/images/2783125/flow.png)
Source: Mutation Testing Better Code by Making Bugs Filip van Laenen
Humbug
Eats Code Coverage for breakfast
Humbug
- Running all tests for each mutation is inefficient
- Before running tests, gather coverage data from phpunit
- Next only run the tests that cover the mutated code
- Stop testing a mutation as soon as at least 1 test fails
- Sort the test on their execution time
Humbug - Installation
- Git
- Phar
- Composer
composer global require 'humbug/humbug=~1.0@dev'
export PATH=~/.composer/vendor/bin:$PATH
Humbug - Configuration
humbug configure
// humbug.json.dist
{
"timeout": 10,
"source": {
"directories": [
"src"
]
},
"logs": {
"text": "humbuglog.txt",
"json": "humbuglog.json"
}
}
Humbug - Run tests
// all
humbug
// custom timeput
humbug --timeout=10
// selected file
humbug --file=PrimeFactor.php
// files mask
humbug --file=*Driver.php
Results - .txt
Humbug running test suite to generate logs and code coverage data...
Humbug has completed the initial test run successfully.
Tests: 105 Line Coverage: 95.48%
Humbug is analysing source files...
Mutation Testing is commencing on 61 files...
(.: killed, M: escaped, S: uncovered, E: fatal error, T: timed out)
............................................M............... | 60 ( 7/61)
...................................M.....M.........M....MMMM | 120 (11/61)
M.................................M.................M....... | 180 (11/61)
...M.........M....M...............M......S....S.SM.MM.M..M.. | 240 (19/61)
M..M.MM.M.MM.M.MSM.M....S......MM...MMMM..M.M.MMT...M...TMMM | 300 (21/61)
M.....TMMM.MMMMM.M.MT.M.MT.M......M....M......M.....M....... | 360 (30/61)
......M.M...............MM........M.MM...MM...M.MMM......... | 420 (49/61)
....T.........M....M.SSS......MM.....
457 mutations were generated:
363 mutants were killed
8 mutants were not covered by tests
80 covered mutants were not detected
0 fatal errors were encountered
6 time outs were encountered
Metrics:
Mutation Score Indicator (MSI): 81%
Mutation Code Coverage: 98%
Covered Code MSI: 82%
Remember that some mutants will inevitably be harmless (i.e. false positives).
Time: 3.65 minutes Memory: 10.00MB
Humbug results are being logged as TEXT to: humbuglog.txt
------
Escapes
------
1) \Humbug\Mutator\Number\IntegerValue
Diff on \Phpml\Math\Statistic\StandardDeviation::population() in /var/www/php-ml/src/Phpml/Math/Statistic/StandardDeviation.php:
--- Original
+++ New
@@ @@
if ($sample && $n === 1) {
- throw InvalidArgumentException::arraySizeToSmall(2);
+ throw InvalidArgumentException::arraySizeToSmall(3);
}
$mean = Mean::arithmetic($a);
$carry = 0.0;
foreach ($a as $val) {
$d = $val - $mean;
Results - .txt
29) \Humbug\Mutator\Arithmetic\Division
Diff on \Phpml\Clustering\KMeans\Cluster::updateCentroid()
in /var/www/php-ml/src/Phpml/Clustering/KMeans/Cluster.php:
--- Original
+++ New
@@ @@
for ($n = 0; $n < $this->dimension; ++$n) {
- $this->coordinates[$n] = $centroid->coordinates[$n] / $count;
+ $this->coordinates[$n] = $centroid->coordinates[$n] * $count;
}
}
/**
* @return Point[]|SplObjectStorage
*/
30) \Humbug\Mutator\ConditionalBoundary\LessThan
Diff on \Phpml\Clustering\KMeans\Space::__construct() in /var/www/php-ml/src/Phpml/Clustering/KMeans/Space.php:
--- Original
+++ New
@@ @@
{
- if ($dimension < 1) {
+ if ($dimension <= 1) {
throw new LogicException('a space dimension cannot be null or negative');
}
$this->dimension = $dimension;
}
Results - .json
{
"summary": {
"total": 457,
"kills": 364,
"escapes": 79,
"errors": 0,
"timeouts": 6,
"notests": 8,
"covered_score": 82,
"combined_score": 81,
"mutation_coverage": 98
},
"escaped": [
{
"file": "src\/Phpml\/Math\/Statistic\/StandardDeviation.php",
"mutator": "\\Humbug\\Mutator\\Number\\IntegerValue",
"class": "\\Phpml\\Math\\Statistic\\StandardDeviation",
"method": "population",
"line": 28,
"diff": "--- Original\n+++ New\n@@ @@\n if ($sample && $n === 1) {\n- throw InvalidArgumentException::arraySizeToSmall(2);\n+ throw InvalidArgumentException::arraySizeToSmall(3);\n }\n \n $mean = Mean::arithmetic($a);\n $carry = 0.0;\n foreach ($a as $val) {\n $d = $val - $mean;",
"tests": [
"test\\Phpml\\Math\\StandardDeviation\\StandardDeviationTest::testThrowExceptionOnToSmallArray"
],
"stderr": "",
"stdout": "TAP version 13"
},
{
"file": "src\/Phpml\/Math\/Matrix.php"
Selected projects
Project | Coverage | MSI | MCC | CCM | Time |
---|---|---|---|---|---|
php-ai/php-ml | 95,7% | 83% | 98% | 85% | 8m |
twigphp/Twig | 82% | 68% | 86% | 79% | 26m |
mathiasverraes/money | 96% | 92% | 100% | 92% | 25s |
symfony/event-dispatcher | 85% | 54% | 69% | 78% | 40s |
Summary
Summary
- Write tests
- Separate fast unit tests from slow integration tests
- Mutation testing will improve the quality of your tests
Other languages
- Java: Pitest (http://pitest.org/)
- Ruby: Mutant (https://github.com/mbj/mutant)
- C#: NinjaTurtles (https://ninjaturtles.codeplex.com/)
- JavaScript: Stryker (https://github.com/stryker-mutator/stryker)
- Python: Cosmic Ray (https://github.com/sixty-north/cosmic-ray)
Q&A
Thanks for listening
https://slides.com/arkadiuszkondas
https://github.com/itcraftsmanpl/phpcon-2016
Mutated PHP - mutation testing for beginners
By Arkadiusz Kondas
Mutated PHP - mutation testing for beginners
Testowanie mutacyjne to technika pozwalając na pomiar jakości testów. Polega ona na celowym wprowadzaniu małych zmian (mutacji) w kodzie, a następnie sprawdzeniu czy przynajmniej jeden test nie przechodzi. Podczas prezentacji przedstawię koncepcję testów mutacyjnych wraz z praktycznym wdrożeniem na podstawie biblioteki padraic/humbug. Pokaże jak interpretować wyniku testów oraz jak samemu napuścić mutantów na swój kod.
- 1,689