Mutated PHP 

mutation testing for beginners

ARKADIUSZ KONDAS

Backend Team Leader

@ Da Vinci Studio

Zend Certified Engineer

Code Craftsman

Blogger

Ultra Runner

@ ArkadiuszKondas

itcraftsman.pl

Zend Certified Architect

Agenda:

  1. Code Coverage
  2. Mutation testing
  3. Humbug

Rasmus Lerdorf

PHP

1994

PHP 7.0
206,128 instances classified in 30 seconds 
(6,871 per second)

https://github.com/syntheticminds/php7-ml-comparison

Python 2.7
106,879 instances classified in 30 seconds 
(3,562 per second)

NodeJS v5.11.1
245,227 instances classified in 30 seconds 
(8,174 per second)

Java 8
16,809,048 instances classified in 30 seconds 
(560,301 per second)

Code Quality

Static Code Analysis

Static Code Analysis

  • Code Climate
  • Scrutinizer
  • SensioLabsInsight

For PHP:

Tests

  • PHPUnit
  • phpspec
  • Codeception
  • Behat
  • ...

Code Coverage

Measure how thoroughly tests exercise application

Code Coverage

sebastianbergmann/php-code-coverage

Code Coverage

Line Coverage / Statement coverage

The Line Coverage software metric measures whether each executable line was executed.

Code Coverage

Line Coverage / Statement coverage

Lines: 4/5          Coverage: 80%

Code Coverage

Line Coverage / Statement coverage

Code Coverage

Function and Method Coverage

The Function and Method Coverage software metric measures whether each function or method has been invoked.

 

PHP_CodeCoverage only considers a function or method as covered when all of its executable lines are covered.

Code Coverage

Function and Method Coverage

Code Coverage

Class and Trait Coverage

The Class and Trait Coverage software metric measures whether each method of a class or trait is covered.

 

PHP_CodeCoverage only considers a class or trait as covered when all of its methods are covered.

Code Coverage

Class and Trait Coverage

Code Coverage

Change Risk Anti-Patterns (CRAP) Index

The Change Risk Anti-Patterns (CRAP) Index is calculated based on the cyclomatic complexity and code coverage of a unit of code. Code that is not too complex and has an adequate test coverage will have a low CRAP index.

 

The CRAP index can be lowered by writing tests and by refactoring the code to lower its complexity.

Code Coverage

Change Risk Anti-Patterns (CRAP) Index

CRAP1(m) = comp(m)^2 * (1 – cov(m)/100)^3 + comp(m)

Code Coverage

Change Risk Anti-Patterns (CRAP) Index

Code Coverage

Metrics are a sharp tool. Dont't poke you eye out.

Code Coverage

<?php

declare (strict_types = 1);

class GoldCustomer implements CustomerSpecification
{
    /**
     * @param Customer $candidate
     *
     * @return bool
     */
    public function isSatisfiedBy(Customer $candidate): bool
    {
        return $candidate->getTotalPurchases() >= 100;
    }
}

Code Coverage

<?php

declare (strict_types = 1);

class GoldCustomerTest extends \PHPUnit_Framework_TestCase
{

    public function testIsSatisfiedByCustomer()
    {
        $customer = new Customer();
        $goldCustomer = new GoldCustomer();

        $this->assertFalse($goldCustomer->isSatisfiedBy($customer));
    }

}

100% Code Coverage, but ...

Code Coverage

/**
 * @param int $totalPurchases
 * @param bool $expected
 *
 * @dataProvider customerProvider
 */
public function testIsSatisfiedReturnsCorrectResult(int $totalPurchases, bool $expected)
{
    $customer = new Customer($totalPurchases);
    $goldCustomer = new GoldCustomer();

    $this->assertEquals($expected, $goldCustomer->isSatisfiedBy($customer));
}

/**
 * @return array
 */
public function customerProvider()
{
    return [
        [1, false],
        [99, false],
        [100, true],
        [200, true]
    ];
}

Code Coverage

Mutation Testing
to the rescue

Mutation testing

A tool to provide insight in stability of your code

Mutation testing

  • run test siut and check if pass
  • generate all possible mutations
  • for each mutation run test siut and check if fail*
  • evaluate results

Mutation testing
example

public function isInRetirementAge() {
    return $this->age > 67;
}
function testIsInRetirementAge() {
    $this->assertFalse((new Applicant('John', 30))->isInRetirementAge());
    $this->assertTrue((new Applicant('John', 70))->isInRetirementAge());
}
public function isInRetirementAge() {
-    return $this->age > 67;
+    return $this->age >= 67;
}

The mutant is alive  == has escaped

Mutation testing
example

function testIsInRetirementAge() {
-   $this->assertFalse((new Applicant('John', 30))->isInRetirementAge());
+   $this->assertFalse((new Applicant('John', 67))->isInRetirementAge());
    $this->assertTrue((new Applicant('John', 70))->isInRetirementAge());
}
public function isInRetirementAge() {
-    return $this->age > 67;
+    return $this->age >= 67;
}

The mutant is killed

Mutation

  • killed if at least 1 test fails
  • escaped if at all test pass
  • uncovered mutant is not covered
    by a test
  • fatal mutant produces a fatal error
  • timout unit tests exceed allowed timeout

a piece of code that has been mutated by mutator

Mutators

  • Binary Arithmetic
  • Boolean Substitution
  • Conditional Boundaries
  • Return values
  • Negated Conditionals
  • Increments
  • Literal Numbers

Binary Arithmetic

Source Mutation
+ -
- +
* /
/ *
% *

Binary Arithmetic

26) \Humbug\Mutator\Arithmetic\PlusEqual

--- Original
+++ New
@@ @@
             for ($n = 0; $n < $this->dimension; ++$n) {
-                $centroid->coordinates[$n] += $point->coordinates[$n];
+                $centroid->coordinates[$n] -= $point->coordinates[$n];
             }
         }
 
         for ($n = 0; $n < $this->dimension; ++$n) {
             $this->coordinates[$n] = $centroid->coordinates[$n] / $count;
         }

Boolean Substitution

Source
true

false
&&
||
and
or
!

Mutation
false

true

||

&&

or

and

Boolean Substitution

19) \Humbug\Mutator\Boolean\TrueValue

--- Original
+++ New
@@ @@
             }
-            $visited[$index] = true;
+            $visited[$index] = false;
 
             $regionSamples = $this->getSamplesInRegion($sample, $samples);
             if (count($regionSamples) >= $this->minSamples) {
                 $clusters[] = $this->expandCluster($regionSamples, $visited);
             }
         }

Conditional Boundaries

Source

>
<
>=
<=

 

Mutation >=
<=
>
<


Conditional Boundaries

50) \Humbug\Mutator\ConditionalBoundary\GreaterThan

--- Original
+++ New
@@ @@
             foreach ($this as $point) {
-                if (($sum -= $distances[$point]) > 0) {
+                if (($sum -= $distances[$point]) >= 0) {
                     continue;
                 }
 
                 $clusters[] = new Cluster($this, $point->getCoordinates());
                 break;
             }

Return values

Source

return true;
return 1.0;
return $this;
return (stmt);
return 1.0;

Mutation

return false;
return -( + 1);
return null;
(stmt); return null;
return 0.0;

 

Return values

181) \Humbug\Mutator\ReturnValue\FunctionCall

--- Original
+++ New
@@ @@
     {
-        return implode(static::$separator, $words);
+        implode(static::$separator, $words); return null;
     }
 
     protected static function strlen($text)
     {
         return function_exists('mb_strlen') ? mb_strlen($text, 'UTF-8') : strlen($text);
     }

Other types of operators

Negated Conditionals

!==     ===

Increments

++      --

Literal Numbers

changes literal int and float values

Other types of operators

Literal Numbers

--- Original
+++ New
@@ @@
     {
-        if ($clustersNumber <= 0) {
+        if ($clustersNumber <= 1) {
             throw InvalidArgumentException::invalidClustersNumber();
         }
 
         $this->clustersNumber = $clustersNumber;
         $this->initialization = $initialization;
     }

Implementation

  • The filesystem
  • The Abstract Syntax Tree (AST)
  • The bytecode / opcode

Humbug = filesystem

Metrics

  • Mutation Score Indicator (MSI):
    percentage of mutants covered & killed by tests

  • Mutation Code Coverage:
    percentage of mutants covered by tests

  • Covered Code MSI:
    percentage of killed mutants that were coverd by tests

Metrics Example

Mutation Score Indicator (MSI): 83%
Mutation Code Coverage: 98%
Covered Code MSI: 85%

https://github.com/php-ai/php-ml

651 mutations were generated:
     522 mutants were killed
      10 mutants were not covered by tests
      99 covered mutants were not detected
       0 fatal errors were encountered
      20 time outs were encountered

TDD Flow

Source: Mutation Testing Better Code by Making Bugs Filip van Laenen

Humbug

Eats Code Coverage for breakfast

Humbug

  • Running all tests for each mutation is inefficient
  • Before running tests, gather coverage data from phpunit
  • Next only run the tests that cover the mutated code
  • Stop testing a mutation as soon as at least 1 test fails
  • Sort the test on their execution time

Humbug - Installation

  • Git
  • Phar
  • Composer
composer global require 'humbug/humbug=~1.0@dev'


export PATH=~/.composer/vendor/bin:$PATH

Humbug - Configuration

humbug configure
// humbug.json.dist

{
    "timeout": 10,
    "source": {
        "directories": [
            "src"
        ]
    },
    "logs": {
        "text": "humbuglog.txt",
        "json": "humbuglog.json"
    }
}

Humbug - Run tests

// all
humbug

// custom timeput
humbug --timeout=10

// selected file
humbug --file=PrimeFactor.php

// files mask 
humbug --file=*Driver.php

Results - .txt

Humbug running test suite to generate logs and code coverage data...
Humbug has completed the initial test run successfully.
Tests: 105 Line Coverage: 95.48%
Humbug is analysing source files...
Mutation Testing is commencing on 61 files...
(.: killed, M: escaped, S: uncovered, E: fatal error, T: timed out)
............................................M............... |   60 ( 7/61)
...................................M.....M.........M....MMMM |  120 (11/61)
M.................................M.................M....... |  180 (11/61)
...M.........M....M...............M......S....S.SM.MM.M..M.. |  240 (19/61)
M..M.MM.M.MM.M.MSM.M....S......MM...MMMM..M.M.MMT...M...TMMM |  300 (21/61)
M.....TMMM.MMMMM.M.MT.M.MT.M......M....M......M.....M....... |  360 (30/61)
......M.M...............MM........M.MM...MM...M.MMM......... |  420 (49/61)
....T.........M....M.SSS......MM.....
457 mutations were generated:
     363 mutants were killed
       8 mutants were not covered by tests
      80 covered mutants were not detected
       0 fatal errors were encountered
       6 time outs were encountered

Metrics:
    Mutation Score Indicator (MSI): 81%
    Mutation Code Coverage: 98%
    Covered Code MSI: 82%

Remember that some mutants will inevitably be harmless (i.e. false positives).
Time: 3.65 minutes Memory: 10.00MB
Humbug results are being logged as TEXT to: humbuglog.txt
------
Escapes
------


1) \Humbug\Mutator\Number\IntegerValue
Diff on \Phpml\Math\Statistic\StandardDeviation::population() in /var/www/php-ml/src/Phpml/Math/Statistic/StandardDeviation.php:
--- Original
+++ New
@@ @@
         if ($sample && $n === 1) {
-            throw InvalidArgumentException::arraySizeToSmall(2);
+            throw InvalidArgumentException::arraySizeToSmall(3);
         }
 
         $mean = Mean::arithmetic($a);
         $carry = 0.0;
         foreach ($a as $val) {
             $d = $val - $mean;

Results - .txt

29) \Humbug\Mutator\Arithmetic\Division
Diff on \Phpml\Clustering\KMeans\Cluster::updateCentroid()
 in /var/www/php-ml/src/Phpml/Clustering/KMeans/Cluster.php:
--- Original
+++ New
@@ @@
         for ($n = 0; $n < $this->dimension; ++$n) {
-            $this->coordinates[$n] = $centroid->coordinates[$n] / $count;
+            $this->coordinates[$n] = $centroid->coordinates[$n] * $count;
         }
     }
 
     /**
      * @return Point[]|SplObjectStorage
      */


30) \Humbug\Mutator\ConditionalBoundary\LessThan
Diff on \Phpml\Clustering\KMeans\Space::__construct() in /var/www/php-ml/src/Phpml/Clustering/KMeans/Space.php:
--- Original
+++ New
@@ @@
     {
-        if ($dimension < 1) {
+        if ($dimension <= 1) {
             throw new LogicException('a space dimension cannot be null or negative');
         }
 
         $this->dimension = $dimension;
     }

Results - .json

{
    "summary": {
        "total": 457,
        "kills": 364,
        "escapes": 79,
        "errors": 0,
        "timeouts": 6,
        "notests": 8,
        "covered_score": 82,
        "combined_score": 81,
        "mutation_coverage": 98
    },
    "escaped": [
        {
            "file": "src\/Phpml\/Math\/Statistic\/StandardDeviation.php",
            "mutator": "\\Humbug\\Mutator\\Number\\IntegerValue",
            "class": "\\Phpml\\Math\\Statistic\\StandardDeviation",
            "method": "population",
            "line": 28,
            "diff": "--- Original\n+++ New\n@@ @@\n         if ($sample && $n === 1) {\n-            throw InvalidArgumentException::arraySizeToSmall(2);\n+            throw InvalidArgumentException::arraySizeToSmall(3);\n         }\n \n         $mean = Mean::arithmetic($a);\n         $carry = 0.0;\n         foreach ($a as $val) {\n             $d = $val - $mean;",
            "tests": [
                "test\\Phpml\\Math\\StandardDeviation\\StandardDeviationTest::testThrowExceptionOnToSmallArray"
            ],
            "stderr": "",
            "stdout": "TAP version 13"
        },
        {
            "file": "src\/Phpml\/Math\/Matrix.php"

Selected projects

Project Coverage MSI MCC CCM Time
php-ai/php-ml 95,7% 83% 98% 85% 8m
twigphp/Twig 82% 68% 86% 79% 26m
mathiasverraes/money 96% 92% 100% 92% 25s
symfony/event-dispatcher 85% 54% 69% 78% 40s

Summary

Summary

  • Write tests
  • Separate fast unit tests from slow integration tests
  • Mutation testing will improve the quality of your tests

Other languages

  • Java: Pitest (http://pitest.org/)
  • Ruby: Mutant (https://github.com/mbj/mutant)
  • C#: NinjaTurtles (https://ninjaturtles.codeplex.com/)
  • JavaScript: Stryker (https://github.com/stryker-mutator/stryker)
  • Python: Cosmic Ray (https://github.com/sixty-north/cosmic-ray)

Q&A

Thanks for listening

https://slides.com/arkadiuszkondas

https://github.com/itcraftsmanpl/phpcon-2016

Mutated PHP - mutation testing for beginners

By Arkadiusz Kondas

Made with Slides.com

Mutated PHP - mutation testing for beginners

Testowanie mutacyjne to technika pozwalając na pomiar jakości testów. Polega ona na celowym wprowadzaniu małych zmian (mutacji) w kodzie, a następnie sprawdzeniu czy przynajmniej jeden test nie przechodzi. Podczas prezentacji przedstawię koncepcję testów mutacyjnych wraz z praktycznym wdrożeniem na podstawie biblioteki padraic/humbug. Pokaże jak interpretować wyniku testów oraz jak samemu napuścić mutantów na swój kod.

  • 1,886

More from Arkadiusz Kondas