Positive Mutation

how mutation tests will improve your work

ARKADIUSZ KONDAS

Data Scientist

@ Buddy.Works

Code Craftsman

Blogger

Ultra Runner

@ ArkadiuszKondas

github.com/akondas

arkadiuszkondas.com

Agenda:

  1. Story
  2. Theory
  3. Demo

Story

Coders Guild

Systems

Testers Guild

Coverage Report

Who will guard the guards themselves?

Introduction of mutations

Eternal prosperity

Theory

Code Coverage

Measure how thoroughly tests exercise application

Code Coverage

Line Coverage / Statement coverage

Lines: 4/5          Coverage: 80%

Code Coverage

Line Coverage / Statement coverage

Code Coverage

Function and Method Coverage

Code Coverage

Class and Trait Coverage

Code Coverage

Branch/patch coverage

class AgeVerification
{
    public function passes(int $age): bool
    {
        if($age <= 0) {
            throw new \InvalidArgumentException('Age cannot be negative.');
        }

        return $age >= 18 && $age <= 99;
    }
}

Code Coverage

Branch/patch coverage

Code Coverage

Change Risk Anti-Patterns (CRAP) Index

The Change Risk Anti-Patterns (CRAP) Index is calculated based on the cyclomatic complexity and code coverage of a unit of code. Code that is not too complex and has an adequate test coverage will have a low CRAP index.

 

The CRAP index can be lowered by writing tests and by refactoring the code to lower its complexity.

Code Coverage

Change Risk Anti-Patterns (CRAP) Index

CRAP1(m) = comp(m)^2 * (1 – cov(m)/100)^3 + comp(m)

Code Coverage

Change Risk Anti-Patterns (CRAP) Index

Code Coverage

Metrics are a sharp tool. Dont't poke you eye out.

Code Coverage

<?php

declare (strict_types = 1);

class GoldCustomer implements CustomerSpecification
{
    /**
     * @param Customer $candidate
     *
     * @return bool
     */
    public function isSatisfiedBy(Customer $candidate): bool
    {
        return $candidate->getTotalPurchases() >= 100;
    }
}

Code Coverage

<?php

declare (strict_types = 1);

class GoldCustomerTest extends \PHPUnit_Framework_TestCase
{

    public function testIsSatisfiedByCustomer()
    {
        $customer = new Customer();
        $goldCustomer = new GoldCustomer();

        $this->assertFalse($goldCustomer->isSatisfiedBy($customer));
    }

}

100% Code Coverage, but ...

Code Coverage

/**
 * @param int $totalPurchases
 * @param bool $expected
 *
 * @dataProvider customerProvider
 */
public function testIsSatisfiedReturnsCorrectResult(int $totalPurchases, bool $expected)
{
    $customer = new Customer($totalPurchases);
    $goldCustomer = new GoldCustomer();

    $this->assertEquals($expected, $goldCustomer->isSatisfiedBy($customer));
}

/**
 * @return array
 */
public function customerProvider()
{
    return [
        [1, false],
        [99, false],
        [100, true],
        [200, true]
    ];
}

Code Coverage

Mutation Testing
to the rescue

Mutation testing

A tool to provide insight in stability of your code

Mutation testing

  • run test siut and check if pass
  • generate all possible mutations
  • for each mutation run testsiut and check if fails
  • evaluate results

Mutation testing
example

public function isInRetirementAge() {
    return $this->age > 67;
}
function testIsInRetirementAge() {
    $this->assertFalse((new Applicant('John', 30))->isInRetirementAge());
    $this->assertTrue((new Applicant('John', 70))->isInRetirementAge());
}
public function isInRetirementAge() {
-    return $this->age > 67;
+    return $this->age >= 67;
}

The mutant is alive  == has escaped

Mutation testing
example

function testIsInRetirementAge() {
-   $this->assertFalse((new Applicant('John', 30))->isInRetirementAge());
+   $this->assertFalse((new Applicant('John', 67))->isInRetirementAge());
    $this->assertTrue((new Applicant('John', 70))->isInRetirementAge());
}
public function isInRetirementAge() {
-    return $this->age > 67;
+    return $this->age >= 67;
}

The mutant is killed

Mutation

  • killed if at least 1 test fails
  • escaped if at all test pass
  • uncovered mutant is not covered
    by a test
  • fatal mutant produces a fatal error
  • timeout unit tests exceed allowed timeout
  • skipped could not be tested

a piece of code that has been mutated by mutator

Mutators

Binary Arithmetic

Binary Arithmetic

26) \Humbug\Mutator\Arithmetic\PlusEqual

--- Original
+++ New
@@ @@
             for ($n = 0; $n < $this->dimension; ++$n) {
-                $centroid->coordinates[$n] += $point->coordinates[$n];
+                $centroid->coordinates[$n] -= $point->coordinates[$n];
             }
         }
 
         for ($n = 0; $n < $this->dimension; ++$n) {
             $this->coordinates[$n] = $centroid->coordinates[$n] / $count;
         }

Mutators

Metrics

  • Mutation Score Indicator (MSI):
    percentage of mutants covered & killed by tests

  • Mutation Code Coverage:
    percentage of mutants covered by tests

  • Covered Code MSI:
    percentage of killed mutants that were coverd by tests

Metrics Example

https://github.com/munusphp/munus

506 mutations were generated:
     401 mutants were killed
      47 mutants were not covered by tests
      45 covered mutants were not detected
       4 errors were encountered
       9 time outs were encountered
       0 mutants required more time than configured

Workflow

Source: Mutation Testing Better Code by Making Bugs Filip van Laenen

CHANGED_FILES=$(git diff origin/master --diff-filter=AM --name-only | grep src/ | paste -sd "," -);
INFECTION_FILTER="--filter=${CHANGED_FILES} --ignore-msi-with-no-mutations";

infection --threads=4 $INFECTION_FILTER

Run mutation testing on changesets

Implementation

  • The filesystem
  • The Abstract Syntax Tree (AST)
  • The bytecode / opcode

Demo

source: https://newsmeter.in/teslas-cybertruck-armour-glass-demo-goes-wrong/

Infection

Eats Code Coverage for breakfast

Infection

  • Running all tests for each mutation is inefficient
  • Before running tests, gather coverage data from phpunit
  • Next only run the tests that cover the mutated code
  • Stop testing a mutation as soon as at least 1 test fails
  • Sort the test on their execution time

Let's run CLI ...

Summary

Summary

  • Write tests
  • Separate fast unit tests from slow integration tests
  • False positive will show up
  • Mutation testing will improve the quality of your tests

Other languages

  • Java: Pitest (http://pitest.org/)
  • Ruby: Mutant (https://github.com/mbj/mutant)
  • C#: NinjaTurtles* (https://ninjaturtles.codeplex.com/)
  • JavaScript: Stryker (https://github.com/stryker-mutator/stryker)
  • Python: mutmut
    (https://github.com/boxed/mutmut)

Q&A

Resources/attributions

  • https://infection.github.io/guide/
  • https://doug.codes/php-code-coverage
  • https://phpunit.readthedocs.io/en/9.3/code-coverage-analysis.html
  • http://crestweb.cs.ucl.ac.uk/resources/mutation_testing_repository/
  • https://github.com/mariuszgil/aggregates-by-example
  • https://pixabay.com/
  • Illustration by Freepik Stories: https://stories.freepik.com/web

Thanks for listening

@ ArkadiuszKondas

github.com/akondas

Made with Slides.com