Infection

@infection_php

Maks Rafalko

What is really

Code Coverage?

@maks_rafalko

Coverage Types

Line Coverage
Branch Coverage
Condition Coverage

0, 1, 1, 2, 3, 5, 8, 13, 21, 34 ...

@maks_rafalko

function fibonacci(int $n) {
+    return ($n === 0 || $n === 1) ? $n : fibonacci($n - 1) + fibonacci($n - 2);
}

fibonacci(0); // 0

Path Coverage in PHPUnit

@maks_rafalko

<?xml version="1.0" encoding="UTF-8"?>
<phpunit xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:noNamespaceSchemaLocation="https://schema.phpunit.de/9.3/phpunit.xsd">
    <testsuites>
        <testsuite name="default">
            <directory>tests</directory>
        </testsuite>
    </testsuites>

    <coverage pathCoverage="true">
        <include>
            <directory suffix=".php">src</directory>
        </include>
    </coverage>
</phpunit>

https://doug.codes/php-code-coverage

Line Coverage

Shows executed lines, not tested code

Line Coverage, 100%

Tested

Code, 60%

Issue #1

@maks_rafalko

class IdGenerator
{
    public function uuid(): string
    {
        // generating logic ...

        return $id;
    }
}

Issue #2: implicit coverage

@maks_rafalko

class UserCreator
{
    public function __construct(
        private IdGenerator $idGenerator
    ) {}

    public function create(string $name, int $age): User
    {
        // some logic ...

        return new User(
            $this->idGenerator->uuid(),
            $age,
            $name
        );
    }
}

@maks_rafalko

Issue #2: implicit coverage

class UserCreatorTest extends TestCase
{
    public function test(): void
    {
        $userCreator = new UserCreator(
            new IdGenerator()
        );

        self::assertInstanceOf(
            User::class,
            $userCreator->create('Ivan', 33)
        );

        // .. other asserts
    }
}

@maks_rafalko

Issue #2: implicit coverage

@covers

<phpunit>
    <!-- ... -->

    <listeners>
        <listener class="Symfony\Bridge\PhpUnit\CoverageListener" />
    </listeners>
</phpunit>

/**
 * @covers App\UserCreator
 */
class UserCreatorTest extends TestCase
{
    public function test(): void
    {
        $userCreator = new UserCreator(
            new IdGenerator()
        );

        // ...
    }
}

@maks_rafalko

Code Coverage marks as executed not only subject under test, but all related code.

Use @covers annotation

@maks_rafalko

https://martinfowler.com/bliki/TestCoverage.html

@maks_rafalko

Coverage shows what code is not tested

https://martinfowler.com/bliki/AssertionFreeTesting.html

Assertion Free Testing

@maks_rafalko

Experiment

@maks_rafalko

100% Coverage says nothing about tests quality

@maks_rafalko

How to measure the effectiveness of the test suite?

Mutation Testing

@maks_rafalko

Basic of Mutation Testing

Mutation - single change of code

- $a = $b + $c;
+ $a = $b - $c;

Mutant - mutated source code

Mutation Operator

Original	Mutated
>	>=
===	!==
&&	\|\|
'/^test$/'	'/test$/' and '/^test/'
return true;	return false;
foreach ($someVar as …)	foreach ([] as …);
...	...

@maks_rafalko

Mutation Testing Algorithm

Step by step Mutates the source code
Runs tests for *each* Mutant
Tests fail - Mutant has been killed 👍
Tests pass - Mutant has been survived 👎
Provides results (Mutation Score Indicator)
Idempotency

MSI = (TotalKilledMutants / TotalMutantsCount) * 100;

@maks_rafalko

Test # Mutation #	1	2	3	4	Result
1 ("+" ➝ "-")	✅	✅	✅	❌	Killed
2 (">" ➝ ">=")	⏳
3 ("==" ➝ "!=)
4 (true ➝ false)

@maks_rafalko

Test # Mutation #	1	2	3	4	5	6	7	8	Result
1 ("+" ➝ "-")	✅	✅	✅	❌					Killed
2 (">" ➝ ">=")	✅	✅	✅	✅	✅	✅	✅	✅	Escaped
3 ("==" ➝ "!=)	⏳
4 (true ➝ false)

@maks_rafalko

Test # Mutation #	1	2	3	4	5	6	7	8	Result
1 ("+" ➝ "-")	✅	✅	✅	❌					Killed
2 (">" ➝ ">=")	✅	✅	✅	✅	✅	✅	✅	✅	Escaped
3 ("==" ➝ "!=)	✅	❌							Killed
4 (true ➝ false)	✅	✅	❌

@maks_rafalko

class UserAgeFilter
{
    private const AGE_THRESHOLD = 18;

    private $ageThreshold;

    public function __construct(int $ageThreshold = self::AGE_THRESHOLD)
    {
        $this->ageThreshold = $ageThreshold;
    }

    /** @param array<int, User> $collection */
    public function __invoke(array $collection)
    {
        return array_filter(
            $collection,
            function (User $user) {
                return $user->getAge() >= $this->ageThreshold;
            }
        );
    }
}

Example

public function test_it_filters_adults(): void
{
    $users = [
        User::withAge(20),
    ];

    $filter = new UserAgeFilter();

    $filteredUsers = $filter($users);

    assertCount(1, $filteredUsers);
}

@maks_rafalko

Infection

PHP Mutation Testing Library

@maks_rafalko

Result

class UserAgeFilter
{
    // ...

    /** @param array<int, User> $collection */
    public function __invoke(array $collection)
    {
-        return array_filter($collection, function (User $user) {
-            return $user->getAge() >= $this->ageThreshold;
-        });
+        return $collection;
    }
}

Mutation #1

@maks_rafalko

public function test_it_filters_adults(): void
{
    $users = [
        User::withAge(20),
    ];

    $filter = new UserAgeFilter();

    $filteredUsers = $filter($users);

    assertCount(1, $filteredUsers);
}

@maks_rafalko

public function test_it_filters_adults(): void
{
    $users = [
+       User::withAge(15),
        User::withAge(20),
    ];

    $filter = new UserAgeFilter();

    $filteredUsers = $filter($users);

    assertCount(1, $filteredUsers);
}

@maks_rafalko

class UserAgeFilter
{
    // ...

    /** @param array<int, User> $collection */
    public function __invoke(array $collection)
    {
        return array_filter(
            $collection,
            function (User $user) {
-                return $user->getAge() >= $this->ageThreshold;
+                return $user->getAge() > $this->ageThreshold;
            }
        );
    }
}

Mutation #2

@maks_rafalko

public function test_it_filters_adults(): void
{
    $users = [
        User::withAge(15),
        User::withAge(20),
    ];

    $filter = new UserAgeFilter();

    $filteredUsers = $filter($users);

    assertCount(1, $filteredUsers);
}

@maks_rafalko

public function test_it_filters_adults(): void
{
    $users = [
        User::withAge(15),
+       User::withAge(18),
        User::withAge(20),
    ];

    $filter = new UserAgeFilter();

    $filteredUsers = $filter($users);

-   assertCount(1, $filteredUsers);
+   assertCount(2, $filteredUsers);
}

@maks_rafalko

Mutation Testing finds useless code

@maks_rafalko

public function deactivateInactiveUsers(): void
{
    $users = $this->userRepository->findInactiveUsers();

    foreach ($users as $user) {
        $user->deactivate();

-       $this->entityManager->persist($user);
    }

    $this->entitymanager->flush();
}

Problems

Speed (N * t)
- N - the number of Mutants
- t - tests execution time
Equivalent Mutants

@maks_rafalko

- $a = $b * (-1);
+ $a = $b / (-1);

- $a = $b * $c;
+ $a = $b / $c;

How to speed MT up?

Execute a subset of tests for mutated line
Run the fastest tests first
Run mutations in parallel
Avoid useless mutations
Run MT only for changed files

@maks_rafalko

- $result = [$a, $b] + [$c, $d];
+ $result = [$a, $b] - [$c, $d];

https://infection.github.io/guide/command-line-options.html#git-diff-filter

Run for changed files only

@maks_rafalko

infection --git-diff-filter=AM --git-diff-base=master

Run killing tests first

@maks_rafalko

Test # Mutation #	1	2	3	Result
1	✅ 0.132s	✅ 0.387s	❌ 0.399s	Killed
2	✅	✅	✅	Escaped
3	✅	❌		Killed
4	✅	✅	❌

Run killing tests first

@maks_rafalko

Test # Mutation #	1	2	3	Result
1	❌ 0.399s	✅ 0.387s	✅ 0.132s	Killed
2	✅	✅	✅	Escaped
3	❌	✅		Killed
4	❌	✅	✅

Run killing tests first

@maks_rafalko

Test # Mutation #	1	2	3	Result
1	❌ 0.399s			Killed
2	✅	✅	✅	Escaped
3	❌			Killed
4	❌

Run killing tests first

@maks_rafalko

- Time: 1m 49s. Memory: 0.07GB (first run)
+ Time: 38s. Memory: 0.07GB (second run)

How to use MT on daily bases?

Write a new feature, cover by tests
run Infection

$ infection --git-diff-filter=AM --threads=4 --show-mutations

Run on CI server

$ infection --min-covered-msi=100

@maks_rafalko

5) /srv/api/src/User/Status/StatusTransitioner.php:22    [M] PublicVisibility

--- Original
+++ New
@@ @@

-    public function isAllowedTransitionTo(User $user, UserStatus $toStatus) : bool
+    protected function isAllowedTransitionTo(User $user, UserStatus $toStatus) : bool
     {
         if ($user->isDraft()) {
             return $toStatus->isActive() || $toStatus->isDeleted();