Minsk, Belarus

Infection

@infection_php

Maks Rafalko

Test Suite Metrics

What is really

Code Coverage?

Coverage Types

 

  • Line Coverage
  • Branch Coverage
  • Condition Coverage
function fibonacci($n) {
+    return ($n === 0 || $n === 1) ? $n : fibonacci($n - 1) + fibonacci($n - 2);
}

fibonacci(0); // 0

0, 1, 1, 2, 3, 5, 8, 13, 21, 34 ...

Line Coverage

  • Shows executed lines, not tested code

Line Coverage, 100%

Tested

Code, 60%

Issue #1

class IdGenerator
{
    public function uuid(): string
    {
        // generating logic ...

        return $id;
    }
}
class UserCreatorTest extends TestCase
{
    public function test(): void
    {
        $userCreator = new UserCreator(
            new IdGenerator()
        );

        self::assertInstanceOf(
            User::class,
            $userCreator->create('Ivan', 33)
        );

        // .. other asserts
    }
}
class UserCreator
{
    private $idGenerator;

    public function __construct(IdGenerator $idGenerator)
    {
        $this->idGenerator = $idGenerator;
    }

    public function create(string $name, int $age): User
    {
        // some logic ...

        return new User(
            $this->idGenerator->uuid(),
            $age,
            $name
        );
    }
}

Issue #2: Implicit Coverage

@covers

<phpunit>
    <!-- ... -->

    <listeners>
        <listener class="Symfony\Bridge\PhpUnit\CoverageListener" />
    </listeners>
</phpunit>
/**
 * @covers App\UserCreator
 */
class UserCreatorTest extends TestCase
{
    public function test(): void
    {
        $userCreator = new UserCreator(
            new IdGenerator()
        );

        // ...
    }
}

Code Coverage marks  as executed not only subject under test, but *all* related code.

 

Use @covers annotation

https://martinfowler.com/bliki/TestCoverage.html

https://martinfowler.com/bliki/AssertionFreeTesting.html

Assertion Free Testing

Experiment

100% Coverage says nothing about tests quality

How to measure​ the effectiveness of the test suite?

Mutation Testing

Basic of Mutation Testing

  • Mutation - single change of code
- $a = $b + $c;
+ $a = $b - $c;
  • Mutant - mutated source code
  • Mutation Operator
Original Mutated
> >=
=== !==
&& ||
foreach ($someVar as …) foreach ([] as …);
return true; return false;
... ...

Mutation Testing Algorithm

  • Step by step Mutates the source code
  • Runs tests for *each* Mutant
  • Tests fail - Mutant has been killed 👍
  • Tests pass - Mutant has been survived 👎
  • Provides results (Mutation Score Indicator)
  • Idempotency
MSI = (TotalKilledMutants / TotalMutantsCount) * 100;
class UserAgeFilter
{
    private const AGE_THRESHOLD = 18;

    private $ageThreshold;

    public function __construct(int $ageThreshold = self::AGE_THRESHOLD)
    {
        $this->ageThreshold = $ageThreshold;
    }

    public function __invoke(array $collection)
    {
        return array_filter(
            $collection,
            function (User $user) {
                return $user->getAge() >= $this->ageThreshold;
            }
        );
    }
}

Example

public function test_it_filters_adults(): void
{
    $users = [
        User::withAge(20),
    ];

    $filter = new UserAgeFilter();

    $filteredUsers = $filter($users);

    assertCount(1, $filteredUsers);
}

Infection

PHP Mutation Testing Framework

Result

class UserAgeFilter
{
    // ...

    public function __invoke(array $collection)
    {
-        return array_filter($collection, function (User $user) {
-            return $user->getAge() >= $this->ageThreshold;
-        });
+        return $collection;
    }
}

Mutation #1

public function test_it_filters_adults(): void
{
    $users = [
        User::withAge(20),
    ];

    $filter = new UserAgeFilter();

    $filteredUsers = $filter($users);

    assertCount(1, $filteredUsers);
}

public function test_it_filters_adults(): void
{
    $users = [
+       User::withAge(15),
        User::withAge(20),
    ];

    $filter = new UserAgeFilter();

    $filteredUsers = $filter($users);

    assertCount(1, $filteredUsers);
}

class UserAgeFilter
{
    // ...

    public function __invoke(array $collection)
    {
        return array_filter(
            $collection,
            function (User $user) {
-                return $user->getAge() >= $this->ageThreshold;
+                return $user->getAge() > $this->ageThreshold;
            }
        );
    }
}

Mutation #2

public function test_it_filters_adults(): void
{
    $users = [
        User::withAge(15),
        User::withAge(20),
    ];

    $filter = new UserAgeFilter();

    $filteredUsers = $filter($users);

    assertCount(1, $filteredUsers);
}

public function test_it_filters_adults(): void
{
    $users = [
        User::withAge(15),
+       User::withAge(18),
        User::withAge(20),
    ];

    $filter = new UserAgeFilter();

    $filteredUsers = $filter($users);

-   assertCount(1, $filteredUsers);
+   assertCount(2, $filteredUsers);
}

class UserAgeFilter
{
    // ...

    public function __invoke(array $collection)
    {
-       return array_filter(
+       array_filter(
            $collection,
            function (User $user) {
                return $user->getAge() >= $this->ageThreshold;
            }
        );
+       return null;   
    }
}

Mutation #3

-    public function __invoke(array $collection)
+    public function __invoke(array $collection): array

Problems

  • Speed (N * t)
    • N - the number of Mutants
    • ​t - tests execution time​
  • Equivalent Mutants
- $a = $b * (-1);
+ $a = $b / (-1);
- $a = $b * $c;
+ $a = $b / $c;
switch ($i) {
    case 1:
        // ...
-       break;
+       continue;
}

Equivalent Mutant

How to speed MT up?

  • Execute a subset of tests for mutated line
  • Run fast tests first
  • Run mutations in parallel
  • Avoid useless mutations
  • Run MT only for changed files
- $result = [$a, $b] + [$c, $d];
+ $result = [$a, $b] - [$c, $d];
CHANGED_FILES=$(git diff origin/master --diff-filter=AM --name-only | grep src/ | paste -sd "," -);


infection --threads=4 --filter=${CHANGED_FILES}

https://infection.github.io/guide/how-to.html#How-to-run-Infection-only-for-changed-files

Run for changed files only

~2.5x faster for Infection itself

How to use MT on daily bases?

  • Write a new class, e.g. UserAgeFilter
  • Write tests for this class
  • run Infection
$ infection --threads=4 --filter=UserFilterAge.php --show-mutations
  • Run on CI server
$ infection --threads=4 --min-msi=100

Supported Test Frameworks:

  • PHPUnit
  • phpspec

 

Supported Coverage Drivers:

  • Xdebug
  • phpdbg
  • pcov

 

Supported PHP: 7.1+

AST

Abstract Syntax Tree

$a = $b + $c;

array(
    0: Expr_Assign(
        var: Expr_Variable(
            name: a
        )
        expr: Expr_BinaryOp_Plus(
            left: Expr_Variable(
                name: b
            )
            right: Expr_Variable(
                name: c
            )
        )
    )
)

=

a

+

b

c

https://github.com/nikic/PHP-Parser

array(
    0: Expr_Assign(
        var: Expr_Variable(
            name: a
        )
-        expr: Expr_BinaryOp_Plus(
+        expr: Expr_BinaryOp_Minus(
            left: Expr_Variable(
                name: b
            )
            right: Expr_Variable(
                name: c
            )
        )
    )
)

=

a

-

b

c

Code → AST → Mutate AST → Mutated Code

- $a = $b + $c;
+ $a = $b - $c;
final class Plus extends Mutator
{
    /**
     * Replaces "+" with "-"
     *
     * @param Node\Expr\BinaryOp\Plus $node
     *
     * @return Node\Expr\BinaryOp\Minus
     */
    public function mutate(Node $node)
    {
        return new Node\Expr\BinaryOp\Minus($node->left, $node->right, $node->getAttributes());
    }

    protected function mutatesNode(Node $node): bool
    {
        if (!($node instanceof Node\Expr\BinaryOp\Plus)) {
            return false;
        }

        if ($node->left instanceof Array_ || $node->right instanceof Array_) {
            return false;
        }

        return true;
    }
}

Conclusion

  • More reliable tests
  • Less bugs
  • Finds dead code
  • Condition coverage instead of line coverage

Links

 

Questions?