Infection
@infection_php
Maks Rafalko
What is really
Code Coverage?
Coverage Types
- Line Coverage
- Branch Coverage
- Condition Coverage
0, 1, 1, 2, 3, 5, 8, 13, 21, 34 ...
function fibonacci(int $n) {
+ return ($n === 0 || $n === 1) ? $n : fibonacci($n - 1) + fibonacci($n - 2);
}
fibonacci(0); // 0Path Coverage in PHPUnit
<?xml version="1.0" encoding="UTF-8"?>
<phpunit xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="https://schema.phpunit.de/9.3/phpunit.xsd">
<testsuites>
<testsuite name="default">
<directory>tests</directory>
</testsuite>
</testsuites>
<coverage pathCoverage="true">
<include>
<directory suffix=".php">src</directory>
</include>
</coverage>
</phpunit>
Line Coverage
- Shows executed lines, not tested code
Line Coverage, 100%
Tested
Code, 60%
Issue #1
class IdGenerator
{
public function uuid(): string
{
// generating logic ...
return $id;
}
}Issue #2: implicit coverage
class UserCreator
{
public function __construct(
private IdGenerator $idGenerator
) {}
public function create(string $name, int $age): User
{
// some logic ...
return new User(
$this->idGenerator->uuid(),
$age,
$name
);
}
}Issue #2: implicit coverage
class UserCreatorTest extends TestCase
{
public function test(): void
{
$userCreator = new UserCreator(
new IdGenerator()
);
self::assertInstanceOf(
User::class,
$userCreator->create('Ivan', 33)
);
// .. other asserts
}
}
Issue #2: implicit coverage
@covers
<phpunit>
<!-- ... -->
<listeners>
<listener class="Symfony\Bridge\PhpUnit\CoverageListener" />
</listeners>
</phpunit>/**
* @covers App\UserCreator
*/
class UserCreatorTest extends TestCase
{
public function test(): void
{
$userCreator = new UserCreator(
new IdGenerator()
);
// ...
}
}
Code Coverage marks as executed not only subject under test, but *all* related code.
Use @covers annotation
Coverage shows what code is not tested
Assertion Free Testing
Experiment
100% Coverage says nothing about tests quality
How to measureβ the effectiveness of the test suite?
Mutation Testing
Basic of Mutation Testing
- Mutation - single change of code
- $a = $b + $c;
+ $a = $b - $c;- Mutant - mutated source code
- Mutation Operator
| Original | Mutated |
|---|---|
| > | >= |
| === | !== |
| && | || |
| '/^test$/' | '/test$/' and '/^test/' |
| return true; | return false; |
| foreach ($someVar as β¦) | foreach ([] as β¦); |
| ... | ... |
Mutation Testing Algorithm
- Step by step Mutates the source code
- Runs tests for *each* Mutant
- Tests fail - Mutant has been killed π
- Tests pass - Mutant has been survived π
- Provides results (Mutation Score Indicator)
- Idempotency
MSI = (TotalKilledMutants / TotalMutantsCount) * 100;
| Test # Mutation # |
1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | Result |
|---|---|---|---|---|---|---|---|---|---|
| 1 ("+" β "-") | β | β | β | β | Killed | ||||
| 2 (">" β ">=") | β³ | ||||||||
| 3 ("==" β "!=) | |||||||||
| 4 (true β false) |
| Test # Mutation # |
1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | Result |
|---|---|---|---|---|---|---|---|---|---|
| 1 ("+" β "-") | β | β | β | β | Killed | ||||
| 2 (">" β ">=") | β | β | β | β | β | β | β | β | Escaped |
| 3 ("==" β "!=) | β³ | ||||||||
| 4 (true β false) |
| Test # Mutation # |
1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | Result |
|---|---|---|---|---|---|---|---|---|---|
| 1 ("+" β "-") | β | β | β | β | Killed | ||||
| 2 (">" β ">=") | β | β | β | β | β | β | β | β | Escaped |
| 3 ("==" β "!=) | β | β | Killed | ||||||
| 4 (true β false) | β | β | β |
class UserAgeFilter
{
private const AGE_THRESHOLD = 18;
private $ageThreshold;
public function __construct(int $ageThreshold = self::AGE_THRESHOLD)
{
$this->ageThreshold = $ageThreshold;
}
/** @param array<int, User> $collection */
public function __invoke(array $collection)
{
return array_filter(
$collection,
function (User $user) {
return $user->getAge() >= $this->ageThreshold;
}
);
}
}Example
public function test_it_filters_adults(): void
{
$users = [
User::withAge(20),
];
$filter = new UserAgeFilter();
$filteredUsers = $filter($users);
assertCount(1, $filteredUsers);
}
Infection
PHP Mutation Testing Library
Result
class UserAgeFilter
{
// ...
/** @param array<int, User> $collection */
public function __invoke(array $collection)
{
- return array_filter($collection, function (User $user) {
- return $user->getAge() >= $this->ageThreshold;
- });
+ return $collection;
}
}Mutation #1
public function test_it_filters_adults(): void
{
$users = [
User::withAge(20),
];
$filter = new UserAgeFilter();
$filteredUsers = $filter($users);
assertCount(1, $filteredUsers);
}
public function test_it_filters_adults(): void
{
$users = [
+ User::withAge(15),
User::withAge(20),
];
$filter = new UserAgeFilter();
$filteredUsers = $filter($users);
assertCount(1, $filteredUsers);
}
class UserAgeFilter
{
// ...
/** @param array<int, User> $collection */
public function __invoke(array $collection)
{
return array_filter(
$collection,
function (User $user) {
- return $user->getAge() >= $this->ageThreshold;
+ return $user->getAge() > $this->ageThreshold;
}
);
}
}Mutation #2
public function test_it_filters_adults(): void
{
$users = [
User::withAge(15),
User::withAge(20),
];
$filter = new UserAgeFilter();
$filteredUsers = $filter($users);
assertCount(1, $filteredUsers);
}
public function test_it_filters_adults(): void
{
$users = [
User::withAge(15),
+ User::withAge(18),
User::withAge(20),
];
$filter = new UserAgeFilter();
$filteredUsers = $filter($users);
- assertCount(1, $filteredUsers);
+ assertCount(2, $filteredUsers);
}
Mutation Testing finds useless code
public function deactivateInactiveUsers(): void
{
$users = $this->userRepository->findInactiveUsers();
foreach ($users as $user) {
$user->deactivate();
- $this->entityManager->persist($user);
}
$this->entitymanager->flush();
}Problems
-
Speed (N * t)
- N - the number of Mutants
- βt - tests execution timeβ
- Equivalent Mutants
- $a = $b * (-1);
+ $a = $b / (-1);- $a = $b * $c;
+ $a = $b / $c;How to speed MT up?
- Execute a subset of tests for mutated line
- Run the fastest tests first
- Run mutations in parallel
- Avoid useless mutations
- Run MT only for changed files
- $result = [$a, $b] + [$c, $d];
+ $result = [$a, $b] - [$c, $d];Run for changed files only
infection --git-diff-filter=AM --git-diff-base=masterRun killing tests first
| Test # Mutation # |
1 | 2 | 3 | Result |
|---|---|---|---|---|
| 1 | β 0.132s | β
0.387s |
β 0.399s |
Killed |
| 2 | β | β | β | Escaped |
| 3 | β | β | Killed | |
| 4 | β | β | β |
Run killing tests first
| Test # Mutation # |
1 | 2 | 3 | Result |
|---|---|---|---|---|
| 1 | β 0.399s |
β
0.387s |
β 0.132s | Killed |
| 2 | β | β | β | Escaped |
| 3 | β | β | Killed | |
| 4 | β | β | β |
Run killing tests first
| Test # Mutation # |
1 | 2 | 3 | Result |
|---|---|---|---|---|
| 1 | β 0.399s |
Killed | ||
| 2 | β | β | β | Escaped |
| 3 | β | Killed | ||
| 4 | β |
Run killing tests first
- Time: 1m 49s. Memory: 0.07GB (first run)
+ Time: 38s. Memory: 0.07GB (second run)How to use MT on daily bases?
- Write a new feature, cover by tests
- run Infection
$ infection --git-diff-filter=AM --threads=4 --show-mutations- Run on CI server
$ infection --min-covered-msi=1005) /srv/api/src/User/Status/StatusTransitioner.php:22 [M] PublicVisibility
--- Original
+++ New
@@ @@
- public function isAllowedTransitionTo(User $user, UserStatus $toStatus) : bool
+ protected function isAllowedTransitionTo(User $user, UserStatus $toStatus) : bool
{
if ($user->isDraft()) {
return $toStatus->isActive() || $toStatus->isDeleted();How MT helps with code review?
Supported Test Frameworks:
- PHPUnit
- Pest
- Codeception
- phpspec
Supported Coverage Drivers:
- Xdebug
- phpdbg
- pcov
Supported PHP: 7.1+
Infection Playground
Infection Playground
Conclusion
- More reliable tests
- Less bugs
- Helps with Code Review
- Finds dead and useless code
- Condition coverage instead of line coverage
Links
Questions?
Mutation Testing - Infection 2021
By born_free
Mutation Testing - Infection 2021
Mutation Testing in PHP. Infection - Mutation testing framework
