Minsk, Belarus
Infection
@infection_php
Maks Rafalko
Test Suite Metrics
What is really
Code Coverage?
Coverage Types
- Line Coverage
- Branch Coverage
- Condition Coverage
function fibonacci($n) {
+ return ($n === 0 || $n === 1) ? $n : fibonacci($n - 1) + fibonacci($n - 2);
}
fibonacci(0); // 0
0, 1, 1, 2, 3, 5, 8, 13, 21, 34 ...
Line Coverage
- Shows executed lines, not tested code
Line Coverage, 100%
Tested
Code, 60%
Issue #1
class IdGenerator
{
public function uuid(): string
{
// generating logic ...
return $id;
}
}
class UserCreatorTest extends TestCase
{
public function test(): void
{
$userCreator = new UserCreator(
new IdGenerator()
);
self::assertInstanceOf(
User::class,
$userCreator->create('Ivan', 33)
);
// .. other asserts
}
}
class UserCreator
{
private $idGenerator;
public function __construct(IdGenerator $idGenerator)
{
$this->idGenerator = $idGenerator;
}
public function create(string $name, int $age): User
{
// some logic ...
return new User(
$this->idGenerator->uuid(),
$age,
$name
);
}
}
Issue #2: Implicit Coverage
@covers
<phpunit>
<!-- ... -->
<listeners>
<listener class="Symfony\Bridge\PhpUnit\CoverageListener" />
</listeners>
</phpunit>
/**
* @covers App\UserCreator
*/
class UserCreatorTest extends TestCase
{
public function test(): void
{
$userCreator = new UserCreator(
new IdGenerator()
);
// ...
}
}
Code Coverage marks as executed not only subject under test, but *all* related code.
Use @covers annotation
Assertion Free Testing
Experiment
100% Coverage says nothing about tests quality
How to measure the effectiveness of the test suite?
Mutation Testing
Basic of Mutation Testing
- Mutation - single change of code
- $a = $b + $c;
+ $a = $b - $c;
- Mutant - mutated source code
- Mutation Operator
Original | Mutated |
---|---|
> | >= |
=== | !== |
&& | || |
foreach ($someVar as …) | foreach ([] as …); |
return true; | return false; |
... | ... |
Mutation Testing Algorithm
- Step by step Mutates the source code
- Runs tests for *each* Mutant
- Tests fail - Mutant has been killed 👍
- Tests pass - Mutant has been survived 👎
- Provides results (Mutation Score Indicator)
- Idempotency
MSI = (TotalKilledMutants / TotalMutantsCount) * 100;
class UserAgeFilter
{
private const AGE_THRESHOLD = 18;
private $ageThreshold;
public function __construct(int $ageThreshold = self::AGE_THRESHOLD)
{
$this->ageThreshold = $ageThreshold;
}
public function __invoke(array $collection)
{
return array_filter(
$collection,
function (User $user) {
return $user->getAge() >= $this->ageThreshold;
}
);
}
}
Example
public function test_it_filters_adults(): void
{
$users = [
User::withAge(20),
];
$filter = new UserAgeFilter();
$filteredUsers = $filter($users);
assertCount(1, $filteredUsers);
}
Infection
PHP Mutation Testing Framework
Result
class UserAgeFilter
{
// ...
public function __invoke(array $collection)
{
- return array_filter($collection, function (User $user) {
- return $user->getAge() >= $this->ageThreshold;
- });
+ return $collection;
}
}
Mutation #1
public function test_it_filters_adults(): void
{
$users = [
User::withAge(20),
];
$filter = new UserAgeFilter();
$filteredUsers = $filter($users);
assertCount(1, $filteredUsers);
}
public function test_it_filters_adults(): void
{
$users = [
+ User::withAge(15),
User::withAge(20),
];
$filter = new UserAgeFilter();
$filteredUsers = $filter($users);
assertCount(1, $filteredUsers);
}
class UserAgeFilter
{
// ...
public function __invoke(array $collection)
{
return array_filter(
$collection,
function (User $user) {
- return $user->getAge() >= $this->ageThreshold;
+ return $user->getAge() > $this->ageThreshold;
}
);
}
}
Mutation #2
public function test_it_filters_adults(): void
{
$users = [
User::withAge(15),
User::withAge(20),
];
$filter = new UserAgeFilter();
$filteredUsers = $filter($users);
assertCount(1, $filteredUsers);
}
public function test_it_filters_adults(): void
{
$users = [
User::withAge(15),
+ User::withAge(18),
User::withAge(20),
];
$filter = new UserAgeFilter();
$filteredUsers = $filter($users);
- assertCount(1, $filteredUsers);
+ assertCount(2, $filteredUsers);
}
class UserAgeFilter
{
// ...
public function __invoke(array $collection)
{
- return array_filter(
+ array_filter(
$collection,
function (User $user) {
return $user->getAge() >= $this->ageThreshold;
}
);
+ return null;
}
}
Mutation #3
- public function __invoke(array $collection)
+ public function __invoke(array $collection): array
Problems
-
Speed (N * t)
- N - the number of Mutants
- t - tests execution time
- Equivalent Mutants
- $a = $b * (-1);
+ $a = $b / (-1);
- $a = $b * $c;
+ $a = $b / $c;
switch ($i) {
case 1:
// ...
- break;
+ continue;
}
Equivalent Mutant
How to speed MT up?
- Execute a subset of tests for mutated line
- Run fast tests first
- Run mutations in parallel
- Avoid useless mutations
- Run MT only for changed files
- $result = [$a, $b] + [$c, $d];
+ $result = [$a, $b] - [$c, $d];
CHANGED_FILES=$(git diff origin/master --diff-filter=AM --name-only | grep src/ | paste -sd "," -);
infection --threads=4 --filter=${CHANGED_FILES}
Run for changed files only
~2.5x faster for Infection itself
How to use MT on daily bases?
- Write a new class, e.g. UserAgeFilter
- Write tests for this class
- run Infection
$ infection --threads=4 --filter=UserFilterAge.php --show-mutations
- Run on CI server
$ infection --threads=4 --min-msi=100
Supported Test Frameworks:
- PHPUnit
- phpspec
Supported Coverage Drivers:
- Xdebug
- phpdbg
- pcov
Supported PHP: 7.1+
AST
Abstract Syntax Tree
$a = $b + $c;
array(
0: Expr_Assign(
var: Expr_Variable(
name: a
)
expr: Expr_BinaryOp_Plus(
left: Expr_Variable(
name: b
)
right: Expr_Variable(
name: c
)
)
)
)
=
a
+
b
c
array(
0: Expr_Assign(
var: Expr_Variable(
name: a
)
- expr: Expr_BinaryOp_Plus(
+ expr: Expr_BinaryOp_Minus(
left: Expr_Variable(
name: b
)
right: Expr_Variable(
name: c
)
)
)
)
=
a
-
b
c
Code → AST → Mutate AST → Mutated Code
- $a = $b + $c;
+ $a = $b - $c;
final class Plus extends Mutator
{
/**
* Replaces "+" with "-"
*
* @param Node\Expr\BinaryOp\Plus $node
*
* @return Node\Expr\BinaryOp\Minus
*/
public function mutate(Node $node)
{
return new Node\Expr\BinaryOp\Minus($node->left, $node->right, $node->getAttributes());
}
protected function mutatesNode(Node $node): bool
{
if (!($node instanceof Node\Expr\BinaryOp\Plus)) {
return false;
}
if ($node->left instanceof Array_ || $node->right instanceof Array_) {
return false;
}
return true;
}
}
Conclusion
- More reliable tests
- Less bugs
- Finds dead code
- Condition coverage instead of line coverage
Links
Questions?
Mutation Testing - Infection
By born_free
Mutation Testing - Infection
Mutation Testing in PHP. Infection - Mutation testing framework
- 2,154