![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/8711785/Титульник.png)
Мутационное тестирование в PHP
Макс Рафалко
Минск, Беларусь
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6181302/itransition.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/4484041/logo.png)
Infection
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6181303/twitter.png)
@infection_php
Макс Рафалко
Что же такое
Code Coverage?
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6181303/twitter.png)
Типы Code Coverage
- Line Coverage
- Branch Coverage
- Condition Coverage
function fibonacci(int $n) {
+ return ($n === 0 || $n === 1) ? $n : fibonacci($n - 1) + fibonacci($n - 2);
}
fibonacci(0); // 0
0, 1, 1, 2, 3, 5, 8, 13, 21, 34 ...
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6181303/twitter.png)
Path Coverage в PHPUnit
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6181303/twitter.png)
<?xml version="1.0" encoding="UTF-8"?>
<phpunit xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="https://schema.phpunit.de/9.3/phpunit.xsd">
<testsuites>
<testsuite name="default">
<directory>tests</directory>
</testsuite>
</testsuites>
<coverage pathCoverage="true">
<include>
<directory suffix=".php">src</directory>
</include>
</coverage>
</phpunit>
Тип Line Coverage
- Показывает выполненные строки, а не протестированные
Покрытие кода, 100%
Протестированный
код, 60%
Проблема #1
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6181303/twitter.png)
class IdGenerator
{
public function uuid(): string
{
// generating logic ...
return $id;
}
}
Проблема #2: неявное покрытие
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6181303/twitter.png)
class UserCreator
{
public function __construct(
private IdGenerator $idGenerator
) {}
public function create(string $name, int $age): User
{
// some logic ...
return new User(
$this->idGenerator->uuid(),
$age,
$name
);
}
}
Проблема #2: неявное покрытие
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6181303/twitter.png)
class UserCreatorTest extends TestCase
{
public function test(): void
{
$userCreator = new UserCreator(
new IdGenerator()
);
self::assertInstanceOf(
User::class,
$userCreator->create('Ivan', 33)
);
// .. other asserts
}
}
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/4476227/mt-pass-100.png)
Проблема #2: неявное покрытие
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6072510/thumbs-down-sign_1f44e.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6181303/twitter.png)
@covers
<phpunit>
<!-- ... -->
<listeners>
<listener class="Symfony\Bridge\PhpUnit\CoverageListener" />
</listeners>
</phpunit>
/**
* @covers App\UserCreator
*/
class UserCreatorTest extends TestCase
{
public function test(): void
{
$userCreator = new UserCreator(
new IdGenerator()
);
// ...
}
}
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/4476224/mt-build-passing.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6072549/codecov_65.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6111419/1f44d-1f3fe__1_.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6181303/twitter.png)
Code Coverage помечает как выполненный весь "затронутый" код
Используйте @covers-аннотацию
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6181303/twitter.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6099856/fowler.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6181303/twitter.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/8675091/phpunit-html-coverage.png)
Coverage показывает, какой код не протестирован
"Assertion Free"-тестирование
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6181303/twitter.png)
Эксперимент
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6107882/bad_coverage.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6181303/twitter.png)
100% Coverage ничего не говорит о качестве тестов
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6107935/bad_coverage_2.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/4476227/mt-pass-100.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6181303/twitter.png)
Как измерить качество автоматических тестов?
Мутационное Тестирование
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6181303/twitter.png)
Терминология Мутационного Тестирования
- Mutation - единичное изменение кода
- $a = $b + $c;
+ $a = $b - $c;
- Mutant - мутированный исходный код
- Mutation Operator
Оригинал | Мутация |
---|---|
> | >= |
=== | !== |
&& | || |
'/^test$/' | '/test$/' и '/^test/' |
return true; | return false; |
foreach ($someVar as …) | foreach ([] as …); |
... | ... |
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6181303/twitter.png)
Алгоритм Мутационного Тестирования
- Шаг за шагом мутируем исходный код
- Запускаем тесты для каждого Мутанта
- Тесты падают - Мутант считается убитым 👍
- Тесты проходят - Мутант считается выжившим 👎
- Получаем метрику (Mutation Score Indicator)
- Идемпотентность
MSI = (TotalKilledMutants / TotalMutantsCount) * 100;
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6073372/codecov_100.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6073369/msi_70.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6073370/msi_100.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6072554/thumbs-up-sign_1f44d.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6072510/thumbs-down-sign_1f44e.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6181303/twitter.png)
Тест # Мутант # |
1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | Результат |
---|---|---|---|---|---|---|---|---|---|
1 ("+" ➝ "-") | ✅ | ✅ | ✅ | ❌ | Убит | ||||
2 (">" ➝ ">=") | ⏳ | ||||||||
3 ("==" ➝ "!=) | |||||||||
4 (true ➝ false) |
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6181303/twitter.png)
Тест # Мутант # |
1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | Результат |
---|---|---|---|---|---|---|---|---|---|
1 ("+" ➝ "-") | ✅ | ✅ | ✅ | ❌ | Убит | ||||
2 (">" ➝ ">=") | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | Выжил |
3 ("==" ➝ "!=) | ⏳ | ||||||||
4 (true ➝ false) |
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6181303/twitter.png)
Тест # Мутант # |
1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | Результат |
---|---|---|---|---|---|---|---|---|---|
1 ("+" ➝ "-") | ✅ | ✅ | ✅ | ❌ | Убит | ||||
2 (">" ➝ ">=") | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | Выжил |
3 ("==" ➝ "!=) | ✅ | ❌ | Убит | ||||||
4 (true ➝ false) | ✅ | ✅ | ❌ | Убит |
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6181303/twitter.png)
class UserAgeFilter
{
private const AGE_THRESHOLD = 18;
private int $ageThreshold;
public function __construct(int $ageThreshold = self::AGE_THRESHOLD)
{
$this->ageThreshold = $ageThreshold;
}
/** @param array<int, User> $collection */
public function __invoke(array $collection)
{
return array_filter(
$collection,
function (User $user) {
return $user->getAge() >= $this->ageThreshold;
}
);
}
}
Пример
public function test_it_filters_adults(): void
{
$users = [
User::withAge(20),
];
$filter = new UserAgeFilter();
$filteredUsers = $filter($users);
assertCount(1, $filteredUsers);
}
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/4476227/mt-pass-100.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6181303/twitter.png)
Infection
PHP Mutation Testing Library
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/4484041/logo.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6181303/twitter.png)
Результат
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6096038/mt_run_1.png)
class UserAgeFilter
{
// ...
/** @param array<int, User> $collection */
public function __invoke(array $collection)
{
- return array_filter($collection, function (User $user) {
- return $user->getAge() >= $this->ageThreshold;
- });
+ return $collection;
}
}
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/4476227/mt-pass-100.png)
Мутация #1
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6181303/twitter.png)
public function test_it_filters_adults(): void
{
$users = [
User::withAge(20),
];
$filter = new UserAgeFilter();
$filteredUsers = $filter($users);
assertCount(1, $filteredUsers);
}
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6181303/twitter.png)
public function test_it_filters_adults(): void
{
$users = [
+ User::withAge(15),
User::withAge(20),
];
$filter = new UserAgeFilter();
$filteredUsers = $filter($users);
assertCount(1, $filteredUsers);
}
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6181303/twitter.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6096091/mt_run_2.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6181303/twitter.png)
class UserAgeFilter
{
// ...
/** @param array<int, User> $collection */
public function __invoke(array $collection)
{
return array_filter(
$collection,
function (User $user) {
- return $user->getAge() >= $this->ageThreshold;
+ return $user->getAge() > $this->ageThreshold;
}
);
}
}
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/4476227/mt-pass-100.png)
Мутация #2
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6181303/twitter.png)
public function test_it_filters_adults(): void
{
$users = [
User::withAge(15),
User::withAge(20),
];
$filter = new UserAgeFilter();
$filteredUsers = $filter($users);
assertCount(1, $filteredUsers);
}
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6181303/twitter.png)
public function test_it_filters_adults(): void
{
$users = [
User::withAge(15),
+ User::withAge(18),
User::withAge(20),
];
$filter = new UserAgeFilter();
$filteredUsers = $filter($users);
- assertCount(1, $filteredUsers);
+ assertCount(2, $filteredUsers);
}
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6181303/twitter.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6096130/mt_run_3.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6181303/twitter.png)
public function deactivateInactiveUsers(): void
{
$users = $this->userRepository->findInactiveUsers();
foreach ($users as $user) {
$user->deactivate();
- $this->entityManager->persist($user);
}
$this->entitymanager->flush();
}
Мутационное Тестирование находит бесполезный код
public function deletePhoneNumber(User $data, int $id): void
{
// ...
$data->getPhoneNumbers()->removeElement($phoneNumber);
- $this->em->remove($phoneNumber);
$this->em->flush();
}
Мутационное Тестирование находит бесполезный код
Проблемы
-
Скорость (N * t)
- N - количество Мутантов
- t - время выполнения тестов
- Эквивалентные Мутанты
- $a = $b * (-1);
+ $a = $b / (-1);
- $a = $b * $c;
+ $a = $b / $c;
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6181303/twitter.png)
Как ускорить Мутационное Тестирование?
- Запускать тесты только для мутированной строки
- Запускать наиболее быстрые тесты первыми
- Запускать мутации параллельно
- Избегать бесполезные мутации
- Запускать МТ только для измененных файлов
- $result = [$a, $b] + [$c, $d];
+ $result = [$a, $b] - [$c, $d];
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6181303/twitter.png)
infection --git-diff-filter=AM --git-diff-base=master
Запуск для измененных файлов
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6181303/twitter.png)
Запуск "убивающих" тестов первыми
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6181303/twitter.png)
Тест # Мутация # |
1 | 2 | 3 | Результат |
---|---|---|---|---|
1 | ✅ 0.132с | ✅ 0.387с |
❌ 0.399с |
Убит |
2 | ✅ | ✅ | ✅ | Выжил |
3 | ✅ | ❌ | Убит | |
4 | ✅ | ✅ | ❌ | Убит |
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6181303/twitter.png)
Тест # Мутация # |
1 | 2 | 3 | Результат |
---|---|---|---|---|
1 | ❌ 0.399с |
✅ 0.387с |
✅ 0.132с | Убит |
2 | ✅ | ✅ | ✅ | Выжил |
3 | ❌ | ✅ | Убит | |
4 | ❌ | ✅ | ✅ | Убит |
Запуск "убивающих" тестов первыми
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6181303/twitter.png)
Тест # Мутация # |
1 | 2 | 3 | Результат |
---|---|---|---|---|
1 | ❌ 0.399с |
Убит | ||
2 | ✅ | ✅ | ✅ | Выжил |
3 | ❌ | Убит | ||
4 | ❌ | Убит |
Запуск "убивающих" тестов первыми
Как использовать МТ ежедневно?
- Пишете новый функционал, покрываете тестами
- Запускаете Infection
$ infection --git-diff-filter=AM --threads=4 --show-mutations
- Запуск на CI сервере
$ infection --min-covered-msi=100
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6181303/twitter.png)
5) /srv/api/src/User/Status/StatusTransitioner.php:22 [M] PublicVisibility
--- Original
+++ New
@@ @@
- public function isAllowedTransitionTo(User $user, UserStatus $toStatus) : bool
+ protected function isAllowedTransitionTo(User $user, UserStatus $toStatus) : bool
{
if ($user->isDraft()) {
return $toStatus->isActive() || $toStatus->isDeleted();
Как МТ помогает с Code Review?
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6181303/twitter.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/8663449/github-logger.png)
Infection Playground
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6181303/twitter.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/8663564/infection-playground.png)
Infection Playground
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6181303/twitter.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/8668055/infection-playground2.png)
Поддерживаемые фреймворки:
- PHPUnit
- Pest
- Codeception
- phpspec
Поддерживаемые Coverage драйвера:
- Xdebug
- phpdbg
- pcov
Версии PHP: ^7.1 || ^8.0
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6181303/twitter.png)
AST
Abstract Syntax Tree
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6181303/twitter.png)
Абстрактные Синтаксические Деревья
$a = $b + $c;
array(
0: Expr_Assign(
var: Expr_Variable(
name: a
)
expr: Expr_BinaryOp_Plus(
left: Expr_Variable(
name: b
)
right: Expr_Variable(
name: c
)
)
)
)
=
a
+
b
c
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6181303/twitter.png)
array(
0: Expr_Assign(
var: Expr_Variable(
name: a
)
- expr: Expr_BinaryOp_Plus(
+ expr: Expr_BinaryOp_Minus(
left: Expr_Variable(
name: b
)
right: Expr_Variable(
name: c
)
)
)
)
=
a
-
b
c
Code → AST → Мутация AST → Мутант
- $a = $b + $c;
+ $a = $b - $c;
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6181303/twitter.png)
final class Plus extends Mutator
{
/**
* Replaces "+" with "-"
*
* @param Node\Expr\BinaryOp\Plus $node
*/
public function mutate(Node $node): iterable
{
yield new Node\Expr\BinaryOp\Minus($node->left, $node->right, $node->getAttributes());
}
public function canMutate(Node $node): bool
{
if (!$node instanceof Node\Expr\BinaryOp\Plus) {
return false;
}
if ($node->left instanceof Array_ || $node->right instanceof Array_) {
return false;
}
return true;
}
}
Выводы
- Более надежные тесты
- Меньше ошибок в коде
- Помощь в Code Review
- Поиск мертвого и бесполезного кода
- "Condition coverage" вместо "Line Coverage"
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6181303/twitter.png)
- @infection_php - Twitter
- https://github.com/infection/infection
- Документация https://infection.github.io/guide
- Playground - https://infection-php.dev/
![](https://s3.amazonaws.com/media-p.slid.es/uploads/761392/images/6181303/twitter.png)
Ссылки
Вопросы?
Мутационное тестирование в PHP - PHP Russia 2021
By born_free
Мутационное тестирование в PHP - PHP Russia 2021
Мутационное тестирование в PHP - Infection
- 196