8 types of automated testing for your PHP app

@tomnewbyau

tomnewby.net

About Hyra iQ

A digital contracting platform for retail and commercial leasing
Greenfields build ~12 months ago
Long-term maintainability
Small team with lofty goals
Experienced with Symfony, but not so much testing
"Does it work?" - "Will it work tomorrow?"

Outline

First line of defence: types
From PhpUnit to browser-based tests
Contract testing
Mutation testing
Domain-specific tests

Types

Type your code as much as possible
Requires some changes to the way you write code
Use associative arrays sparingly
Use static analysers like vimeo/psalm or PhpStan
declare(strict_types=1);
Faster than running a whole test suite
Easier refactoring/better feedback in IDEs

Unit tests

Comprehensive tests of the logic in a single unit

Probably PhpUnit
Low memory consumption
Pure PHP, no external deps (cache/db)
Safe to parallelize: paratestphp/paratest
Take this to the next level with Infection: infection/infection
Ideally as minimal mocking as possible

Example unit

<?php

declare(strict_types=1);

// ... imports go here

final class WorkspacePermissions
{
    private function __construct()
    {
    }

    public static function canIViewThis(Workspace $workspace, User $user): bool
    {
        foreach ($workspace->getOrganisations() as $organisation) {
            if (OrganisationPermissions::canISeeThis($organisation, $user)) {
                return true;
            }
        }

        return false;
    }
}

Example unit test

<?php

declare(strict_types=1);

// ... imports go here

class WorkspacePermissionsTest extends TestCase
{
    public function testUserCanViewWhenOrgInWorkspace(): void
    {
        $user      = (new UserBuilder())->withOrganisation()->create();
        $workspace = (new WorkspaceBuilder())->withOwner($user->getOrganisation())->create();

        static::assertTrue(WorkspacePermissions::canIViewThis($workspace, $user));
    }

    public function testUserCannotViewWhenOrgNotInWorkspace(): void
    {
        $user      = (new UserBuilder())->withOrganisation()->create();
        $workspace = (new WorkspaceBuilder())->create();

        static::assertFalse(WorkspacePermissions::canIViewThis($workspace, $user));
    }
}

Unit tests

Good for:

Testing specific logic: can a user see a workspace?

Not great for:

General system correctness:
- can a user view the workspace page
- can a user see the workspace in their workspace list

Unit tests

Functional core, imperative shell
Test the core with unit tests, builders over mocks

Lots of @dataProvider based tests

Reference stats:

544 tests
Local, xdebug off: 2.5s, 52MB

X unit tests, 0 integration tests

Integration tests

Combine multiple units and test them as a group

Terminology gets fuzzy and varied here
Still in PHP + PhpUnit, but more memory intensive
Boot the framework
Interacting with services in the DI container
Possibly external dependencies, but usually alternatives
- Database (sqlite or in-memory)
- Cache (in-memory, disk)
Maybe database fixtures
Depending on external dependencies, might not parallelise easy

Integration Test Example

Test subject: CancelInviteService

It should: mark the invite as cancelled, notify the cancelled person and the other people working on the project via emails

<?php

class CancelInviteTest extends WebTestCase
{
    function testItCancelsAndNotifies(): void
    {
        // ... arrange the fixture $invite

        static::bootKernel();
        /** @var CancelInviteService $service */
        $service = static::$container->get(CancelInviteService::class);
        // Act
        $service->cancel($invite);

        // Assert
        static::assertTrue($invite->isCancelled());

        /** @var StubMailer $stubMailer */
        $stubMailer = static::$container->get(StubMailer::class);
        static::assertTrue($stubMailer->hasOneMessageForEachRecipient(...$recipients));
    }
}

Integration tests

Good for:

Units interacting:

When I cancel an invite, emails get sent (how? idc)

General system correctness

Cancelling an invite, notifies the cancelled person and the other people working on the project via email

Testing things that interact with DI + Event

Not great for:

Specific logic, edge cases:

"Cancelled users with no projects or invites should be marked as inactive unless they're from an organisation with SSO"

Reference stats:

216 tests
Local, xdebug off: 55s, 108MB

Functional tests

Testing web requests a real user would make, but not in a browser

Aka HTTP tests in Laravel
Still PhpUnit, so pure PHP
Boot the framework
Pass in a web request crafted in PHP
No actual browser used
Very similar to integration tests (external deps etc)

Functional test snippets

<?php

// Load a page, assert on the status code
$client = static::createClient();
$client->request('GET', '/');
static::assertSame(200, $client->getResponse()->getStatusCode());

// Make a DOM crawler to inspect the page
$crawler = $client->getCrawler();

$form = $crawler->selectButton('Login')->form();

$form['_username'] = $user->getEmail();
$form['_password'] = TestParameters::DEFAULT_PASSWORD;

// Submit forms and assert you get redirected
$client->submit($form);

static::assertTrue($client->getResponse()->isRedirect()));

Functional tests

Good for covering:

Everything that integration tests covered
Request parsing (json formatting)
Request/response headers
Firewall and security
If a user actually does this thing, will it work

Tend to be long, slow, cumbersome to write

Functional tests

We rely on these the most! Most services not that complicated

Reference stats:

346 tests
Locally, xdebug off: 7 minutes
We split by namespace, but worst is 3min, 126MB

Unit, Integration and Functional illustrated

Browser-based testing

Boot a real browser and click around
Selenium, Chrome WebDriver, Puppeteer, Cypress.io
PHP associated projects: Laravel Dusk, Symfony Panther

slow, fragile, but...

almost exactly what your user will see

Browser-based testing

Real browser connecting to a real webserver
Test runner probably not written in PHP, can't access the container or database
Based on user acceptance scripts
Can everyone run it locally?
Where is the test code living?

Browser-based testing

We don't have anything running regularly here
Percy.io visual regression
Well tested API + well tested frontend components = 80% of the way
Questions we want answered with tests here
- Does the frontend boot without errors?
- Do common, broad user flows still work? Do they look different?

Contract Testing

I want to guarantee my API meets a contract, that the frontend was developed against
I.E. if I don't serialise the correct properties to send to the frontend, what test will break?
json-schema.org
DIY: PhpUnit + Symfony WebTestCase + justinrainbow/json-schema
Test process:
- Load db fixtures
- Hit endpoint
- Validate JSON against stored schema

JSON Schema Example

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "additionalProperties": false,
  "required": ["deals", "totals"],
  "properties": {
    "deals": {
      "type": "array",
      "items": {
        "$ref": "./deal.json"
      }
    },
    "totals": {
      "required": ["all", "acting"],
      "type": "object",
      "properties": {
        "all": {
          "type": "integer"
        },
        "acting": {
          "type": "integer"
        }
      }
    }
  }
}

{
  "deals": [{
    "id": "some-uuid",
    "name": "Krusty Burger",
    ...
  }],
  "totals": {
    "all": 32,
    "acting": 7
  }
}

Example JSON response

Example JSON schema

Contract Testing

Applied here to API responses
Great for catching regressions in data serialization
Applicable in other circumstances

Mutation Testing

How do we know our tests are good?

Is code coverage enough?

infection/infection to the rescue

Demo Screenshots

Mutation testing with infection

Fast, but pretty slow still
Too many false-positives in framework code (controllers)
Security + functional core namespaces are mutated

Reference Stats:

Mutating 544 unit tests/107 files
- Initial run takes 1m 44s with xdebug
Produces 616 mutants
- Runs in 1min 39s

Domain-specific testing

Testability informs architecture

Message Tests

Our app sends ~50 unique emails
Content is really important
- Don't leak data
- Links need to work
MessageInterface abstraction is easier to test
Emerging design as we understand the problem better

Message Example

<?php

// This can be generated wherever and passed around
$messageObject = new RequestPasswordResetMessage($user, $token);

// But eventually it needs to be sent
$this->mailer->send($messageObject);

<?php

class RequestPasswordResetMessage extends AbstractPersonMessage
{
    public function __construct(User $user, string $token)
    {
        $subject = \sprintf('%s, forgot your password?', $user->getFirstName());
        parent::__construct($user, $subject);

        $this->addContext([
            'token' => $token,
        ]);
    }

    public function getHTMLTemplate(): string
    {
        return 'User/Message/request-password-reset.html.twig';
    }

    public function getPlainTemplate(): string
    {
        return 'User/Message/request-password-reset.txt.twig';
    }
}

Message Example

<?php

class RequestPasswordResetMessageTest extends MessageTestCase
{
    public function testMessage(): void
    {
        // Create the message
        $message = new RequestPasswordResetMessage($user, $token);

        $test = (new MessageTestBuilder($variation->getMessage()))
            ->toEmail($user->getEmail())
            ->subjectContains(\sprintf('%s, forgot your password?', $user->getFirstName()))
            ->salutation($user)
            ->bodyContains(
                'Forgot your password?',
                'Click the link below to reset your password'
            )
            ->link('Reset your password', 'user.password_reset.claim', ['token' => $token])
        ;

        $this->assertMessageTest($test);
    }
}

Message Tests

Extension of WebTestCase to add utility
Builder to improve the developer experience
Driven out of specificity to our domain
How might this apply to your domain?

Reference Stats:

68 tests in 12 seconds

Any question about testing?

me@tomnewby.net

@tomnewbyau on Twitter