8 types of automated testing for your PHP app

@tomnewbyau

tomnewby.net

About Hyra iQ

  •  A digital contracting platform for retail and commercial leasing
  • Greenfields build ~12 months ago
  • Long-term maintainability
  • Small team with lofty goals
  • Experienced with Symfony, but not so much testing
  • "Does it work?" - "Will it work tomorrow?"

Outline

  • First line of defence: types
  • From PhpUnit to browser-based tests
  • Contract testing
  • Mutation testing
  • Domain-specific tests

Types

  • Type your code as much as possible
  • Requires some changes to the way you write code
  • Use associative arrays sparingly 
  • Use static analysers like vimeo/psalm or PhpStan
  • declare(strict_types=1);
  • Faster than running a whole test suite
  • Easier refactoring/better feedback in IDEs

Unit tests

Comprehensive tests of the logic in a single unit

  • Probably PhpUnit
  • Low memory consumption
  • Pure PHP, no external deps (cache/db)
  • Safe to parallelize: paratestphp/paratest
  • Take this to the next level with Infection: infection/infection
  • Ideally as minimal mocking as possible

Example unit

<?php

declare(strict_types=1);

// ... imports go here

final class WorkspacePermissions
{
    private function __construct()
    {
    }

    public static function canIViewThis(Workspace $workspace, User $user): bool
    {
        foreach ($workspace->getOrganisations() as $organisation) {
            if (OrganisationPermissions::canISeeThis($organisation, $user)) {
                return true;
            }
        }

        return false;
    }
}

Example unit test

<?php

declare(strict_types=1);

// ... imports go here

class WorkspacePermissionsTest extends TestCase
{
    public function testUserCanViewWhenOrgInWorkspace(): void
    {
        $user      = (new UserBuilder())->withOrganisation()->create();
        $workspace = (new WorkspaceBuilder())->withOwner($user->getOrganisation())->create();

        static::assertTrue(WorkspacePermissions::canIViewThis($workspace, $user));
    }

    public function testUserCannotViewWhenOrgNotInWorkspace(): void
    {
        $user      = (new UserBuilder())->withOrganisation()->create();
        $workspace = (new WorkspaceBuilder())->create();

        static::assertFalse(WorkspacePermissions::canIViewThis($workspace, $user));
    }
}

Unit tests

Good for:

  • Testing specific logic: can a user see a workspace?

Not great for:

  • General system correctness:
    • can a user view the workspace page
    • can a user see the workspace in their workspace list

Unit tests

Functional core, imperative shell
Test the core with unit tests, builders over mocks

Lots of @dataProvider based tests

 

Reference stats:

  • 544 tests
  • Local, xdebug off: 2.5s, 52MB

X unit tests, 0 integration tests

Integration tests

Combine multiple units and test them as a group

  • Terminology gets fuzzy and varied here
  • Still in PHP + PhpUnit, but more memory intensive
  • Boot the framework
  • Interacting with services in the DI container
  • Possibly external dependencies, but usually alternatives
    • Database (sqlite or in-memory)
    • Cache (in-memory, disk)
  • Maybe database fixtures
  • Depending on external dependencies, might not parallelise easy 

Integration Test Example

Test subject: CancelInviteService

It should: mark the invite as cancelled, notify the cancelled person and the other people working on the project via emails

<?php

class CancelInviteTest extends WebTestCase
{
    function testItCancelsAndNotifies(): void
    {
        // ... arrange the fixture $invite

        static::bootKernel();
        /** @var CancelInviteService $service */
        $service = static::$container->get(CancelInviteService::class);
        // Act
        $service->cancel($invite);

        // Assert
        static::assertTrue($invite->isCancelled());

        /** @var StubMailer $stubMailer */
        $stubMailer = static::$container->get(StubMailer::class);
        static::assertTrue($stubMailer->hasOneMessageForEachRecipient(...$recipients));
    }
}

Integration tests

Good for:

Units interacting:

  • When I cancel an invite, emails get sent (how? idc)

 

General system correctness

  • Cancelling an invite, notifies the cancelled person and the other people working on the project via email

 

Testing things that interact with DI + Event

Not great for:

Specific logic, edge cases:

  • "Cancelled users with no projects or invites should be marked as inactive unless they're from an organisation with SSO"

Reference stats:

  • 216 tests
  • Local, xdebug off: 55s, 108MB

Functional tests

Testing web requests a real user would make, but not in a browser

  • Aka HTTP tests in Laravel
  • Still PhpUnit, so pure PHP
  • Boot the framework
  • Pass in a web request crafted in PHP
  • No actual browser used
  • Very similar to integration tests (external deps etc)

Functional test snippets

<?php

// Load a page, assert on the status code
$client = static::createClient();
$client->request('GET', '/');
static::assertSame(200, $client->getResponse()->getStatusCode());

// Make a DOM crawler to inspect the page
$crawler = $client->getCrawler();

$form = $crawler->selectButton('Login')->form();

$form['_username'] = $user->getEmail();
$form['_password'] = TestParameters::DEFAULT_PASSWORD;

// Submit forms and assert you get redirected
$client->submit($form);

static::assertTrue($client->getResponse()->isRedirect()));

Functional tests

Good for covering:

  • Everything that integration tests covered
  • Request parsing (json formatting)
  • Request/response headers
  • Firewall and security
  • If a user actually does this thing, will it work

 

Tend to be long, slow, cumbersome to write

Functional tests

We rely on these the most! Most services not that complicated

 

Reference stats:

  • 346 tests
  • Locally, xdebug off: 7 minutes
  • We split by namespace, but worst is 3min, 126MB

Unit, Integration and Functional illustrated

Unit, Integration and Functional illustrated

Browser-based testing

  • Boot a real browser and click around
  • Selenium, Chrome WebDriver, Puppeteer, Cypress.io
  • PHP associated projects: Laravel Dusk, Symfony Panther

 

slow, fragile, but...

 

almost exactly what your user will see

Browser-based testing

  • Real browser connecting to a real webserver
  • Test runner probably not written in PHP, can't access the container or database
  • Based on user acceptance scripts
  • Can everyone run it locally?
  • Where is the test code living?

Browser-based testing

  • We don't have anything running regularly here
  • Percy.io visual regression
  • Well tested API + well tested frontend components = 80% of the way
  • Questions we want answered with tests here
    • Does the frontend boot without errors?
    • Do common, broad user flows still work? Do they look different?

Contract Testing

  • I want to guarantee my API meets a contract, that the frontend was developed against
  • I.E. if I don't serialise the correct properties to send to the frontend, what test will break?
  • json-schema.org
  • DIY: PhpUnit + Symfony WebTestCase + justinrainbow/json-schema
  • Test process:
    • Load db fixtures
    • Hit endpoint
    • Validate JSON against stored schema

JSON Schema Example

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "additionalProperties": false,
  "required": ["deals", "totals"],
  "properties": {
    "deals": {
      "type": "array",
      "items": {
        "$ref": "./deal.json"
      }
    },
    "totals": {
      "required": ["all", "acting"],
      "type": "object",
      "properties": {
        "all": {
          "type": "integer"
        },
        "acting": {
          "type": "integer"
        }
      }
    }
  }
}

{
  "deals": [{
    "id": "some-uuid",
    "name": "Krusty Burger",
    ...
  }],
  "totals": {
    "all": 32,
    "acting": 7
  }
}

Example JSON response

Example JSON schema

Contract Testing

  • Applied here to API responses
  • Great for catching regressions in data serialization
  • Applicable in other circumstances

Mutation Testing

How do we know our tests are good?

Is code coverage enough?

infection/infection to the rescue

Demo Screenshots

Demo Screenshots

Mutation testing with infection

  • Fast, but pretty slow still
  • Too many false-positives in framework code (controllers)
  • Security + functional core namespaces are mutated

Reference Stats:

  • Mutating 544 unit tests/107 files
    • Initial run takes 1m 44s with xdebug
  • Produces 616 mutants
    • Runs in 1min 39s

Domain-specific testing

Testability informs architecture

Message Tests

  • Our app sends ~50 unique emails
  • Content is really important
    • Don't leak data
    • Links need to work
  • MessageInterface abstraction is easier to test
  • Emerging design as we understand the problem better

Message Example

<?php

// This can be generated wherever and passed around
$messageObject = new RequestPasswordResetMessage($user, $token);

// But eventually it needs to be sent
$this->mailer->send($messageObject);

<?php

class RequestPasswordResetMessage extends AbstractPersonMessage
{
    public function __construct(User $user, string $token)
    {
        $subject = \sprintf('%s, forgot your password?', $user->getFirstName());
        parent::__construct($user, $subject);

        $this->addContext([
            'token' => $token,
        ]);
    }

    public function getHTMLTemplate(): string
    {
        return 'User/Message/request-password-reset.html.twig';
    }

    public function getPlainTemplate(): string
    {
        return 'User/Message/request-password-reset.txt.twig';
    }
}

Message Example

<?php

class RequestPasswordResetMessageTest extends MessageTestCase
{
    public function testMessage(): void
    {
        // Create the message
        $message = new RequestPasswordResetMessage($user, $token);

        $test = (new MessageTestBuilder($variation->getMessage()))
            ->toEmail($user->getEmail())
            ->subjectContains(\sprintf('%s, forgot your password?', $user->getFirstName()))
            ->salutation($user)
            ->bodyContains(
                'Forgot your password?',
                'Click the link below to reset your password'
            )
            ->link('Reset your password', 'user.password_reset.claim', ['token' => $token])
        ;

        $this->assertMessageTest($test);
    }
}

Message Tests

  • Extension of WebTestCase to add utility
  • Builder to improve the developer experience
  • Driven out of specificity to our domain
  • How might this apply to your domain?

Reference Stats:

  • 68 tests in 12 seconds

Any question about testing?

me@tomnewby.net

@tomnewbyau on Twitter

8 types of automated testing for your PHP app

By tomnewbyau

Made with Slides.com

8 types of automated testing for your PHP app

  • 825

More from tomnewbyau