Continuous Testing and Beyond

Software Quality in Web Projects

Carsten Windler

Global Head of Software Development, HolidayPirates Group

Continuous Testing and Beyond


  1. Test your web project
  2. Further testing techniques
  3. Software Quality measures
  4. Continuous Testing

Ok, who


1. Test your web project

Ok, so let's write tests



Integration / Component

Unit Tests

The Test Pyramid

E2E Tests

Unit Tests

The Web Test Pyramid

Functional  and Integration Tests

End-to-End (E2E) Tests

  • aka Acceptance Tests

  • user interaction is simulated using a (headless) browser

  + tests the whole application

  + fully supports JavaScript

  + no refactoring or code knowledge  required

  -  slow

  -  fragile

  -  require test mode 

Functional Tests

  • close to E2E tests, except that no web server and browser is needed

  • a request is created and thrown at the code (figuratively)

  + great for testing REST APIs

  + no JavaScript

  + fast (compared to E2E tests)

  + quite robust

  -  require test mode 

Integration Tests

  • like Unit Tests, but not ran in isolation (e.g. using "real" databases)

  • dependencies are not mocked

  + slower as Unit Tests, but fast enough

  + reveal problems early

  -  require test mode 

Setting up the fixtures

  • tests must reliable and reproducable

  • tests should be fast / as fast as possible

  • problematic if your app deals with any external data source (e.g. databases, 3rd party APIs)

  • you need to prepare test fixtures

  • setup a test environment which is only using these fixtures

  • => put your app into test mode

Test Mode

  • for web apps, you would specify a certain test URL (testing.myapp.local) on your local or on the test environment under which the app only uses the prepared fixtures

  • initial test mode setup can be quite some effort for legacy projects (e.g. creating database seeders and migrations)

  • however it allows you to write E2E and functional and tests without modifying the code itself 

Does it blend?

  • Unit tests as foundation
  • Integration tests for at least the most critical parts of the app
  • Functional tests if your app provides an API
  • E2E tests if your app has a GUI

2. Further Testing Techniques


Visual Regression Testing



Test run



  • catch style issues early
  • could act as additional (kind-of-)integration tests
  • setup can be relatively easy...


  • ... if you already got a test mode
  • gets more complex if there are lots of dynamic parts (Ads, Iframes, Date/Time) 
  • requires storage for the reference images (git lfs)

Mutation Testing

Mutation Testing

  • modifies your code using so called Mutators
  • a Mutator amongst others changes logical operators randomly (e.g. > to >=)
  • each mutated version (a Mutant)  is executed against the tests
  • if a Mutant leads to failed tests, the mutant has been killed
  • if the tests pass, the Mutant has escaped
  • Mutant Score Indicator (MSI) tells you the % of mutants detected by the tests

but... why??


Tests needed for 100%

Code Coverage


Tests needed for bug-free code


Mutant hunt

Example: Infection for PHP

You might ask yourself now:

"An MSI of 78% sounds ok-ish, but what exactly can I do with this information now?"

Mutant hunt

  • metrics can only give you an indication on the quality of your tests
  • surely there is no quick fix for bad tests
  • go through the logs and harden each test which lets a Mutant escape
  • not suitable for every line of code, but probably your core business logics

Generated Tests

Generated Tests

  • tests will automatically generated from project documentation
  • requires standardized documentation
  • Use Case #1 in web development: RESTful APIs

Generated Tests Example

  • api blueprint together with PHP test framework atoum
  • uses JSON Schema to validate HTTP requests and responses

Generated Tests

  • no need to write tests manually
  • will keep documentation up to date
  • almost always requires test mode

Let's not forget

3. Software Quality measures

But... it works


Arguing about code quality is easier when you have objective standards

Why is my code

so bad?

Software Metrics

OMG more numbers

Your project on a T-Shirt


Source Code Analyzer



  • indicate the quality of your code base
  • make developers aware of software quality
  • helps identifying problem areas

Source Code Analyzers

  • will give you direct hints where in your code things go wrong
  • Copy & Paste detectors will not allow lazyness
  • prevents bad code from getting added to the codebase

Code Style Guides

  • Improve readability
  • use existing code standards
  • sniffers help you check and reformat automatically
  • supported by most IDEs
  • run e.g. on pre-commit on changed files only

4. Continuous Testing

Web developers daily grind

Developers are lazy.

Don't expect them to run the tests manually every time. 

They just won't.

Continuous Integration (CI)

  • build the whole app from scratch
  • triggered upon commits to VCS
  • run tests against this build
  • if all tests pass, the build package is ready to be deployed

Continuous Testing (CT)

  • run ALL tests for EVERY build
  • on a dedicated server
  • without blocking the developers local machine
  • catch problems before they go live
  • failed builds will not be deployed*

(* ideally, but for hotfixes you should be able to surpass this)

Continuous Delivery (CD)

  • deploy often (small increments instead big bang releases)
  • deploy with confidence (fewer adrenaline boosts)
  • deploy automatically if build was successfull
  • easy rollbacks (you never know)

Example Setup

Local Environment

fast running tasks on pre-commit hook

  1. Code linter (check syntax)*
  2. Code style sniffer*
  3. Mess Detector*
  4. Unit Tests

(* only on changed files)


=> save CI resources (and save developer time) by running these steps locally before code can be committed

CI Server

run all the tests!


  1. Code linter
  2. Code style sniffer
  3. Mess Detector*
  4. Unit Tests
  5. Integration Tests
  6. Functional Tests
  7. E2E Tests

* will not break the build, but issue warnings

CI Server

more options

  1. Visual Regression Tests
  2. Generated Tests
  3. Performance Tests
  4. Mutation Tests (via MSI threshold)

Deployment successful

What else is there?

Continuous testing is not only about running automated tests

Thank you for your attention!


Carsten Windler

Image credits

Continuous Testing and Beyond

By Carsten Windler

Continuous Testing and Beyond

  • 2,082