Continuous Testing and Beyond

Software Quality in Web Projects

Carsten Windler

Global Head of Software Development, HolidayPirates Group

https://slides.com/carstenwindler/continuous-testing-and-beyond

Continuous Testing and Beyond

Agenda

1. Test your web project
    
2. Further testing techniques
    
3. Software Quality measures
    
4. Continuous Testing

Ok, who
deployed

the
interns
code...

1. Test your web project

Ok, so let's write tests

but

UI

Integration / Component

Unit Tests

The Test Pyramid

E2E Tests

Unit Tests

The Web Test Pyramid

Functional  and Integration Tests

End-to-End (E2E) Tests

• aka Acceptance Tests

• user interaction is simulated using a (headless) browser

Functional Tests

• close to E2E tests, except that no web server and browser is needed

• a request is created and thrown at the code (figuratively)

Integration Tests

• like Unit Tests, but not ran in isolation (e.g. using "real" databases)

• dependencies are not mocked

Setting up the fixtures

• tests must reliable and reproducable

• tests should be fast / as fast as possible

• problematic if your app deals with any external data source (e.g. databases, 3rd party APIs)

• you need to prepare test fixtures

• setup a test environment which is only using these fixtures

• => put your app into test mode

Test Mode

• for web apps, you would specify a certain test URL (testing.myapp.local) on your local or on the test environment under which the app only uses the prepared fixtures

• initial test mode setup can be quite some effort for legacy projects (e.g. creating database seeders and migrations)

• however it allows you to write E2E and functional and tests without modifying the code itself 

Does it blend?

• Unit tests as foundation
• Integration tests for at least the most critical parts of the app
• Functional tests if your app provides an API
• E2E tests if your app has a GUI

2. Further Testing Techniques

Visual Regression Testing

Reference

VRT

• catch style issues early
• could act as additional (kind-of-)integration tests
• setup can be relatively easy...

• ... if you already got a test mode
• gets more complex if there are lots of dynamic parts (Ads, Iframes, Date/Time) 
• requires storage for the reference images (git lfs)

Mutation Testing

Mutation Testing

• modifies your code using so called Mutators
• a Mutator amongst others changes logical operators randomly (e.g. > to >=)
• each mutated version (a Mutant)  is executed against the tests
• if a Mutant leads to failed tests, the mutant has been killed
• if the tests pass, the Mutant has escaped
• Mutant Score Indicator (MSI) tells you the % of mutants detected by the tests

but... why??

Mutant hunt

Example: Infection for PHP

Mutant hunt

• metrics can only give you an indication on the quality of your tests
• surely there is no quick fix for bad tests
• go through the logs and harden each test which lets a Mutant escape

• not suitable for every line of code, but probably your core business logics

Generated Tests

Generated Tests

• tests will automatically generated from project documentation
• requires standardized documentation
• Use Case #1 in web development: RESTful APIs

Generated Tests Example

• api blueprint together with PHP test framework atoum
• uses JSON Schema to validate HTTP requests and responses

https://github.com/Hywan/atoum-apiblueprint-extension

Generated Tests

• no need to write tests manually
• will keep documentation up to date
• almost always requires test mode

https://github.com/apiaryio/dredd

Let's not forget

3. Software Quality measures

But... it works

¯\_(ツ)_/¯

Software Metrics

OMG more numbers

Your project on a T-Shirt

Source Code Analyzer

Software

Metrics

• indicate the quality of your code base
• make developers aware of software quality
• helps identifying problem areas

• will give you direct hints where in your code things go wrong
• Copy & Paste detectors will not allow lazyness
• prevents bad code from getting added to the codebase

Code Style Guides

• Improve readability
• use existing code standards
• sniffers help you check and reformat automatically
• supported by most IDEs
• run e.g. on pre-commit on changed files only

4. Continuous Testing

Web developers daily grind

Developers are lazy.

Don't expect them to run the tests manually every time. 

Continuous Integration (CI)

• build the whole app from scratch
• triggered upon commits to VCS
• run tests against this build
• if all tests pass, the build package is ready to be deployed

Continuous Testing (CT)

• run ALL tests for EVERY build
• on a dedicated server
• without blocking the developers local machine
• catch problems before they go live
• failed builds will not be deployed*

Continuous Delivery (CD)

• deploy often (small increments instead big bang releases)
• deploy with confidence (fewer adrenaline boosts)
• deploy automatically if build was successfull
• easy rollbacks (you never know)

Example Setup

Local Environment

1. Code linter (check syntax)*
2. Code style sniffer*
3. Mess Detector*
4. Unit Tests

CI Server

run all the tests!

1. Code linter
2. Code style sniffer
3. Mess Detector*
4. Unit Tests
5. Integration Tests
6. Functional Tests
7. E2E Tests

CI Server

more options

1. Visual Regression Tests
2. Generated Tests
3. Performance Tests
4. Mutation Tests (via MSI threshold)

Deployment successful

What else is there?

Continuous testing is not only about running automated tests

Thank you for your attention!

Contact

https://www.linkedin.com/in/carsten-windler-399a68147/

https://www.xing.com/profile/Carsten_Windler

c.windler@holidaypirates.com

Carsten Windler

https://www.pexels.com/photo/adult-dark-depressed-face-262218/

https://www.pexels.com/photo/man-person-relaxation-steps-4129/

https://www.pexels.com/photo/adventure-attraction-blur-calm-618545/

https://www.pexels.com/photo/biotechnology-bright-chemical-chemistry-207585/

https://www.pexels.com/photo/woman-reading-a-book-256546/

Image credits