SENG3011

💫 4.3 - Microservice Testing

In this lecture

  • Types of tests; broad categories and specific types
  • Trade-offs in testing
  • Pre-production vs in-production testing
  • Chaos engineering
  • Defining a testing strategy

Test Design: Recap

  • Software, correct on purpose
  • Shifting left - moving risk earlier in the development process
  • Need to test individual pieces of functionality (unit tests) and elements of the system working together (integration)
  • Verification vs. Validation

Agile Testing Quadrants

Acceptance Testing; manual/automated

Unit/Integration Testing

(AKA Functional Testing); automated

Exploratory Testing;

manual

Property Testing; automated

Business facing

Technology facing

Support programming; Deliver with confidence

Critique the Product

Testing Pyramid

Testing Pyramid

Greater scope

  • Slower
  • Worse isolation
  • More confidence
  • Harder to write

Lower scope

  • Faster
  • Better Isolation
  • Less confidence
  • Easier to write

Testing Icecream

Unit Testing

  • What do we mean by a unit?
    • Test a single unit of the system in isolation
    • A function, a class, a "feature", an API endpoint
  • Every bug that is detected should be reproducible in a unit test
  • Most useful in test-driven development

Solitary Unit Testing

  • Control everything except the unit you are testing - controlled variables in a scientific experiment
  • A form of whitebox testing
  • Dependencies are mocked or stubbed:
    • Stub - dependencies are a dummy class returning predefined values 
    • Mock - a stub, where we verify the call was made by the unit
  • Trade-offs
    • Faster
    • More isolatable 
    • Easy to write
    • Often very brittle
    • Less confidence in correctness

Sociable Unit Testing

  • Don't mock downstream services/classes - use the real thing
  • Blackbox tests
  • Trade-offs
    • Slower to run
    • Less isolatable 
    • Harder to write
    • More robust to changes
    • Greater confidence in correctness

Contract Testing

  • When two services couple over an interface, they form a contract
  • One service is the producer/provider/ADT/API and another is the consumer/client
  • The contract has defined preconditions, postconditions and invariants
  • This can include expectations of I/O data structures and side effects
  • Different consumers of the same component form different contracts with the API based on their requirements

Contract Testing

  • Contract tests allow us to verify if an API still fulfils its agreement with the consumer to avoid regression
  • Each consumer will have its own set of contract tests with the producer
  • The sum of all consumer contract tests defines the overall service contract
  • Tools such as Pact allow for easy contract testing

Contract Testing and Cross-Team Collaboration

  • Dependency is always an issue in large organisations
  • Conway's law - our architecture mirrors the structure of our organisation
  • When one team makes a change on their API, other teams need to be notified of any potential regressions/breakages - this is facilitated by contract tests
  • Allows teams to maintain backwards compatibility with their consumers
  • Forms a discussion point between teams

Integration Testing

  • What do we mean by integration?
    • Any two or more "units" interacting with one another
    • Testing flows of interaction or use-cases
    • Can test horizontally (multiple endpoints/classes/features) or vertically (multiple layers of the stack)
  • Create a web of dependencies to catch any bugs lurking in the cracks that the unit tests didn't pick up
  • In microservices, integration testing is often referred to in relation to integration between microservices

Integration Testing in Microservices

  • Cover basic success/error paths
  • Gateway integration tests allow us to check for protocol errors:
    • E.g. missing HTTP headers, incorrect SSL handling
    • Request/response shapes
    • Can test against external components which are mocked
  • Persistence integration tests - check that the schema assumed by the code matches the persistence layer
  • Need to test error paths such as timeouts on db/network connections

Integration Testing: Trade-offs

  • Fast feedback
  • Can fail for more than one reason - harder to isolate
  • Can be technical/time consuming/tedious to write

Component Testing

Component Testing

End to End (E2E) Testing

  • We still need to test everything working as it would in production - from one end of the system to another
  • Form of automated acceptance testing - test our user stories/acceptance criteria/use cases
  • Often done on a UI layer (e.g. Selenium)
  • Trade-offs
    • Harder to write
    • Harder to maintain
    • Ownership can be a problem
    • More moving parts, can be less deterministic
  • Generally, you want a very small number of these tests
  • When do you run E2E tests on a deployment?

End to End (E2E) Testing

All of this testing is preproduction, but can we do better?

In-Production Testing

  • Why test in production?
    • Our best monitoring is in production
    • We can work with real-life data instead of dummy data
    • Exposure to cases we otherwise wouldn't have thought of/hard to replicate scenarios
    • Reduce overall risk

In-Production Testing

  • Liveness Tests - ping a healthcheck endpoint
  • Performance Tests / Benchmarks - test request/response latency
  • Smoke Tests - coarse tests to reveal simple failures
    • E.g. does the program run? Does the UI open?  
    • Fast to run
  • Feature flags - useful for in-production validation

Many, Many Other Types of Tests

  • Volume/stress Tests - put the system under a large load
  • Soak Testing - let the software soak in a staging environment for a period of time
  • Penetration Tests - find flaws in security
  • Compatibility Tests
    • Is your software backwards compatible?
    • Browser compatibility testing
  • Regression Testing - ensure no regression of functionality

Chaos Engineering

  • The practice of proactively finding vulnerabilities in production systems
  • Create more resilient software
  • Netflix: Terminating services during business hours, keep engineers on their toes to fix issues immediately
  • Plan, Experiment, Analyse, Mitigate