SENG3011

Testing Pyramid

Testing Pyramid

Greater scope

• Slower
• Worse isolation
• More confidence
• Harder to write

Lower scope

• Faster
• Better Isolation
• Less confidence
• Easier to write

Testing Icecream

Unit Testing

• What do we mean by a unit?
  • Test a single unit of the system in isolation
  • A function, a class, a "feature", an API endpoint
• Every bug that is detected should be reproducible in a unit test
• Most useful in test-driven development

Solitary Unit Testing

• Control everything except the unit you are testing - controlled variables in a scientific experiment
• A form of whitebox testing
• Dependencies are mocked or stubbed:
  • Stub - dependencies are a dummy class returning predefined values 
  • Mock - a stub, where we verify the call was made by the unit
• Trade-offs
  • Faster
  • More isolatable 
  • Easy to write
  • Often very brittle
  • Less confidence in correctness

Sociable Unit Testing

• Don't mock downstream services/classes - use the real thing
• Blackbox tests
• Trade-offs
  • Slower to run
  • Less isolatable 
  • Harder to write
  • More robust to changes
  • Greater confidence in correctness

Contract Testing

• When two services couple over an interface, they form a contract
• One service is the producer/provider/ADT/API and another is the consumer/client
• The contract has defined preconditions, postconditions and invariants
• This can include expectations of I/O data structures and side effects
• Different consumers of the same component form different contracts with the API based on their requirements

Contract Testing

• Contract tests allow us to verify if an API still fulfils its agreement with the consumer to avoid regression
• Each consumer will have its own set of contract tests with the producer
• The sum of all consumer contract tests defines the overall service contract
• Tools such as Pact allow for easy contract testing

Contract Testing and Cross-Team Collaboration

• Dependency is always an issue in large organisations
• Conway's law - our architecture mirrors the structure of our organisation
• When one team makes a change on their API, other teams need to be notified of any potential regressions/breakages - this is facilitated by contract tests
• Allows teams to maintain backwards compatibility with their consumers
• Forms a discussion point between teams

Integration Testing

• What do we mean by integration?
  • Any two or more "units" interacting with one another
  • Testing flows of interaction or use-cases
  • Can test horizontally (multiple endpoints/classes/features) or vertically (multiple layers of the stack)
• Create a web of dependencies to catch any bugs lurking in the cracks that the unit tests didn't pick up
• In microservices, integration testing is often referred to in relation to integration between microservices

Integration Testing in Microservices

• Cover basic success/error paths
• Gateway integration tests allow us to check for protocol errors:
  • E.g. missing HTTP headers, incorrect SSL handling
  • Request/response shapes
  • Can test against external components which are mocked
• Persistence integration tests - check that the schema assumed by the code matches the persistence layer
• Need to test error paths such as timeouts on db/network connections

Integration Testing: Trade-offs

• Fast feedback
• Can fail for more than one reason - harder to isolate
• Can be technical/time consuming/tedious to write

Component Testing

Component Testing

End to End (E2E) Testing

• We still need to test everything working as it would in production - from one end of the system to another
• Form of automated acceptance testing - test our user stories/acceptance criteria/use cases
• Often done on a UI layer (e.g. Selenium)
• Trade-offs
  • Harder to write
  • Harder to maintain
  • Ownership can be a problem
  • More moving parts, can be less deterministic
• Generally, you want a very small number of these tests
• When do you run E2E tests on a deployment?

End to End (E2E) Testing

All of this testing is preproduction, but can we do better?

In-Production Testing

• Why test in production?
  • Our best monitoring is in production
  • We can work with real-life data instead of dummy data
  • Exposure to cases we otherwise wouldn't have thought of/hard to replicate scenarios
  • Reduce overall risk

In-Production Testing

• Liveness Tests - ping a healthcheck endpoint
• Performance Tests / Benchmarks - test request/response latency
• Smoke Tests - coarse tests to reveal simple failures
  • E.g. does the program run? Does the UI open?  
  • Fast to run
• Feature flags - useful for in-production validation