Test Design & Testing Theory

Recap: Verification & Validation

  • Verification - The system has been built right
  • Validation - The right system has been built
  • The nature of testing: your code tests your tests, and your tests test your code
  • Writing code is the easy part

Why test-first?

  • Start with the end
  • Test to an interface, not to an implementation (Black Box Testing)
  • The failing unit test:
    • Karl Popper & philosophy of falsification
    • Testing as a science
    • Test-last justifies code written, whereas we want to falsify 

Designing Tests

  • Devise a test list
    • ​What is being tested
    • Given inputs
    • Expected outputs
  • Adopt a systematic approach to writing test lists
  • Exhaust all possibilities at an abstract level
  • Test all flows in the program - optimal, optional and exceptional

Designing Tests

  • Normal design principles apply - DRY, KISS
    • Abstract away common functionality
    • Fixtures - @BeforeEach, @AfterEach, @BeforeAll, @AfterAll
    • Helper Functions
    • Helper Classes
  • Avoid complexity in tests
    • Be careful of loops, if statements

Designing Tests

  • Using class-scoped variables as part of tests
    • This is OK, but be careful
    • Tests must be atomic - run in any order

What are we testing anyway?

  • Behaviour of the system/class/function matches a specification
  • Articulate the contents of a test in the test name
  • How is behaviour produced?
    • Return values
    • Side effects

Designing Tests

  • How can we meaningfully test objects returned?
    • Getters and setters
    • Intermediary communication objects (e.g. EntityInfoResponse)
    • JSON

Types of Testing, First Glance

  • Unit Testing - test the individual parts of the system in isolation
  • Integration Testing - test interconnecting parts of the system working together
  • System Testing - test the system working as a whole

Types of Testing

  • Unit tests - white-box
    • Impossible to write with Abstract Data Types
    • ​E.g. message/send, channel/messages
  • Integration tests - black-box
    • ​Test dependencies - functions rely on other functions to work

Unit Testing Abstract Data Types

  • Unit testing an ADT is impossible
  • Make tests as 'unit-like' as possible
  • Treat like a scientific experiment:
    • Pick one thing to test (independent variable)
    • Only make assertions around the behaviour of that function
    • Reduce dependencies on other functions as much as possible (keep other variables controlled)
    • Avoid sanity-checking other unrelated values
  • Anything that is asserted in the test should be encapsulated in the test name
  • Do this for each unit of functionality
  • A failing test pinpoints the exact broken functionality

Integration Testing Abstract Data Types

  • Weave a web of dependencies
  • Test and sanity-check as much as possible
  • Catch any bugs that may have been missed by unit tests when things come together
  • Any bug caught by an integration test should be replicable in a corresponding unit tests

System Testing

  • What do we mean by "testing the system as a whole?"
    • Treat the system as one large black box
    • Interact with the entire system via the interface
    • E.g. BlackoutController methods, Project API
  • System tests may be similarly structured to integration tests

Acceptance Testing & Debugging

  • Checking that the system fulfils the requirements
  • Occurs at the most abstract level
    • Test code works on the frontend
  • Bug finding
    • Diagnose and reproduce the error on the frontend
    • Write a failing unit test + add to test bank
    • Write code to pass the test
    • Check the bug is resolved

Assumptions

  • Clear up an ambiguity in the specification / contract
  • A good assumption articulates a behaviour rather than an implementation
  • Assumptions are black-box
  • An assumption is accompanied by a corresponding unit test

We can design tests, but how do we test design?

  • Verification vs. Validation
  • Verification tests design
  • Validation tests requirements
  • tbc

Randomised Testing

  • TBA