Schema-based API-Testing

 

 

Automatically generate test-cases based on your API-schemas with schemathesis

Alexander Hultnér, FlaskCon, 2020

@ahultner

Alexander Hultnér

Founder of Hultnér Technologies

 

 

@ahultner

Outline

  • Short introduction to API-schemas
  • Problems
  • Solution
    • Property based testing, Hypothesis
    • Schemathesis overview and features
    • Demo
    • CLI Interface
    • pytest
    • Stateful Testing
  • The future
  • Questions and Answers

@ahultner

API Schemas

API Schemas increasingly more popular. Today's focus on OpenAPI.

  • REST/JSON based
    • OpenAPI 3
    • Swagger
      • Predecessor to OpenAPI, nowadays a UI to OpenAPI
  • GraphQL (Schemathesis support WIP)
    • Typed Query Langague
    • Schema and data format

@ahultner

There's plenty of libraries implementing OpenAPI for Flask. Won't cover all in this talk, examples from two approaches.

API Schemas

Flask Implementations of Swagger/OpenAPI


 

@ahultner

  • Inaccurate data
  • Unexpected user requests
  • Mismatch between database layer and application layer
  • Library defect
  • Human error
  • Invalid schemas
  • Missed edge cases
  • Etc.

The Problem

@ahultner

  • Incorrect/non conforming schema, lower severity
    • Waste engineering time 😴🥱
    • Leads to incorrect assumptions 🧐
    • Breaks client code generation 🤪
  • Unhandled error, lower severity
    • Looks bad 🤨
    • Inconvienience 😔😠
    • Confusion 😕
    • Further escalation? 😈

The Problem

Spectrum of defects, not all errors are equal but they're never good
 

@ahultner

  • Logic errors, medium-high severity
    • Data corruption 😢
    • Incorrect behaviour 🤔😨
    • Crashed application 🤯🤬
    • Negative/incorrect billing 🧾⚠️📉
  • Security problems, high-critical severity
    • Denial of Service (DOS) 🚧⏳💸
    • Data leak 📇💳📤😡
    • Authentication bypass 👺🔓
    • Remote code exectuion (RCE) ⚰️🧨

The Solution

Property based testing with Hypothesis

Property based testing (PBT) is great at finding corner casing.

Let the computer do the heavy lifting in creating exhaustive tests.

 

@ahultner

Hypothesis

The defacto standard for PBT in Python. A property models the behaviour of a piece of code given a certain type of input.

The Solution

Modelling properties

To test the application we need to model the properties and their input, but can we do better?


Schemas specify the expected behaviour of application for a defined type of input, sounds a bit like properties, doesn't it?

 

Can we leverage this? Yes we can with Schemathesis!

 

@ahultner

Schemathesis

Model properties and strategies from schemas

Takes over where it's spiritual predecessor Swagger-Conformance left of, both are based on Hypothesis. Inspired from the QuickREST research paper.

 

Automatically generates test cases based on what we already know about or application from our specs.

 

@ahultner

Schemathesis

What do we already know?

Turns out we already know quite a bit about our application

  • Application should respond
  • Server shouldn't crash
  • Stateful links should behave in expected manners (new feature, optional)
    • Create resource
    • Query created resource
    • Update created resource
    • Delete created resource
  • etc

 

@ahultner

# Pseudo code
r = response
r.status_code < 500
r.status_code in endpoint.responses

r.headers['Content-Type']
	in endpoint.responses[r.status_code]

matches_spec(
  r.content,
  endpoint
  	.responses[r.status_code]
  	[r.headers['C…-Type'] # schema spec
)

Demo

Let's pray to the demo god's!

@ahultner

As you could see lot's for random requests

were generated for us!

Feature overview

What can schemathesis do more?

Schemathesis can do quite a lot and more is coming!

  • CLI
  • Built in WSGI (and now ASGI) support
    • Can import app directly for faster testing
    • Uses flask internally for testing
  • HTTP interface
    • Language/framework agnostic universal interface
  • pytest-interface

@ahultner

  • Stateful testing
  • Fixups
    • Bultin fixups for FastAPI non-conformance to current OpenAPI spec.
  • Hooks (Global, Test, Schema)
    • Customize the behaviour of schemathesis through hooks.
  • Targeted property based testing
    • Search for desired goals, combines property based testing with search parameters, reducing randomness.
  • Record into VCR-cassettes
    • Extra fields: http_interactions, id, status (SUCCES, FAILURE, ERROR), seed (for --hypothesis-seed, and elapsed for timing).
    • Replay recorded cassettes.

Schemathesis

CLI Interface

Schemathesis implements a rather comprehensive CLI-interface, exposing a lot of the functionality.

 

Well documented under

$ schemathesis [CMD] --help
$ schemathesis run http://127.0.0.1:5000/swagger.json
    
================== Schemathesis test session starts ==================
platform …
rootdir: /~/dev/hultner_technologies/Schema-based-API-Testing/code
hypothesis profile 'default' -> database=Directory…
Schema location: http://127.0.0.1:5000/swagger.json
Base URL: http://127.0.0.1:5000/
Specification version: Swagger 2.0
Workers: 1
collected endpoints: 5

GET /todos/ .                                                  [ 20%]
POST /todos/ .                                                 [ 40%]
DELETE /todos/{id} .                                           [ 60%]
GET /todos/{id} .                                              [ 80%]
PUT /todos/{id} .                                              [100%]

============================== SUMMARY ===============================

Performed checks:
    not_a_server_error                    500 / 500 passed          PASSED

========================= 5 passed in 5.32s ==========================

Minimal example

@ahultner

Schemathesis

CLI Interface, WSGI/ASGI interface vs HTTP interface

I use both, it's great to test the entire chain for a true end to end test.

 

The WSGI/ASGI interface is useful for quicker local branch testing and for larger amounts of examples/data.

$ schemathesis run --app=flask_example:app /swagger.json

================== Schemathesis test session starts ==================
platform Darwin -- Python 3.8.2, schemathesis-2.0.0, hypothesis-5.…
rootdir: /~/dev/hultner_technologies/Schema-based-API-Testing/code
hypothesis profile 'default' -> database=DirectoryBasedExample…s')
Schema location: /swagger.json
Base URL: /
Specification version: Swagger 2.0
Workers: 1
collected endpoints: 5

GET /todos/ .                                                  [ 20%]
POST /todos/ .                                                 [ 40%]
GET /todos/{id} .                                              [ 60%]
DELETE /todos/{id} .                                           [ 80%]
PUT /todos/{id} .                                              [100%]

============================== SUMMARY ===============================

Performed checks:
    not_a_server_error                    500 / 500 passed          PASSED

========================= 5 passed in 3.73s ==========================

WSGI Flask example

@ahultner

HTTP Interface is framework, even language agnostic.
Test any service with a schema specification, be it Python or Cobol.

Schemathesis

pytest interface

Schemathesis can be used to generate strategies for pytest where we can define our own properties.

 

Previously mentioned built in checks:

  • Not a server error (<500)
  • Status code conformance
  • Content type conformance
  • Response schema conformance

@ahultner

Some suggestions to extend with:

  • Complex business rules
  • Response time/SLA
  • Authentication / 401

Schemathesis

pytest interface

From the documentation, tests that any data fitting the schema doesn't cause a server error

@ahultner

# test_api.py
import requests
import schemathesis

schema = schemathesis.from_uri("http://0.0.0.0:8080/swagger.json")

@schema.parametrize()
def test_no_server_errors(case):
    # `requests` will make an appropriate call under the hood
    response = case.call()  # use `call_wsgi` if you used `schemathesis.from_wsgi`
    # You could use built-in checks
    case.validate_response(response)
    # Or assert the response manually
    assert response.status_code < 500

Schemathesis

Stateful testing

Shown to enhance detection of certain defects in the QuickREST research, recently added to schemathesis.

  • Schemathesis reuse data from previous requests and responses.
  • Resulting tests tend to reach further into the codebase.

 

Requires links between objects, feature came with OpenAPI 3.0, but can be used with 2.0/Swagger with the x-links extension.
(will not work with Flask-RESTX out of the box)
 

@ahultner

schemathesis run --stateful=links http://0.0.0.0/swagger.yaml

...

POST /api/users/ .                                     [ 33%]
    -> GET /api/users/{user_id} .                      [ 50%]
        -> PATCH /api/users/{user_id} .                [ 60%]
    -> PATCH /api/users/{user_id} .                    [ 66%]
GET /api/users/{user_id} .                             [ 83%]
    -> PATCH /api/users/{user_id} .                    [ 85%]
PATCH /api/users/{user_id} .                           [100%]

...

The Future

Schemathesis of tomorrow

We need your help to grow!

 

  • GraphQL
  • Schema standard agnostic
  • OpenAPI 3.1
  • Faster test generation
  • Grow the community
  • Improve documentation

@ahultner

Conclusion

  • Spend less time writing test
    • Cover more
  • Let your computer do heavy lifting
  • Gain deeper confidence in your services
  • More things coming
    • ​Actively developed
    • Growing community
  • Try it out!

@ahultner

Questions

Contact me if you have any further

questions.

 

Want to learn more?

Available for training, workshops and freelance consulting.
 

Sign up for Hypothesis course

forms.gle/yRWapypPXdPFSLME7

Previous Hypothesis talk on YouTube

Links

@ahultner

Made with Slides.com