Schema-based API-Testing
Automatically generate test-cases based on your API-schemas with schemathesis
Alexander Hultnér, FlaskCon, 2020
@ahultner
Alexander Hultnér
Founder of Hultnér Technologies
@ahultner
Outline
- Short introduction to API-schemas
- Problems
- Solution
- Property based testing, Hypothesis
- Schemathesis overview and features
- Demo
- CLI Interface
- pytest
- Stateful Testing
- The future
- Questions and Answers
@ahultner
API Schemas
API Schemas increasingly more popular. Today's focus on OpenAPI.
- REST/JSON based
- OpenAPI 3
- Swagger
- Predecessor to OpenAPI, nowadays a UI to OpenAPI
- GraphQL (Schemathesis support WIP)
- Typed Query Langague
- Schema and data format
@ahultner
There's plenty of libraries implementing OpenAPI for Flask. Won't cover all in this talk, examples from two approaches.
-
Spec first, generate logic
- Connexion by Zalando
-
Code first, generate spec
- Flask-RESTX (replaces Flask RESTPlus)
- Flasgger,
- APISpec, and many more.
API Schemas
Flask Implementations of Swagger/OpenAPI
@ahultner
- Inaccurate data
- Unexpected user requests
- Mismatch between database layer and application layer
- Library defect
- Human error
- Invalid schemas
- Missed edge cases
- Etc.
The Problem
@ahultner
- Incorrect/non conforming schema, lower severity
- Waste engineering time 😴🥱
- Leads to incorrect assumptions 🧐
- Breaks client code generation 🤪
- Unhandled error, lower severity
- Looks bad 🤨
- Inconvienience 😔😠
- Confusion 😕
- Further escalation? 😈
The Problem
Spectrum of defects, not all errors are equal but they're never good
@ahultner
- Logic errors, medium-high severity
- Data corruption 😢
- Incorrect behaviour 🤔😨
- Crashed application 🤯🤬
- Negative/incorrect billing 🧾⚠️📉
- Security problems, high-critical severity
- Denial of Service (DOS) 🚧⏳💸
- Data leak 📇💳📤😡
- Authentication bypass 👺🔓
- Remote code exectuion (RCE) ⚰️🧨
The Solution
Property based testing with Hypothesis
Property based testing (PBT) is great at finding corner casing.
Let the computer do the heavy lifting in creating exhaustive tests.
@ahultner
Hypothesis
The defacto standard for PBT in Python. A property models the behaviour of a piece of code given a certain type of input.
The Solution
Modelling properties
To test the application we need to model the properties and their input, but can we do better?
Schemas specify the expected behaviour of application for a defined type of input, sounds a bit like properties, doesn't it?
Can we leverage this? Yes we can with Schemathesis!
@ahultner
Schemathesis
Model properties and strategies from schemas
Takes over where it's spiritual predecessor Swagger-Conformance left of, both are based on Hypothesis. Inspired from the QuickREST research paper.
Automatically generates test cases based on what we already know about or application from our specs.
@ahultner
Schemathesis
What do we already know?
Turns out we already know quite a bit about our application
- Application should respond
- Server shouldn't crash
- Stateful links should behave in expected manners (new feature, optional)
- Create resource
- Query created resource
- Update created resource
- Delete created resource
- etc
@ahultner
# Pseudo code
r = response
r.status_code < 500
r.status_code in endpoint.responses
r.headers['Content-Type']
in endpoint.responses[r.status_code]
matches_spec(
r.content,
endpoint
.responses[r.status_code]
[r.headers['C…-Type'] # schema spec
)
Demo
Let's pray to the demo god's!
@ahultner
As you could see lot's for random requests
were generated for us!
Feature overview
What can schemathesis do more?
Schemathesis can do quite a lot and more is coming!
- CLI
-
Built in WSGI (and now ASGI) support
- Can import app directly for faster testing
- Uses flask internally for testing
-
HTTP interface
- Language/framework agnostic universal interface
- pytest-interface
@ahultner
- Stateful testing
-
Fixups
- Bultin fixups for FastAPI non-conformance to current OpenAPI spec.
-
Hooks (Global, Test, Schema)
- Customize the behaviour of schemathesis through hooks.
-
Targeted property based testing
- Search for desired goals, combines property based testing with search parameters, reducing randomness.
-
Record into VCR-cassettes
- Extra fields: http_interactions, id, status (SUCCES, FAILURE, ERROR), seed (for --hypothesis-seed, and elapsed for timing).
- Replay recorded cassettes.
Schemathesis
CLI Interface
Schemathesis implements a rather comprehensive CLI-interface, exposing a lot of the functionality.
Well documented under
$ schemathesis [CMD] --help
$ schemathesis run http://127.0.0.1:5000/swagger.json
================== Schemathesis test session starts ==================
platform …
rootdir: /~/dev/hultner_technologies/Schema-based-API-Testing/code
hypothesis profile 'default' -> database=Directory…
Schema location: http://127.0.0.1:5000/swagger.json
Base URL: http://127.0.0.1:5000/
Specification version: Swagger 2.0
Workers: 1
collected endpoints: 5
GET /todos/ . [ 20%]
POST /todos/ . [ 40%]
DELETE /todos/{id} . [ 60%]
GET /todos/{id} . [ 80%]
PUT /todos/{id} . [100%]
============================== SUMMARY ===============================
Performed checks:
not_a_server_error 500 / 500 passed PASSED
========================= 5 passed in 5.32s ==========================Minimal example
@ahultner
Schemathesis
CLI Interface, WSGI/ASGI interface vs HTTP interface
I use both, it's great to test the entire chain for a true end to end test.
The WSGI/ASGI interface is useful for quicker local branch testing and for larger amounts of examples/data.
$ schemathesis run --app=flask_example:app /swagger.json
================== Schemathesis test session starts ==================
platform Darwin -- Python 3.8.2, schemathesis-2.0.0, hypothesis-5.…
rootdir: /~/dev/hultner_technologies/Schema-based-API-Testing/code
hypothesis profile 'default' -> database=DirectoryBasedExample…s')
Schema location: /swagger.json
Base URL: /
Specification version: Swagger 2.0
Workers: 1
collected endpoints: 5
GET /todos/ . [ 20%]
POST /todos/ . [ 40%]
GET /todos/{id} . [ 60%]
DELETE /todos/{id} . [ 80%]
PUT /todos/{id} . [100%]
============================== SUMMARY ===============================
Performed checks:
not_a_server_error 500 / 500 passed PASSED
========================= 5 passed in 3.73s ==========================WSGI Flask example
@ahultner
HTTP Interface is framework, even language agnostic.
Test any service with a schema specification, be it Python or Cobol.
Schemathesis
pytest interface
Schemathesis can be used to generate strategies for pytest where we can define our own properties.
Previously mentioned built in checks:
- Not a server error (<500)
- Status code conformance
- Content type conformance
- Response schema conformance
@ahultner
Some suggestions to extend with:
- Complex business rules
- Response time/SLA
- Authentication / 401
Schemathesis
pytest interface
From the documentation, tests that any data fitting the schema doesn't cause a server error
@ahultner
# test_api.py
import requests
import schemathesis
schema = schemathesis.from_uri("http://0.0.0.0:8080/swagger.json")
@schema.parametrize()
def test_no_server_errors(case):
# `requests` will make an appropriate call under the hood
response = case.call() # use `call_wsgi` if you used `schemathesis.from_wsgi`
# You could use built-in checks
case.validate_response(response)
# Or assert the response manually
assert response.status_code < 500Schemathesis
Stateful testing
Shown to enhance detection of certain defects in the QuickREST research, recently added to schemathesis.
- Schemathesis reuse data from previous requests and responses.
- Resulting tests tend to reach further into the codebase.
Requires links between objects, feature came with OpenAPI 3.0, but can be used with 2.0/Swagger with the x-links extension.
(will not work with Flask-RESTX out of the box)
@ahultner
schemathesis run --stateful=links http://0.0.0.0/swagger.yaml
...
POST /api/users/ . [ 33%]
-> GET /api/users/{user_id} . [ 50%]
-> PATCH /api/users/{user_id} . [ 60%]
-> PATCH /api/users/{user_id} . [ 66%]
GET /api/users/{user_id} . [ 83%]
-> PATCH /api/users/{user_id} . [ 85%]
PATCH /api/users/{user_id} . [100%]
...
The Future
Schemathesis of tomorrow
We need your help to grow!
- GraphQL
- Schema standard agnostic
- OpenAPI 3.1
- Faster test generation
- Grow the community
- Improve documentation
- …
@ahultner
Conclusion
- Spend less time writing test
- Cover more
- Let your computer do heavy lifting
- Gain deeper confidence in your services
- More things coming
- Actively developed
- Growing community
- Try it out!
@ahultner
Questions
Contact me if you have any further
questions.
Want to learn more?
Available for training, workshops and freelance consulting.
Previous Hypothesis talk on YouTube
Links
@ahultner