Schemathesis

property-based testing

for Open API schemas

Why?

boost engineers' productivity

- Manual testing is expensive

- Maintaining test cases is expensive

- The later a bug is found the more expensive the fix is

In one word - MONEY

What Schemathesis is?

A tool for automated testing of web applications

- Python library to use in your test suites

- Command Line tool

- Compatible with Open API 2 & 3

- Native WSGI support

- Highly customizable

- Built-in common checks for apps behavior

- Verify examples from schemas

- Built on top of Hypothesis

What problems does it try to solve?

- Schema / Application mismatch

- Data corruption

- Denial of service attacks

- Missing logic for non-common scenarios

- Errors in client implementations

# test_api.py
import schemathesis

schema = schemathesis.from_uri(
    "http://0.0.0.0:8080/swagger.json"
)

@schema.parametrize()
def test_no_server_errors(case):
    response = case.call()
    # You could use built-in checks
    case.validate_response(response)
    # Or assert the response manually
    assert response.status_code < 500

Python tests

Live demo

https://petstore3.swagger.io/api/v3/openapi.json

Guess how many failures will be there

Control Hypothesis settings

Command Line

`--hypothesis-max-examples=1000`

Code

from hypothesis import settings

...
@schema.parametrize(
    endpoint="/api/users", 
    method="POST"
)
@settings(max_examples=1000)
def test_create_user(case):
   ...

Run tests concurrently

$ schemathesis run \
> -w 8 
> --app=module:app
> -H "Authorization: Bearer SECRET"
> /swagger.json

WSGI apps are tested natively without starting a server

Try different checks

Not a server error
Status code conformance
Content-type conformance
Response schema conformance

Or write your own

# checks.py
import schemathesis

@schemathesis.register_check
def not_too_long(response, case):
    assert response.elapsed < timedelta(
        milliseconds=300
    )

$ schemathesis --pre-run checks run -c not_too_long

Limitations

Slow for certain schemas

Underlying `hypothesis-jsonschema` is not feature-complete yet and therefore not optimized

May not reach deep

It is a randomized approach but will be improved with coverage-guided input generation.

Also, some input data (ids of DB entries) might be crafted manually to work this out.

Generates only valid data

Generation of invalid data will be added soon (WIP PR is open)

Try it out!

https://github.com/kiwicom/schemathesis

Looking forward to your feedback

Questions?