Pydantic

Give your data classes super powers with pydantic

Alexander Hultnér

Python Pizza, Remote 2020

@ahultner

hultner.se/

Alexander Hultnér

Founder of Hultnér Technologies

linkedin.com/in/hultner

@ahultner

hultner.se/

Outline

Quick refresher on python data classes
Pydantic introduction
- Prior art
- Minimal example from dataclass
- Runtime type-checking
- JSON (de)serialization
- JSONSchema
- Validators
  - Custom model validators
  - Validation decorator for functions, via @validate_arguments.
- FastAPI framework
  - OpenAPI Specifications
  - Autogenerated tests
Cool features worth mentioning
The future

@ahultner

hultner.se/

Dataclasses

Let's start with a quick @dataclass-refresher.

Well use pizza-based examples in the spirit of python.pizza 🐍🍕

from dataclasses import dataclass
from typing import Tuple

@dataclass
class Pizza:
    style: str
    toppings: Tuple[str, ...]

>>> Pizza(1, ("cheese", "ham"))
Pizza(style=1, toppings=('cheese', 'ham'))

And now let's try it out

@ahultner

hultner.se/

Dataclasses

Now we may want to constrain the toppings to ones we actually offer.

Our pizzeria doesn't offer pineapple 🚫🍍 as a valid topping, hate it 😡 or love it 💕

class Topping(str, Enum):
    mozzarella = 'mozzarella'
    tomato_sauce = 'tomato sauce'
    prosciutto = 'prosciutto'
    basil = 'basil'
    rucola = 'rucola'
    

@dataclass
class Pizza:
    style: str
    toppings: Tuple[Topping, ...]

>>> Pizza(2, ("pineapple", 24))
Pizza(style=2, toppings=('pineapple', 24))

Let's see what happens if we try to create a pizza with pineapple 🍍 topping.

@ahultner

hultner.se/

Python Library 🐍
Great documentation 📖
Data validation using python type annotations 🧹🧐
Runtime type enforcement 💯
User-friendly errors 👨‍💻
No convoluted syntax, pure pythonic classes
(De)serialisation
Predecessors 🏛
- Dataclasses, attrs, marshmallow, valideer, ORM-libraries, etc.

Pydantic

Quick introduction

@ahultner

hultner.se/

With dataclasses the types aren't enforced.

But in this case we'll lean on the shoulders of a giant, pydantic 🧹🐍🧐

from pydantic.dataclasses import dataclass

@dataclass
class Pizza:
    style: str
    toppings: Tuple[Topping, ...]

from pydantic import ValidationError
try:
    Pizza(2, ("pineapple", 24))
except ValidationError as err:
    print(err)

And with that simple chage we can see that our new instance of an invalid pizza actually raises errors 🚫🚨

Additionally these errors are very readable!

2 validation errors for Pizza

toppings -> 0
  value is not a valid enumeration member; permitted: 'mozzarella', 'tomato sauce', 'prosciutto', 'basil', 'rucola' (type=type_error.enum; enum_values=[ … ])

toppings -> 1
  value is not a valid enumeration member; permitted: 'mozzarella', 'tomato sauce', 'prosciutto', 'basil', 'rucola' (type=type_error.enum; enum_values=[ … ])

Pydantic

Runtime type-checking, data class drop-in replacement

@ahultner

hultner.se/

So let's try to create a valid pizza 🍕✅

Pizza(
  "Napoli", 
  (
    Topping.tomato_sauce, 
    Topping.prosciutto, 
    Topping.mozzarella, 
    Topping.basil,
  )
)

Pizza(
  style='Napoli', 
  toppings=(
    <Topping.tomato_sauce: 'tomato sauce'>,
    <Topping.prosciutto: 'prosciutto'>,
    <Topping.mozzarella: 'mozzarella'>,
    <Topping.basil: 'basil'>
  )
)

Pydantic

Runtime type-checking

@ahultner

hultner.se/

Pydantic

BaseModel, JSON

So what about JSON? 🧑‍💻

from pydantic import BaseModel


class Pizza(BaseModel):
    style: str
    toppings: Tuple[Topping, ...]

Dataclass dropin replacement is great for compability

Pydantic BaseModel does more!
(de)serialisation
First class JSON-support

Disclaimer: Pydantic is primarly a parsing library

@ahultner

hultner.se/

Pydantic

JSON (de)serialisation

So what about JSON? 🧑‍💻

Specify arguments using kwargs when using BaseModel

Pizza(
  style="Napoli", 
  toppings=(
    Topping.tomato_sauce, 
   	Topping.prosciutto, 
   	Topping.mozzarella, 
   	Topping.basil,
  )
)

Pizza(
  style='Napoli', 
  toppings=(
    <Topping.tomato_sauce: 'tomato sauce'>,
    <Topping.prosciutto: 'prosciutto'>,
    <Topping.mozzarella: 'mozzarella'>,
    <Topping.basil: 'basil'>
  )
)

>>> _.json()
'{"style": "Napoli", "toppings": ["tomato sauce", "prosciutto", "mozzarella", "basil"]}'

We can now easily encode this object as JSON 👾

There's also built-in support for dict, pickle, immutable copy(). Pydantic will also (de)serialise subclasses. 🥒

@ahultner

hultner.se/

Pydantic

JSON (de)serialisation

So what about JSON? 🧑‍💻

Let's reconstruct our object from the JSON output 🏗

>>> Pizza.parse_raw('{"style": "Napoli", "toppings": ["tomato sauce", "prosciutto", "mozzarella", "basil"]}')

Pizza(
  style='Napoli', 
  toppings=(
    <Topping.tomato_sauce: 'tomato sauce'>,
    <Topping.prosciutto: 'prosciutto'>,
    <Topping.mozzarella: 'mozzarella'>,
    <Topping.basil: 'basil'>
  )
)

We'll use the parse_raw(…) function

@ahultner

hultner.se/

Pydantic

JSON (de)serialisation

So what about JSON? 🧑‍💻

Errors raises a validation error, these can also be represented as JSON 🚨🚫🚧

try:
    Pizza(style="Napoli", toppings=(2,))
except ValidationError as err:
    print(err.json())

[
  {
    "loc": [
      "toppings",
      0
    ],
    "msg": "value is not a valid enumeration member; permitted: 'mozzarella', 'tomato sauce', 'prosciutto', 'basil', 'rucola'",
    "type": "type_error.enum",
    "ctx": {
      "enum_values": [
        "mozzarella",
        "tomato sauce",
        "prosciutto",
        "basil",
        "rucola"
      ]
    }
  }
]

@ahultner

hultner.se/

Pydantic

JSONSchema

JSONSchema can be exported directly from the model

Useful for external clients or to feed a Swagger/OpenAPI-spec 📜✅

>>> Pizza.schema()
{'title': 'Pizza',
 'type': 'object',
 'properties': {'style': {'title': 'Style', 'type': 'string'},
  'toppings': {'title': 'Toppings',
   'type': 'array',
   'items': {'enum': ['mozzarella',
     'tomato sauce',
     'prosciutto',
     'basil',
     'rucola'],
    'type': 'string'}}},
 'required': ['style', 'toppings']}

⚠ Caution: Pydantic uses the latest draft 7 of JSONSchema, this will be used in the comming OpenAPI 3.1 spec but the current 3.0.x spec uses draft 4.

@ahultner

hultner.se/

Pydantic

Validators

That was the built-in validators.

But what about custom ones?

from pydantic import validator, root_validator

class BakedPizza(Pizza):
  oven_temperature: int

  # A validator looking at a single property
  @validator('style')
  def check_style(cls, style):
    house_styles = ("Napoli", "Roman", "Italian")
    if style not in house_styles:
      raise ValueError(f"""
        We only cook the following styles: {house_styles}
        Given: {style}""")
    return style
  
  # Root validators check the entire model
  @root_validator
  def check_temp(cls, values):
    style = values.get("style") 
    temp = values.get("oven_temperature")
        
    if style != "Napoli":
      # No special rules for the rest
      return values

    if 350 <= temp <= 400: 
      # Target temperature 350 - 400°C, ideally around 375°C
      return values

    raise ValueError(f"""
      Napoli style require oven_temperature in 350-400°C range
      Given: {temp}°C""")

Neapolitan pizzas, ideally around 375°C

BakedPizza with oven_temperature and rules based on house styles

@ahultner

hultner.se/

Pydantic

Validators

Now let's see if we bake 👨‍🍳 some invalid pizzas ⚠️🚨

try: 
    BakedPizza(
      style="Panpizza", 
      toppings=["tomato sauce"], 
      oven_temperature=250
    )
except ValidationError as err:
    print(err)

1 validation error for BakedPizza
style
  We only cook the following styles: ('Napoli', 'Roman', 'Italian'), given: Panpizza (type=value_error)

try: 
    BakedPizza(
      style="Napoli", 
      toppings=["tomato sauce"], 
      oven_temperature=300
    )
except ValidationError as err:
    print(err)

1 validation error for BakedPizza
__root__
  Napoli pizzas require a oven_temperature in the range of 350 - 400°C, given: 300°C (type=value_error)

@ahultner

hultner.se/

Pydantic

Validators, functions(…)

Gosh these runtime type checkers are rather useful, but what about functions?

Pydantic got you covered with @validate_arguments.

Still in beta, API may change, release 2020-04-18 in version 1.5

from pydantic import validate_arguments

# Validator on function
# Ensure that we use a valid pizza when making orders
@validate_arguments
def make_order(pizza: Pizza):
    ...

try:
    make_order({
        "style":"Napoli",
        "toppings":(
          "tomato sauce", 
          "mozzarella", 
          "prosciutto", 
          "pineapple",
        ),
    })
except ValidationError as err:
    print(err)

1 validation error for MakeOrder
pizza -> toppings -> 3
  value is not a valid enumeration member; permitted: 'mozzarella', 'tomato sauce', 'prosciutto', 'basil', 'rucola' (type=type_error.enum; enum_values=[<Topping.mozzarella: 'mozzarella'>, …])

@ahultner

hultner.se/

FastAPI

Pydantic-driven APIs

Lean micro framework similar to flask
Automatic OpenAPI-specs
Tight integration with pydantic
Async ASGI

from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()

def make_order(pizza: Pizza):
    # Business logic for making an order
    pass

def dispatch_order(pizza: BakedPizza):
    # Hand over pizza to delivery company
    pass

# Deliver a baked pizza
@app.post("/delivery/pizza")
async def deliver_pizza_order(pizza: BakedPizza):
    dispatch = dispatch_order(pizza)
    return dispatch

@app.post("/order/pizza")
async def order_pizza(pizza: Pizza):
    order = make_order(pizza)
    return order

This is everything we need to create a small API around our models.

@ahultner

hultner.se/

That's the beginning

But there is more…

That's it, a quick introduction to pydantic!

But this is just the tip of the iceberg 🗻 and I want to give you a hint about what more can be done.

I'm not going to go into detail in any of this but feel free to ask me about it in the chat, on Twitter/LinkedIn or via email 💬📨

@ahultner

hultner.se/

And more!

Cool features worth mentioning

Post 1.0, reached this milestone about a year ago
Support for standard library types
Offer useful extra types for every day use
- Email
- HttpUrl (and more, stricturl for custom validation)
- PostgresDsn
- IPvAnyAddress (as well as IPv4Address and IPv6Address from ipaddress)
- PositiveInt
- PaymentCardNumber
- PaymentCardBrand.[amex, mastercard, visa, other] checks luhn, str of digits and BIN-based lenght.
- Constrained types (e.g. conlist, conint, etc.)
- and more…

Supports custom datatypes
Settings management
- Typed configuration management
- Automatically reads from environment variables
- Dotenv (.env) support via defacto standard python-dotenv.
ORM-mode
Recursive models
Works with mypy out of the box, mypy plugin further improves experience.
Postponed annotations, self-referencing models, PEP-563-style.
python-devtools intergration
PyCharm plugin
Fast compared to popular alternatives! But always make your own benchmarks for your own usecase if performance is important for you.

@ahultner

hultner.se/

Conclusion

Pure python syntax
Better validation
Very useful JSON-tools for API's
Easy to migrate from dataclasses
Lots of useful features
More things comming
- Very active development
- Working on strict mode
Try it out!

@ahultner

hultner.se/

Questions

Contact me if you have any further

questions.

Want to learn more?

Available for training, workshops and

freelance consulting.

@ahultner

contact@hultner.se

github.com/hultner

github.com/hultner-technologies/pydantic-introduction

hultner.se

slides.com/hultner

linkedin.com/in/ hultner

Python Pizza Remote 2020: Pydantic – Give your python Dataclasses super powers with pydantic

A quick talk about the fantastic pydantic library and how it makes your dataclasses better!

2,354

Alexander Hultnér

Founder, Hultnér Technologies (https://hultner.se). Want me to speak at your company? Corporate training? Or just a fresh pair of eyes on your project? Contact me for contracts.

Pydantic

Alexander Hultnér

Outline

Dataclasses

Dataclasses

Pydantic

Pydantic

Pydantic

Pydantic

Pydantic

Pydantic

Pydantic

Pydantic

Pydantic

Pydantic

Pydantic

FastAPI

That's the beginning

And more!

Conclusion

Questions

Links

Python Pizza Remote 2020: Pydantic – Give your python Dataclasses super powers with pydantic

More from Alexander Hultnér