Building

a custom  model field

from the ground up

Dmitry Dygalo

About me

  • Tech Lead at Kiwi.com
  • Prague, Czech Republic
  • Python since 2010
  • Co-maintain django-money since 2016
  • Hobbies: Open source & traveling 

Custom model fields

why?

  • Map custom DB types into Python objects
  • Store complex Python objects in the DB

Database

Domain

Case study

Django money

About the project:

  - Integrates Money object to Django ORM

  - Forms, admin integration, template tags, DRF integration, currency rates

Timeline:

  - Forked in May, 2011

  - My first commit in Sep, 2013

  - Co-maintainer since 2016

  - First meet with Benjamin Bach

    (co-maintainer) on Django Europe 2019

Storage

  - Mapping Money to Model

  - Underlying storage

  - Descriptors

 

Queries

  - Lookup API

  - Expressions

Overview

Extras

  - Migrations

  - Serialization

  - Validators

storage

Domain

>>> from moneyed import Money

>>> price = Money(100, "DKK")
>>> price
<Money: 100 DKK>

>>> price.amount
Decimal('100')
>>> price.currency
DKK

Money

  - amount

  - currency

Features

  - arithmetic

  - localization

>>> Money(100, "DKK") + Money(200, "DKK")
<Money: 300 DKK>

>>> from moneyed.localization import format_money

>>> format_money(
    Money(10, "USD"), 
    locale="en_US"
)
$10.00

TIP: Use `Decimal` / `NUMERIC` for money amounts

Django ORM

from django.db import models

from djmoney.models.fields import MoneyField


class Item(models.Model):
    """Item to purchase."""
    name = models.CharField(max_length=255)
    price = MoneyField(...)

    def __str__(self):
        return self.name

Model

Design the interface

>>> bread = Item(
    name="Bread", 
    price=Money(100, "DKK")
)
>>> bread.save()


>>> apple = Item.objects.get(
    name="Apple"
)
>>> apple.price
<Money: 20 DKK>


>>> apple.price = F("price") * 2
>>> apple.save()
>>> apple.refresh_from_db()
>>> apple.price
<Money: 40 DKK>

TIP: Make test cases from it

Basic actions:

  - Create

  - Get

  - Update

No complex queries yet

Design the interface

def test_save():
    price = Money(100, "DKK")
    bread = Item(
        name="Bread",
        price=price
    )
    bread.save()
    assert bread.price == price

def test_get(apple):
    loaded_apple = Item.objects.get(
        name="Apple"
    )
    assert loaded_apple.price == Money(20, "DKK")

def test_update(apple):
    apple.price = F("price") * 2
    apple.save()
    apple.refresh_from_db()
    assert apple.price == Money(40, "DKK")

Storage level

CREATE TABLE item
(
    name           VARCHAR(255),
    price          NUMERIC,
    price_currency VARCHAR(3)
);

INSERT INTO item VALUES (
    'apple', 
    10, 
    'DKK'
);

SELECT 
    item.name, 
    item.price, 
    item.price_currency 
FROM item;

-- Will output
 name  | price | price_currency 
-------+-------+----------------
 apple |    10 | DKK
(1 row)

Separate fields

Pros:

  - Standard SQL compliant

Cons:

  - Managing two fields at once is not trivial

Connect DB to Django

Money

DecimalField

NUMERIC

VARCHAR(3)

CharField

amount

currency

Separate fields

Descriptor

what is that?

Customize attribute access

class CoolDescriptor(object):

    def __init__(self, val):
        self.val = val

    def __get__(self, obj, objtype) -> Any: 
        # Get attribute
        return self.val

    def __set__(self, obj, val) -> None:
        # Set attribute
        self.val = val

    def __delete__(self, obj) -> None:
        # Delete attribute
        del self.val
>>> class Item(object):
...     x = CoolDescriptor(42)

>>> item = Item()

>>> item.x
# CoolDescriptor.__get__ is called


>>> item.x = 0
# CoolDescriptor.__set__ is called


>>> del item.x
# CoolDescriptor.__delete__ is called

Descriptors

class MoneyField(models.DecimalField):

    def contribute_to_class(
        self, 
        cls, 
        name,
        private_only=False,
    ):
        super().contribute_to_class(
            cls, name
        )
        # Add currency field here

        setattr(
            cls, 
            self.name, 
            MoneyFieldProxy(),
        )
class MoneyFieldProxy:

    def __get__(self, obj, objtype=None):
        ...
        # field names are hardcoded 
        # for illustration purposes
        return Money(
            obj.__dict__["price"],
            obj.__dict__["price_currency"]
        )

    def __set__(self, obj, value):
        setattr(
            obj, 
            "price_currency", 
            value
        )
        obj.__dict__["price"] = value

TIP: Customize contribute_to_class

Does it seem hacky?

Alternative

CREATE TYPE djmoney AS (
    amount   NUMERIC,
    currency VARCHAR(3)
);

CREATE TABLE item
(
    name  VARCHAR(255),
    price djmoney
);

INSERT INTO item VALUES (
    'apple', 
    '(10,DKK)'
);

SELECT 
    item.name, 
    (item.price).amount, 
    (item.price).currency
FROM item;

NOTE: PostgreSQL syntax

Pros:

  - Standard SQL compliant

  - Implies a usual way of implementing custom fields in Django

structured type

Cons:

  - Not supported by MySQL

  - Overhead for attribute access

Alternative

class MoneyField(models.Field):

    def db_type(self, connection):
        # PostgreSQL already has `money` type
        # which is a different thing
        return 'djmoney'

    def from_db_value(self, value, expression, connection):
        if value is None:
            return None
        amount, currency = value[1:-1].split(',')
        return Money(amount, currency)

    def get_prep_value(self, value):
        if value is None:
            return None
        amount = value.amount
        currency = value.currency
        return f'({amount},{currency})'

structured type

Classic Django custom field

Summary: part 1

  • Design the interface
  • Discover DB queries
  • Decide on DBMS support
  • One field is simpler than multiple fields
  • Try structured types for composite values

Querying Money field

Querying Money field

>>> Item.objects.filter(
    price__gt=Money(10, "EUR")
)

>>> Item.objects.filter(
    price__currency="DKK"
)

>>> Item.objects.filter(
    price__amount__gt=100
)

>>> Item.objects.filter(
    price=F("discount_price")
)

>>> Item.objects.filter(
    price=None
)

Queries:

  - Lookups

  - Transforms

  - Expressions

TIP: Define behavior unambiguously

Database queries

price__gt=Money(10, "EUR")

price__currency="DKK"

price__amount__gt=100

price=F("discount_price")

WHERE (price).amount > 10 AND                                                    (price).currency = 'EUR'

WHERE (price).currency = 'DKK'

WHERE (price).amount > 100

WHERE (price).amount = (discount_price).amount
      AND (price).currency = (discount_price).currency

lookup api

Lookups

# Item.objects.filter(price__gt=Money(10, "EUR"))


@MoneyField.register_lookup
class MoneyGreaterThan(models.lookups.GreaterThan):

    def as_sql(self, compiler, connection):
        # Construct left-hand side, right-hand side and parameters
        return (
            "(%(lhs)s).amount > %%s AND "
            "(%(lhs)s).currency = %%s" % {"lhs": lhs},
            params,
        )

TIP: Reuse code from existing lookups

lookup api

transforms

# Item.objects.filter(price__amount__gt=100)

@MoneyField.register_lookup
class Amount(models.Transform):
    lookup_name = "amount"
    output_field = models.DecimalField()

    def as_sql(self, compiler, connection):
        lhs, params = compiler.compile(self.lhs)
        return "(%(lhs)s).amount" % {"lhs": lhs}, params


# Item.objects.filter(price__currency="DKK")

@MoneyField.register_lookup
class Currency(models.Transform):
    lookup_name = "currency"
    output_field = models.CharField(max_length=3)

    def as_sql(self, compiler, connection):
        lhs, params = compiler.compile(self.lhs)
        return "(%(lhs)s).currency" % {"lhs": lhs}, params

Expressions

Basic expressions will work out of the box

>>> Item.objects.bulk_create([
        Item(name="apple", price=Money(10, "EUR")),
        Item(name="orange", price=None),
        Item(name="kiwi", price=Money(15, "EUR")),
])

>>> Item.objects.values("price").annotate(number=Count("*"))

<QuerySet [
    {'price': <Money: 10 EUR>, 'number': 1}, 
    {'price': <Money: 15 EUR>, 'number': 1}, 
    {'price': None, 'number': 1}
]>

>>> Item.objects.order_by(F("price").desc(nulls_last=True))

<QuerySet [
    <Item: kiwi: 15.00 €>, 
    <Item: apple: 10.00 €>, 
    <Item: orange: None>
]>

Expressions

F

TIP: Extend magic methods to support F expressions

import moneyed


class Money(moneyed.Money):

    def __add__(self, other):
        # in `py-moneyed`
        # __radd__ is equal to __add__
        if isinstance(other, F):
            return other.__radd__(self)
        return super().__add__(other)

    def __sub__(self, other):
        ...

    def __mul__(self, other):
        ...

F expressions support:

  - Extend magic methods

  - Adapt lookups implementation

Expressions

structured types

TIP: Create custom expressions if needed

from django.db.models import Expression


class SubColumn(Expression):
    def __init__(
        self, base, sub, output_field=None
    ):
        super().__init__(
            output_field=output_field
        )
        self.base, self.sub = (base, sub)

    def as_sql(self, compiler, connection):
        qn = compiler.quote_name_unless_alias
        return (
            "(%s).%s"
            % (qn(self.base), qn(self.sub)),
            [],
        )

>>> Item.objects.values(
    currency=SubColumn(
        "price", 
        "currency", 
        output_field=models.CharField(
            max_length=3
        )
    )
).annotate(number=Count("*"))

<QuerySet [{'currency': 'DKK', 'number': 1}]>

summary: part 2

  • Define lookups & transforms behavior unambiguously
  • Map them to DB queries
  • Use Lookup API to build desired queries
  • Extend magic methods on your entities for `F`  support
  • Create custom expressions for structured types

extras

migrations

class MoneyField(models.Field):

    def __init__(
        self, default_currency="DKK", **kwargs
    ):
        self.default_currency = default_currency
        # some logic for this default currency
        super().__init__(**kwargs)

    def deconstruct(self):
        name, path, args, kwargs = super(
        ).deconstruct()
        # add your custom options for `__init__`
        # to `kwargs`
        kwargs[
            "default_currency"
        ] = self.default_currency
        return name, path, args, kwargs

TIP: extend `deconstruct` for custom field options

field options

operations = [
  migrations.CreateModel(
    name='Item',
    fields=[
      ...,
      (
        'amount', 
        djmoney.models.fields.MoneyField(
          default_currency='DKK'
        )
      ),
      ...
    ]
  )
]

migrations

from django.utils.deconstruct import deconstructible


@deconstructible
class Money(moneyed.Money):
    ...

TIP: use `deconstructible` decorator

domain entities

class MoneyField(models.Field):

    def __init__(self, default=Money(10, "DKK"), **kwargs):
        ...

Serialization

from django.core.serializers.json import Serializer as JSONSerializer

class Serializer(JSONSerializer):

    def _value_from_field(self, obj, field):
        if isinstance(field, MoneyField):
            value = field.value_from_object(obj)
            if value:
                return {
                    "amount": value.amount, 
                    "currency": value.currency.code
                }
            return value
        return super()._value_from_field(obj, field)
# [
#   {
#     "model": "testapp.item", 
#     "pk": 1, 
#     "fields": {
#         "money": {"amount": "10.00", "currency": "EUR"}
#     }
#   }
# ]

Define `Serializer` class and `Deserializer` callable

Serialization

class MoneyField(models.Field):

    ...

    def get_prep_value(self, value):
        if value is None:
            return None
        # deserialization
        if isinstance(value, dict):
            amount = value["amount"]
            currency = value["currency"]
        else:
            amount = value.amount
            currency = value.currency
        return f"({amount},{currency})"

Update the field to support deserialization

Serialization

# djmoney/__init__.py
default_app_config = 'djmoney.apps.MoneyConfig'

TIP: use AppConfig for registering your extension points

register the module

from django.apps import AppConfig
from django.core import serializers


class MoneyConfig(AppConfig):
    name = "djmoney"

    def ready(self):
        serializers.register_serializer(
            "json", 
            "djmoney.serializers"
        )

validators

TIP: extend `django.core.validators`

from django.core.validators import (
    BaseValidator,
    MaxValueValidator,
    MinValueValidator,
)


class BaseMoneyValidator(BaseValidator):

    def get_limit_value(self, cleaned):
        # prepare self.limit_value for
        # comparison

    def __call__(self, value):
        cleaned = self.clean(value)
        limit_value = self.get_limit_value(cleaned)
        if self.compare(cleaned, limit_value):
            raise ValidationError(...)


class MinMoneyValidator(BaseMoneyValidator, MinValueValidator):
    pass


class MaxMoneyValidator(BaseMoneyValidator, MaxValueValidator):
    pass

validators

class BankAccount(models.Model):
    balance = MoneyField(
        max_digits=10,
        decimal_places=2,
        validators=[

            MinMoneyValidator(10),
            MaxMoneyValidator(1500),

            MinMoneyValidator(Money(500, 'NOK')),
            MaxMoneyValidator(Money(900, 'NOK')),

            MinMoneyValidator(
                {'EUR': 100, 'USD': 50}
            ),
            MaxMoneyValidator(
                {'EUR': 1000, 'USD': 500}
            ),

        ]
    )

Rules:

  • All input values should be between 10 and 1500 despite on currency;
  • Norwegian Crowns amount (NOK) should be between 500 and 900;
  • Euros should be between 100 and 1000;
  • US Dollars should be between 50 and 500;

Django Admin

For structured types will work automatically

# DO NOT DO THIS
import django.contrib.admin.utils as admin_utils


def setup_admin_integration():
    original_display_for_field = admin_utils.display_for_field

    def display_for_field(value, field, empty):
        if isinstance(field, MoneyField):
            return text_type(value)
        return original_display_for_field(value, field, empty)

    setattr(admin_utils, 'display_for_field', display_for_field)

separate fields

summary: part 3

  • Extend existing tools from Django
  • Use AppConfig to register your extensions
  • Think about possible use cases for your field

summary

  • Start with the interface design (and use it for tests)
  • Explore underlying DB queries
  • Experiment with your implementation, choose the most simple and unambiguous
  • Try to evaluate possible consequences of chosen approach
  • Django provides a lot of extendable tools for mapping your domain entities to the DB. Use them

@Stranger6667

thank you

Building a custom model field from the ground up

By stranger6667

Made with Slides.com

Building a custom model field from the ground up

  • 1,601

More from stranger6667