Optimize Django Tests

Ivaylo Bachvarov

CTO

Agenda

  • Motivation
  • Quick code tips
  • Quick CI tips
  • Go hard or go home

Waiting for CI or Tests

The Test Runner

  • py.test for test runner

Settings files for tests

$ tree project/settings
project/settings
├── __init__.py
├── base.py
├── local.py
├── production.py
└── test.py

$ python manage.py test --settings project.settings.test
$ pytest --ds project.settings.test

$ cat pytest.init
[pytest]
DJANGO_SETTINGS_MODULE = config.settings.test

Quick Wins

$ cat project/settings/test.py


from .base import *

DJANGO_DEBUG = False

PASSWORD_HASHERS = [
   "django.contrib.auth.hashers.MD5PasswordHasher",
]

DEFAULT_FILE_STORAGE = "inmemorystorage.InMemoryStorage"
EMAIL_BACKEND = "django.core.mail.backends.locmem.EmailBackend"
CACHES = {
    "default": {"BACKEND": "django.core.cache.backends.locmem.LocMemCache"}
}
CELERY_ALWAYS_EAGER = True
CELERY_EAGER_PROPAGATES_EXCEPTIONS = True

Be latest and greatest!

  • Latest python
  • Latest Django
  • Latest PostgresSQL

Tests in Parallel 

$ python manage.py test --parallel
$ py.test --n auto

TestCase 1

TestCase 2

TestCase 3

TestCase 4

Tests in Parallel

TestCase 1

TestCase 2

TestCase 3

TestCase 4

Tests in Parallel

TestCase 1

Ready

TestCase 3

TestCase 4

Spit Test Cases

TestCase 1

Ready

Ready

Ready

Tests in Parallel

TestCase 1

Ready

TestCase 3

TestCase 4

Reuse database

TestCase 1

Ready

TestCase 3

TestCase 4

py.test --nomigrations
py.test --keepdb

Write Proper Tests

On CI

Don't change your database on CI

Cache dependencies

steps:
  - uses: actions/checkout@v1
  - name: Set up Python 3.6.10
    uses: actions/setup-python@v1
    with:
      python-version: 3.6.10
  - name: Cache pip
    uses: actions/cache@v1
    with:
      path: /opt/hostedtoolcache/Python/3.6.10/x64/ # This path is specific to Ubuntu
      # If the requirements files change, the cache will not hit. 
      key: python-${{ hashFiles('requirements/local.txt') }}-${{ hashFiles('requirements/base.txt') }}

GitHub Action Example

Don't cache ~/.cache/pip

Try with in memory postgres

$ docker run \
--mount type=tmpfs,destination=/var/lib/postgresql/data \
--detach \

Use the maximum of the CI hardware

If build is still slow get more hardware

try: https://pypi.org/project/pytest-xdist/

💰

My Article About Django Test Speed

Speed Up Your Django Tests

Optimize Django Tests

By Hack Bulgaria

Made with Slides.com

Optimize Django Tests

  • 824

More from Hack Bulgaria