Dramatiq

A distributed task processing library

Bogdan Popa

RoPython-Cluj

2017-12-07

What is Dramatiq?

DISTRIBUTED

TASK PROCESSING

LIBRARY FOR PYTHON 3

Background Workers

User
User
Crontab
Crontab
Script
Script
Broker (RMQ)
Broker (RMQ)
Worker 1
Worker 1
Worker 2
Worker 2
Worker N
Worker N
...
...

Use Cases

  • Anything that needs to run outside of the request-response cycle (i.e. async tasks)
  • Anything that may need to be retried on failure
  • Background queries and number crunching
  • 3rd party interactions (eg. with external APIs)
  • Long-running actions
  • Periodic tasks

Why A New Library?

Why Not celery?

I wanted a task queueing library that offered:

HIGH RELIABILITY

HIGH PERFORMANCE

A SIMPLE CORE

CONVENTION OVER CONFIGURATION

...and Celery didn't cut it

Feature Comparison

Dramatiq Celery RQ
Python 2 🚫 ✅ ✅
Code auto-reload ✅ 🚫 🚫
​Locks and Rate Limiting ✅ 🚫* 🚫
Queue priorities ✅ 🚫** ✅
Simple implementation ✅ 🚫 ✅
Delayed tasks ​✅ ​✅ 🚫
Result storage ​✅ ​✅ ✅
Automatic retries ✅ 🚫 🚫
Large community 🚫 ✅ ✅
Familiar 🚫 ✅ ✅
High Throughput ✅ ✅ 🚫

* Celery supports rate limiting at the worker level, but not globally.

** Celery supports prioritizing messages within a queue, but not across queues.

Getting Started

$ pipenv install "dramatiq[rabbitmq,watch]"
# example.py
import dramatiq

@dramatiq.actor
def add(x, y):
    print(x + y)
>>> from example import add
>>> add(1, 2)
3

1. Install it from PyPI

2. Write your actor

3. Call it like any other function

Running Workers

$ rabbitmq-server

4. Run RabbitMQ

6. Run Dramatiq and see it in action

5. Send the actor a message

$ env PYTHONPATH=. pipenv run dramatiq example
[2017-12-03 12:59:28,248] [PID 80336] [MainThread] [dramatiq.MainProcess] [INFO] Dramatiq '0.15.0' is booting up.
[2017-12-03 12:59:28,373] [PID 80338] [MainThread] [dramatiq.WorkerProcess(1)] [INFO] Worker process is ready for action.
[2017-12-03 12:59:28,374] [PID 80339] [MainThread] [dramatiq.WorkerProcess(2)] [INFO] Worker process is ready for action.
[2017-12-03 12:59:28,375] [PID 80340] [MainThread] [dramatiq.WorkerProcess(3)] [INFO] Worker process is ready for action.
[2017-12-03 12:59:28,375] [PID 80337] [MainThread] [dramatiq.WorkerProcess(0)] [INFO] Worker process is ready for action.
[2017-12-03 12:59:28,378] [PID 80344] [MainThread] [dramatiq.WorkerProcess(7)] [INFO] Worker process is ready for action.
[2017-12-03 12:59:28,380] [PID 80342] [MainThread] [dramatiq.WorkerProcess(5)] [INFO] Worker process is ready for action.
[2017-12-03 12:59:28,380] [PID 80343] [MainThread] [dramatiq.WorkerProcess(6)] [INFO] Worker process is ready for action.
[2017-12-03 12:59:28,386] [PID 80341] [MainThread] [dramatiq.WorkerProcess(4)] [INFO] Worker process is ready for action.
[2017-12-03 12:59:40,155] [PID 80339] [Thread-7] [example.add] [INFO] Received args=(1, 2) kwargs={}.
3
[2017-12-03 12:59:40,155] [PID 80339] [Thread-7] [example.add] [INFO] Completed after 0.04ms.
>>> from example import add
>>> add.send(1, 2)
Message(queue_name='default', actor_name='add', args=(1, 2), kwargs={}, ...)

Demo Time

Guilt, an e-commerce website

The Problem

Checkout

  1. clears the user's cart and marks the skus "sold"
  2. tells an external warehousing system to prepare the products: slow and failure-prone
  3. sends the user an e-mail congratulating him/her on the purchase: slow

That's all folks!

https://dramatiq.io

@Bogdanp

bogdan@defn.io

Dramatiq

By Bogdan Popa

Dramatiq

An introduction to https://dramatiq.io.

  • 1,607