Dramatiq

A distributed task processing library

Bogdan Popa

RoPython-Cluj

2017-12-07

What is Dramatiq?

DISTRIBUTED

TASK PROCESSING

LIBRARY FOR PYTHON 3

Background Workers

Use Cases

Anything that needs to run outside of the request-response cycle (i.e. async tasks)
Anything that may need to be retried on failure
Background queries and number crunching
3rd party interactions (eg. with external APIs)
Long-running actions
Periodic tasks

Why A New Library?

Why Not celery?

I wanted a task queueing library that offered:

HIGH RELIABILITY

HIGH PERFORMANCE

A SIMPLE CORE

CONVENTION OVER CONFIGURATION

...and Celery didn't cut it

Feature Comparison

	Dramatiq	Celery	RQ
Python 2	🚫	✅	✅
Code auto-reload	✅	🚫	🚫
Locks and Rate Limiting	✅	🚫*	🚫
Queue priorities	✅	🚫**	✅
Simple implementation	✅	🚫	✅
Delayed tasks	✅	✅	🚫
Result storage	✅	✅	✅
Automatic retries	✅	🚫	🚫
Large community	🚫	✅	✅
Familiar	🚫	✅	✅
High Throughput	✅	✅	🚫

* Celery supports rate limiting at the worker level, but not globally.

** Celery supports prioritizing messages within a queue, but not across queues.

Getting Started

$ pipenv install "dramatiq[rabbitmq,watch]"

# example.py
import dramatiq

@dramatiq.actor
def add(x, y):
    print(x + y)

>>> from example import add
>>> add(1, 2)
3

1. Install it from PyPI

2. Write your actor

3. Call it like any other function

Running Workers

$ rabbitmq-server

4. Run RabbitMQ

6. Run Dramatiq and see it in action

5. Send the actor a message

$ env PYTHONPATH=. pipenv run dramatiq example
[2017-12-03 12:59:28,248] [PID 80336] [MainThread] [dramatiq.MainProcess] [INFO] Dramatiq '0.15.0' is booting up.
[2017-12-03 12:59:28,373] [PID 80338] [MainThread] [dramatiq.WorkerProcess(1)] [INFO] Worker process is ready for action.
[2017-12-03 12:59:28,374] [PID 80339] [MainThread] [dramatiq.WorkerProcess(2)] [INFO] Worker process is ready for action.
[2017-12-03 12:59:28,375] [PID 80340] [MainThread] [dramatiq.WorkerProcess(3)] [INFO] Worker process is ready for action.
[2017-12-03 12:59:28,375] [PID 80337] [MainThread] [dramatiq.WorkerProcess(0)] [INFO] Worker process is ready for action.
[2017-12-03 12:59:28,378] [PID 80344] [MainThread] [dramatiq.WorkerProcess(7)] [INFO] Worker process is ready for action.
[2017-12-03 12:59:28,380] [PID 80342] [MainThread] [dramatiq.WorkerProcess(5)] [INFO] Worker process is ready for action.
[2017-12-03 12:59:28,380] [PID 80343] [MainThread] [dramatiq.WorkerProcess(6)] [INFO] Worker process is ready for action.
[2017-12-03 12:59:28,386] [PID 80341] [MainThread] [dramatiq.WorkerProcess(4)] [INFO] Worker process is ready for action.
[2017-12-03 12:59:40,155] [PID 80339] [Thread-7] [example.add] [INFO] Received args=(1, 2) kwargs={}.
3
[2017-12-03 12:59:40,155] [PID 80339] [Thread-7] [example.add] [INFO] Completed after 0.04ms.

>>> from example import add
>>> add.send(1, 2)
Message(queue_name='default', actor_name='add', args=(1, 2), kwargs={}, ...)

Demo Time

Guilt, an e-commerce website

The Problem

Checkout

clears the user's cart and marks the skus "sold"
tells an external warehousing system to prepare the products: slow and failure-prone
sends the user an e-mail congratulating him/her on the purchase: slow

That's all folks!

https://dramatiq.io

@Bogdanp

bogdan@defn.io