Five Sticks of CELERY

1. Requirements

PIP

celery[librabbitmq]==3.1.11
django-celery==3.1.10

sudo apt-get install rabbitmq-server # go for >= 2.7.3
sudo rabbitmqctl add_user eedeep <secret password>
sudo rabbitmqctl add_vhost eedeep_host
sudo rabbitmqctl set_permissions -p eedeep_host eedeep ".*" ".*" ".*"

# For the rabbitmq management console
sudo rabbitmqctl set_user_tags cottonon administrator
sudo rabbitmq-plugins enable rabbitmq_management

2. Monitoring

flower
(but no persistent backend)

djcelery tombstones
(needs celerycam)

3. Retry

@app.task(bind=True, default_retry_delay=60 * 60, max_retries=24)
def push_subscriber(self, subscriber):
    try:
       push_to_some_unreliable_service(subscriber)
    except NoConnectionException:
        log.info("Could not connect to unreliable service, retrying later")
        raise self.retry()

4. Partition your tasks

    from kombu import Exchange, Queue
    CELERY_QUEUES = (
        Queue("default", Exchange("default"), routing_key="default"),
        Queue("important", Exchange("important"), routing_key="important"),
        Queue("medium", Exchange("medium"), routing_key="medium"),
    )

    CELERY_DEFAULT_QUEUE = "default"
    CELERY_DEFAULT_EXCHANGE = "default"
    CELERY_DEFAULT_EXCHANGE_TYPE = "direct"
    CELERY_DEFAULT_ROUTING_KEY = "default"

    CELERY_ROUTES = (
        {"shop.orders.create_order_in_backoffice": {
            "queue": "important",
            "routing_key": "important"}
        },
        {"spam.promotions.send_email_promotion": {
            "queue": "medium",
            "routing_key": "medium"}
        },
        # All other tasks will go to the default queue
    )

5. DEBUGGING

CELERY_ALWAYS_EAGER = True

This makes all your celery tasks run

*synchronously*, which makes interactive

debugging easier (ie, possible).

IF You get stuck....

Thanks for listening!

@eedeep