Understanding Celery

WHAT IS CELERY

Celery is an asynchronous task queue/job queue based on distributed message passing.

It is focused on real-time operation, but supports scheduling as well.

def send_email_for_contest(participant_ids, send_invitation_mail, contest_slug):
    Contest = apps.get_model('contest', 'Contest')
    Contest_participants = apps.get_model('teams', 'Contest_participants')
    contest = Contest.objects.get(slug=contest_slug)
    ctx = {
        'contest': contest,
        'site_url': get_site_url(),
    }

    participants = Contest_participants.objects.filter(id__in=participant_ids)

    for participant in participants:
        ctx["participant"] = participant
        email = mail_builder.contest_welcome_participant(participant.participant, ctx)
        email.send()

Single email takes 2 seconds to process

if there are 200 emails to be sent

Total Time = 200*2 ~Approx 7 mins

Uses-> Everything that can be out of Request-Response cycle

Anything heavy:

Sending Mails
Computing something
Taking backups
Data Sync
Push Notifications

Application

Task Queues

WORKER

Install Celery

pip install celery

def some_function():
    #some code here
    
    #calling task
    task_id = send_emails_for_contest(participant_ids, send_invitation_mail, contest_slug)

def some_function():
    #some code here
    
    #calling task
    task_id = send_emails_for_contest.delay(participant_ids, send_invitation_mail, contest_slug)

Setup a Broker Database

Text

from celery import Celery
from .utils import get_site_url, mail_builder

BROKER_URL = ('redis://localhost:6379')

app = Celery('tasks',borker=BROKER_URL)

from celery import Celery
from .utils import get_site_url, mail_builder

BROKER_URL = ('redis://localhost:6379')

app = Celery('tasks',borker=BROKER_URL)


@app.task(name='send_email_for_contest')
def send_email_for_contest(participant_ids, send_invitation_mail, contest_slug):
    Contest = apps.get_model('contest', 'Contest')
    Contest_participants = apps.get_model('teams', 'Contest_participants')
    contest = Contest.objects.get(slug=contest_slug)
    ctx = {
        'contest': contest,
        'site_url': get_site_url(),
    }

    participants = Contest_participants.objects.filter(id__in=participant_ids)

    for participant in participants:
        ctx["participant"] = participant
        email = mail_builder.contest_welcome_participant(participant.participant, ctx)
        email.send()

celery -A tasks worker --loglevel=info

Retry/ create new tasks

Peroidic tasks

CeleryBeat

Schedules Types:

    Timedelta schedules
    Crontab Schedules
    Solar Schedules

CELERY_TIMEZONE = 'Asia/Kolkata'

CELERYBEAT_SCHEDULE = {
    'notify-start-10-minutes': {
        'task': 'tasks.send_start_notification',
        'schedule': datetime.timedelta(minutes=10),
    },
}

Timedelta schedules

CELERY_TIMEZONE = 'Asia/Kolkata'

CELERYBEAT_SCHEDULE = {
    'weekly': {
        'task': 'tasks.send_start_notification',
        'schedule': crontab(hour=17, minute=30, day_of_week=5),
    },
}

Crontab schedules

from celery.schedules import solar

CELERYBEAT_SCHEDULE = {
    'send_at_sunset': {
        'task': 'tasks.send_start_notification',
        'schedule': solar('sunset',32.2190, 76.3234),
    },
}

Solar schedules

Available event values are:

dawn_astronomical
dawn_nautical
dawn_civil
sunrise
solar_noon
sunset
dusk_civil
dusk_nautical
dusk_astronomical

Thanks

Questions??

CELERY

By Abhyuday Pratap Singh