![](https://s3.amazonaws.com/media-p.slid.es/uploads/1097612/images/6783180/celerylogo.png)
elery is digestive
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1097612/images/6862983/184-1840509_celery-pose-apex-object-show-green.png)
@stalwartcoder
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1097612/images/6868527/new_agenda.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1097612/images/6783181/celery-is-useless1.jpg)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1097612/images/6856847/task-list.png)
Agenda
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1097612/images/6857079/minions_PNG49.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1097612/images/6366754/me_moji__1_.png)
Abhishek
π¨βπ» Software Eng. at Essentia SoftServ
π Pythonista
π¨βπ©βπ§βπ¦Β Community first person π
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1097612/images/6367001/pythonista.png)
connect with me:
π« not "10x Engineer"
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1097612/images/6859726/task2.gif)
Task Queue
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1097612/images/6860240/calendar.png)
Task Queue
Manage background work - outside the usual HTTP request-response cycle.
For example, a web app poll the API every 10 minutes to collect the names of the top crypto currencies.
A task queue would handle invoking code to call the API, process the results and store them in a persistent DB for
later use.
Input to task queue's are a unit of workΒ - separate code.
π¬
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1097612/images/6866905/envelope-with-downwards-arrow-above_1f4e9.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1097612/images/6866905/envelope-with-downwards-arrow-above_1f4e9.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1097612/images/6866905/envelope-with-downwards-arrow-above_1f4e9.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1097612/images/6866905/envelope-with-downwards-arrow-above_1f4e9.png)
Task Queue is a system for parallel execution of tasks.
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1097612/images/6866905/envelope-with-downwards-arrow-above_1f4e9.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1097612/images/6867792/why_.jpg)
Need of Task Queue... π΅οΈ
βΒ Useful in certain situation.
πΒ General guidelines :
- Are you using a lot of cron jobs to process data in the background?
- Does your application takes more than a few seconds to generate response?
- Do you wish you could distribute the processing of the data generated by your application among many servers?
- OR an asynchrounous DB update ?
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1097612/images/6868138/GW5A.gif)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1097612/images/6868155/arrow.gif)
distributes task
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1097612/images/6868187/broker_minion.png)
Broker
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1097612/images/6868155/arrow.gif)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1097612/images/6868155/arrow.gif)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1097612/images/6868155/arrow.gif)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1097612/images/6868189/worker_minion_1.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1097612/images/6868191/worker_minion_5.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1097612/images/6868188/worker_minion_2.png)
sends task
Client
Worker
Worker
Worker
distributes task
distributes task
WAIT!!
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1097612/images/6868269/wait_cat.gif)
Iβve heard Asynchronous before!
Yes, AJAX
Some of the response time issues can be solved :
- With AJAX responses that continually enhance the initial response.
- Only if the AJAX responses also complete within a reasonable amount of time.
You need a Queue :
- Long processing times canβt be avoided in generating responses.
- You want application data to be continuously processed in the background and readily available when requested.
Different Frameworks or Projects for TQΒ
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1097612/images/6868294/juggling-emoji-by-google.png)
π· RQ (Redis Queue)
π· Huey
π· CloudAMQP
Blah....blahh...blaahhh
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1097612/images/6868312/sparkles_2728.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1097612/images/6860400/celery_happy.gif)
Let's talk about *celery*
Do it....
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1097612/images/6783311/celery_name_logo.png)
Celery is an asynchronous task queue/job queue based on distributed message passing.
It is focused on real-time operation, and scheduling as well.
Tasks can execute asynchronously (in the background) or synchronously (wait until ready).
Easy to integrate & multi broker support .
Key concept
- Message Broker
- Worker - Executors
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1097612/images/6860317/celery__1_.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1097612/images/6860317/celery__1_.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1097612/images/6860317/celery__1_.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1097612/images/6860317/celery__1_.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1097612/images/6860317/celery__1_.png)
De facto standard for Python
Why should i use it ?
π€
it's just simple!
from celery import Celery
app = Celery('tasks', broker='pyamqp://guest@localhost//')
@app.task
def add(x, y):
return x + y
>>> from tasks import add
>>> add.delay(4, 4)
<AsyncResult: 889143a6-39a2-4e52-837b-d80d33efb22d>
Define task
Execute task
$ celery -A tasks worker --loglevel=info
Run the celery worker
$ pip install celery
install celery
- Minimize request/response cycle
- Smoother user experience
- Offload time/cpu intensive processes
- Flexibility - many points of customization
- Supports web hook, job routing
- Remote control of workers
- Logging, debugging & monitoring (Flower) support.
- Wide support of web frameworks actively developed.
- Amazing documentation and lot of tutorials!
π οΈ Framework Support
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1097612/images/6860554/web2py.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1097612/images/6860552/Pyramid_web_framework_logo_on_transparent_background.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1097612/images/6860551/tornado.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1097612/images/6860548/django-logo-positive.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1097612/images/6860549/flask.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1097612/images/6860550/pylons.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1097612/images/6863005/61HW01c6VfL.png)
client 1
CeleryWorker 1
client 2
Broker
Task Result Storage
Task Queue 1
Task Queue 2
Task Queue N
...
send tasks
send tasks
distribute tasks
distribute tasks
get task result
store task result
store task result
Celery Architecture
CeleryWorker 2
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1097612/images/6868470/prerequisite.jpg)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1097612/images/6868483/1.png)
Celery
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1097612/images/6868485/2.png)
Broker
![](https://miro.medium.com/max/540/1*ngGAAMf8Sw4jCJ_nP2Qt5A.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1097612/images/6868484/3.png)
DB for task result
RabbitMQ
Redis
AWS SQS
Selecting BrokerΒ
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1097612/images/6868506/ignore_select__804588.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1097612/images/6868509/Screenshot_2019-12-08_at_6.48.06_AM.png)
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1097612/images/6868522/demo.gif)
Finally !! π€¦ββοΈ
Approach for using celery π¨βπ»
- Simple setup : single queue & single workerββ
- Large setup : multiple queue & dedicated worker with concurrency patternβ
- Key choicesΒ :ββ
- Broker
- Result store
More detail :
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1097612/images/6868538/waitamin.png)
Some Tips π¬
- Start simple !
- Watch for the result .
- Monitor it !Β
- Consume faster than you produce.
- Try concurrency over tasks.
- Keep your task clean.
- Replace your cron jobs!
More:
Resources π
Conclusion π‘
- Take a slow task / function
- Decouple it from system
- Run it asynchronously
- Don't use it for everything, they are not magic wand !
![](https://s3.amazonaws.com/media-p.slid.es/uploads/1097612/images/6863015/tahnks.gif)
celery is digestive
By Abhishek Mishra
celery is digestive
- 923