Scaling job queues with RabbitMQ
Tu Tran - Solution Architect
1. Motivation
1. Monitoring & debugging
- DB jobs: MongoDB
- Schedule Jobs
- Message queues: NATS, Kafka
Motivation
2. Redundant Resources
Motivation
3. Scalability
More than one?
Motivation
3. Scalability
Kafka issues?
Motivation
4. Reliability
NATS - Lost data?
Motivation
4. Reliability
Duplicate issue?
Bugs?
Something else?
1. Motivation
1. Redundant Resources
2. Monitoring & debugging
3. Scalablity
4. Reliability
2. Current tech stack
Lock to prevent duplicate
Store job queues
Crontab to schedule jobs
2. Current tech stack
Pub/sub
2. Current tech stack
Job queues
3. Solution
RabbitMQ: Job queues
3. Solution
RabbitMQ: Pub/sub
3. Solution
RabbitMQ: Acknowledgements and Confirms
+ Durable
Error handling
- Prevent stuck jobs
- Retry jobs
- Monitoring
4. Demo
5. Multi-tenant
Still not optimize yet??
Redundant Resources???
5. Multi-tenant
How we move Merchize platform from single-tenant to multi-tenant
Next talk: