Resque to Sidekiq,

another success story

Tamás Michelberger

Secret Sauce Partners, Inc.

Background processing. What? WHy?

• Not everything fits the request-response life-cycle
• Offload long running tasks
• Queues everywhere

Resque

• Redis-backed 
• Forking for reliability
• Go to solution 3-4 years ago

Problems with Resque

• Not well suited for millions of short running tasks
• Lot of worker processes
• Lack of monitoring (resque#1101)
• Lack of retry policy and queue priority
• Basically unmaintained

Resque alternatives

• Rocketjob
• Shoryuken

• Backburner

• Sneakers

• Sidekiq

Why Sidekiq

• We already use Redis
• We already run one on a much smaller scale
• Thread-based
• Very active development and
  awesome documentation

CHallenges

• Pushing a huge number of jobs to
  Redis can be slow
• Memory consumption
• Lost jobs due to killed processes

Pros after 3 weeks

• Performance improvements
• Better resource utilization
• Less worker processes
• Monitoring
• Automatic retry for failed jobs

Cons after 3 weeks

• Memory growth
• Automatic retry for failed jobs

Some numbers

We're hiring!

sspinc.workable.com

Thanks!

Questions?