Asynchronous Processing

Demystifying beanstalk and the workers

Queueing Theory

http://en.wikipedia.org/wiki/Queueing_theory

Why Do It?

1. Don't punish the caller because you're stuff is slow
2. Enhanced response time for end-users
3. Isolate calls to external resources
4. Lighten the load on your app server
5. Poor man's load throttling
6. I'm sure there's other excellent reasons...

Queue

Web

Application

Workers

DB

Queue ==> beanstalkd

(Technically, it's a priority queue)

Web Application == (abaqis, rqi)

Workers == (abaqis, rqi)

(In our implementation, the worker codebase is exactly the same as the Web Application codebase)

Queue

Web

Application

Workers

beanstalkd

• Lightweight priority queue implementation
• Supports multiple 'tubes' for queueing/dequeing
• Multiple states for stored jobs (ready, reserved, delayed, buried)
• Supports "time to run (TTR)" to deal with worker death/hang
• Default port: 11300
• Persistent storage of jobs (survives reboot)
• Service survives reboot (restarts automatically)
• Provides queryability

Workers

• Each worker runs in its own process
• As currently implemented, each worker takes up the same resources as its application (doesn't have to be that way)
• In general, a worker performs a single top-level method call.

Queued Jobs (Messages)

Static Style:

Resident.ayl_send(:clear_stay, 3, stay) yields =>

"Resident.clear_stay(3, Stay.find(32))"

 

Instance Style:

resident.ayl_send(:clear_stay, 3, stay) yields => 

"Resident.find(22).clear_stay(3, Stay.find(32))"

All the worker does is 'eval' the message string.

ayl/ayl_beanstalk

• RubyGem supporting beanstalk job submission and worker APIs.
• All Ruby objects get a set of async methods and marshalling support (ayl_send, ayl_send_opts)
• Provides executable script to query beanstalk (ayl_beanstalk) and manage jobs and tubes
• Provides executable daemon wrappers to instantiate and run worker daemons

abaqis_daemons

• RPM Package
• Provides multi-application configuration for workers and other daemons (e.g. scheduler) (/etc/abaqis/daemons.yml)
• Leverages ayl_beanstalk for workers
• Survives reboot (restarts workers/daemons)
• Since it's a first-class service, you can use standard Linux service control to stop/start/restart workers/daemons

Operations

• ayl_beanstalk list_tubes -
  • in general, there should be one named tube per application
  • tubes are named with the app version
  • workers must be using the same tube as application (duh)
• ayl_beanstalk -t <tube> tube_statistics
  • Shows the number of workers (available and busy)
  • Shows the number of ready jobs
• ayl_beanstalk
  • Shows statistics for all tubes
• ayl_beanstalk -t <tube> kill_worker
  • Used to kill a worker on a particular tube
• ayl_beanstalk --help
  • Shows all the available commands

Local Testing

See https://sites.google.com/a/nursinghomequality.com/abaqis-development/abaqis-async-workers  for details