Ruby Background Jobs with GoWorker

About me

Julio López

@theblasfem

Background Jobs

A computer process that runs "behind the scenes" (in the background) and without user intervention

Redis

http://redis.io

Redis is an open source NOSQL key-value data store with multiple datatypes: strings, hashes, lists, sets,sorted sets, bitmaps, etc.

Resque

https://github.com/resque/resque

Redis-backed library for creating background jobs, placing those jobs on multiple queues, and processing them later.

Created and used by github

https://github.com/blog/542-introducing-resque

GoWorker

http://www.goworker.org/

A go based background worker for Resque. 

• A simple binarie
• Polls Resque queues
• Dispatches jobs via go routines
• Use Ruby to push the jobs and the concurrency of Go to minimize job latency

How fast is it?

For simple dequeuing, deserialization, execution, and updating stats, goworker is 80x faster than Resque and 23x faster than Sidekiq.

Additionally, goworker only used 7MB of memory, while Sidekiq used 74MB and Resque 62MB.

Usage

Write a Go worker

/worker.go

package main

import (
  "fmt"
  "github.com/benmanns/goworker"
)

func main() {
  if err := goworker.Work(); err != nil {
    fmt.Println("Error:", err)
  }
}

Write your worker func

func init() {
  goworker.Register("Hello", helloWorker)
}

func helloWorker(queue string, args ...interface{}) error {
  fmt.Printf("Hello %v", args[0])
  return nil
}

Build and Run it

$ go build

$ ./worker -queues=hello

Build your go file to a binary

Start the Redis Server

$ redis-server

Run your go worker

Enqueue the job

require 'resque'

class Hello
end

Resque.push('hello', 'class' => 'Hello', 'args' => ["World"])

hello.rb

Or you can test it using the redis command line:

$ redis-cli -r 1 RPUSH resque:queue:hello '{"class":"Hello","args":["World"]}'

Demo

Sending mails with GoWorker

https://github.com/TheBlasfem/goworker-mails

Thanks!