All you need to know to get started

Fabio Pitino

Sr Software Engineer

fabio_pitino@symantec.com

(with Ruby examples)

What is a message broker?

 

I asked Wikipedia...

 

A message broker is an architectural pattern for message validation, transformation, and routing. It mediates communication among applications, minimising the mutual awareness that applications should have of each other in order to be able to exchange messages, effectively implementing decoupling.

What is RabbitMQ?

 

  • Open source project - created by Rabbit Technologies Ltd in 2007, currently part of Pivotal Software
  • Implements the AMQP (Advanced Message Queuing Protocol)
  • Built in Erlang programming language
  • Built using OTP (Open Telecom Platform) framework
  • Gateways: AMQP, HTTP, STOMP and MQTT
  • Supports many plug-ins

Basic building blocks

# producer.rb

require 'bunny'

connection = Bunny.new
connection.start

channel = connection.create_channel
exchange = channel.default_exchange

queue = channel.queue('tasks')

10.times do |n|
  exchange.publish("task-#{n}", routing_key: queue.name)
end

connection.close

Basic building blocks

# consumer.rb

require 'bunny'

connection = Bunny.new
connection.start

channel = connection.create_channel
queue = channel.queue('tasks')

begin
  queue.subscribe(block: true) do |_delivery_info, _properties, payload|
    puts "received #{payload}"
    # do some work with it...
  end

rescue => error
  puts error.inspect
  connection.close
end

Worker queue (simple)

# worker.rb

# setup omitted...
queue = channel.queue('tasks')

# using "block: true" is like using "channel.prefetch(1)"
queue.subscribe(manual_ack: true, block: true) do |delivery_info, _, payload|
  puts "received #{payload}"
  # do some work with it...
  channel.ack(delivery_info.delivery_tag)
end
  • 1 worker processes 1 message a the time
  • run multiple instances of the same worker to scale out

Worker queue (with prefetch)

# worker.rb

# setup omitted...
channel.prefetch(1)
queue = channel.queue('tasks')

queue.subscribe(manual_ack: true, block: true) do |delivery_info, _, payload|
  puts "received #{payload}"
  # do some work with it...
  channel.ack(delivery_info.delivery_tag)
end
  • 1 worker processes 1 message a the time
  • run multiple instances of the same worker to scale out

Worker pool

# worker.rb

POOL_SIZE = 10

# setup omitted...
channel.prefetch(POOL_SIZE)
queue = channel.queue('tasks')

queue.subscribe(manual_ack: true, block: false) do |delivery_info, _, payload|
  puts "received #{payload}"
  # do some work with it...
  channel.ack(delivery_info.delivery_tag)
end
  • 1 worker processes up to 10 messages concurrently
  • run 1 instance of the worker
  • "block" mode must be turned off

Persistency


# queue does not survive broker reboot
channel.queue('tasks')


# queue is persisted after broker reboot but not the messages in it
channel.queue('tasks', durable: true)


# queue and messages persisted after broker reboot
queue = channel.queue('tasks', durable: true)
queue.publish(message, persistent: true)

slow

fast

  • By default queues and messages are not persisted
  • If a queue is declared once as "durable" it cannot be re-declared non-durable
  • Queues and Exchanges can also be created using RabbitMQ Admin app.

Exchanges

  • The core idea of RabbitMQ is that the Producer always publishes to an Exchange
  • A Producer knows nothing about it's consumers
  • Types of exchanges available: direct, topic, headers, fanout
  • So far it seems like we have been publishing directly to a queue but we have been implicitly publishing to the "Default Exchange"
# so far we have been using this:
channel.queue('tasks').publish(message)



# ...which is equivalent to this:
channel.default_exchange.publish(message, routing_key: 'tasks')

Pub/Sub with a Fanout Exchange

# publisher.rb

exchange = channel.fanout('logs')
exchange.publish(message)




# subscriber.rb

# declare a temporary queue owned by the subscriber
# with auto-generated name: amq.gen-JzTY20BRgKO-HjmUJj0wLg
queue = channel.queue('', exclusive: true) 

# bind queue to the specific fanout exchange
queue.bind('logs')

# at this point messages published to the "logs" exchange 
# will be copied to each queue bound to it
queue.subscribe(block: true) do |_delivery_info, _, payload|
  puts "received: #{payload}"
end

Routing with Direct Exchange

# publisher.rb
exchange = channel.direct('logs')
exchange.publish(message, routing_key: severity)


# consumer.rb [info] [warn] [error]
exchange = channel.direct('logs')
queue = channel.queue('', exclusive: true)

ARGV.each do |severity|
  queue.bind(exchange, routing_key: severity
end

queue.subscribe(block: true) do |_delivery_info, _, payload|
  puts "received #{payload}"
end

Imagine that for our logging system we have different consumers interested in different severity of messages

More flexibility with Topic Exchange

Direct exchange cannot route message based on multiple criteria.

Enter the Topic Exchange.

 

Messages must contain a routing_key that is a list of words delimited by "dots":

  • stocks.usd.nyse
  • syslog.info
  • europe.ireland.dublin

 

Routing key can contain special characters:

  • * (star) as substitute for 1 word
  • # (hash) as substitute for 0 or more words

Topic Exchange routing: example

A weather forecast server streams real-time weather information to various backend services around the world.

 

Format of the routing key: <continent>.<country>.<city>

 

consumers can use various forms of routing keys to get data:

  • asia.russia.moscow for data related to Moscow city - equivalent to Direct exchange
  • europe.italy.* for all data related to Italy
  • north_america.# for all data related to north America
  • *.*.dublin for both Dublin (IE) and Dublin (OH, AM)
  • # for all data - equivalent to Fanout exchange

Topic Exchange for the win!!!

Well! turns out that...

  • Topic exchange is the slowest at routing messages
  • Direct exchange is much faster
  • Fanout exchange is the fastest

Logging system with a Topic Exchange

# publisher.rb
exchange = channel.topic('logs')
exchange.publish(message, routing_key: "#{source}.#{severity}")



# consumer.rb [key]
# e.g: consumer "rails.info"
# e.g: consumer "syslog.*" "kern.error"
# e.g: consumer "#"
exchange = channel.topic('logs')
queue = channel.queue('', exclusive: true)

ARGV.each do |key|
  queue.bind(exchange, routing_key: key
end

queue.subscribe(block: true) do |delivery_info, _, payload|
  puts "received #{delivery_info.routing_key}: #{payload}"
end

Let's extend the logging system with a more flexible approach

Remote procedure call via RabbitMQ

# server.rb
exchange = channel.default_exchange
queue = channel.queue('fibonacci_server')

queue.subscribe(block: true) do |_delivery, properties, payload|
  result = fibonacci(payload.to_i)

  exchange.publish(
    result.to_s, routing_key: properties.reply_to, correlation_id: properties.correlation_id
  )
end



# client.rb
call_id = SecureRandom.hex(16)

reply_queue = channel.queue('', exclusive: true)
reply_queue.subscribe do |_delivery, properties, payload|
  if properties[:correlation_id] == call_id
    puts "response: #{payload}"
  end
end

exchange = channel.default_exchange
exchange.publish(
  '30', routing_key: 'fibonacci_server', correlation_id: call_id, reply_to: reply_queue.name
)

Advanced use case

Connections VS Channels

Create Exchange

  • internal if "yes" only used for exchange-to-exchange routing
  • alternate-exchange if messages cannot be routed to this exchange, send them to the alternate one

Create Queue

  • auto-expire is a TTL for the queue if remains unused for a period of time
  • max-length is the max number of ready messages a queue can have before it starts to drop messages from the "head" of the queue (sliding window)

Virtual Hosts

Thank you

¿¿Questions??

Made with Slides.com