Messaging in Elixir with Conduit

Allen Madsen

@blatyo

github.com/blatyo

Cinch Financial

Your totally unbiased, comprehensive, personal CFO.

Conduit is a Framework for Working with Message Queues

What is a message queue?

enqueue

dequeue

name: user.created

async

Delivery Guarantees

  • At most once delivery
    • Immediately acknowledge messages
    • Work may or may not get done
  • At least once delivery
    • Acknowledge messages after work is done
    • Work is guaranteed to happen once or more
    • Work needs to be idempotent
  • Exactly once delivery
    • Complex enough that I'm not going to cover it

Buffer

Backup

Push vs Pull

  • Push - Queue sends messages based on quality of service configuration.
    • Backpressure is handled by the message queue.
    • More efficient. Only does something when there is a message.
  • Pull - Consumer polls for messages.
    • Backpressure is your job.
    • Less efficient. Must check with message queue even where there are no messages.

Decoupling

  • Simplified deploys
    • Only needs to know how to talk to queue. Not many other apps.
    • Can be stopped/started arbitrarily as long as setup for at least once guarantee.
  • Choice of language
  • Independent scaling
  • Resiliancy
    • Failure in one system doesn't cascade into others
    • System can recover

Why Build Conduit?

Lots of libraries, not much in regards to a scalable architecture.

Why Build Conduit?

Consider a library that allows you to make a connection versus one that gives you a scalable OTP architecture.

Reusable messaging patterns packaged as plugs.

Why Build Conduit?

What is Conduit?

Conduit is a framework for...

Connecting to a Message Queue Through an Adapter

config :my_app, MyApp.Broker,
  adapter: ConduitAMQP,
  url: "amqp://my_app:secret@my-rabbit-host.com"
config :my_app, MyApp.Broker,
  adapter: ConduitSQS,
  access_key_id: [{:system, "AWS_ACCESS_KEY_ID"}, :instance_role],
  secret_access_key: [{:system, "AWS_SECRET_ACCESS_KEY"}, :instance_role]

And eventually others

config :my_app, MyApp.Stomp,
  adapter: ConduitStomp,
  host: "localhost", 
  port: 61613, 
  login: "guest", 
  passcode: "guest"

Configuring Queues and Exchanges

defmodule MyApp.Broker do
  use Conduit.Broker, otp_app: :my_app

  configure do
    exchange "my.topic", type: "topic", durable: true
    
    queue "my.queue", from: ["#.created.user"], exchange: "amq.topic", durable: true
  end
end

Send Messages with Pipelines

defmodule MyApp.Broker do
  use Conduit.Broker, otp_app: :my_app

  pipeline :out_tracking do
    plug Conduit.Plug.CorrelationId
    plug Conduit.Plug.CreatedBy, app: "my_app"
    plug Conduit.Plug.CreatedAt
    plug Conduit.Plug.LogOutgoing
  end

  pipeline :serialize do
    plug Conduit.Plug.Format, content_type: "application/json"
    plug Conduit.Plug.Encode
  end

  outgoing do
    pipe_through [:out_tracking, :serialize]

    publish :created_user, exchange: "amq.topic", to: "my_app.created.user"
  end
end
import Conduit.Message

message = put_body(%Conduit.Message{}, %{"email" => "bob@gmail.com"})

MyApp.Broker.publish(message, :created_user)

Receive Messages with Pipelines

defmodule MyApp.Broker do
  use Conduit.Broker, otp_app: :my_app

  pipeline :in_tracking do
    plug Conduit.Plug.CorrelationId
    plug Conduit.Plug.LogIncoming
  end

  pipeline :error_handling do
    plug Conduit.Plug.AckException
    plug Conduit.Plug.DeadLetter, broker: MyApp.Broker, publish_to: :error
    plug Conduit.Plug.Retry, attempts: 5
  end

  pipeline :deserialize do
    plug Conduit.Plug.Decode
    plug Conduit.Plug.Parse, content_type: "application/json"
  end

  incoming MyApp do
    pipe_through [:in_tracking, :error_handling, :deserialize]

    subscribe :user_created, SendWelcomeEmailSubscriber, 
      from: "my_app.created.user"
  end
end

Receive Messages with Pipelines (cont)

defmodule MyApp.SendWelcomeEmailSubscriber do
  use Conduit.Subscriber

  def process(message, _) do
    %{"email" => email} = message.body
    
    # send email

    message
  end
end

How do I test with Conduit?

Use the Test Adapter

config :my_app, MyApp.Broker,
  adapter: Conduit.TestAdapter

Calling a Subscriber

import Conduit.Message

message =
  %Conduit.Message{}
  |> put_body(%{"email" => "bob@gmail.com"})
  |> put_correlation_id("123")

message = MyApp.Broker.receives(:user_created, message)

# assert properties about the message
# assert side effects

Through it's pipelines

import Conduit.Message

message =
  %Conduit.Message{}
  |> put_body(%{"email" => "bob@gmail.com"})
  |> put_correlation_id("123")

message = MyApp.UserCreatedSubscriber.run(message)

# assert properties about the message
# assert side effects

Or directly

Sending a Message

import Conduit.Message

message =
  %Conduit.Message{}
  |> put_body(%{"email" => "bob@gmail.com"})
  |> put_correlation_id("123")

MyApp.Broker.publish(:user_created, message)
# or something that calls publish

assert_message_published message

# assert properties about the message

Whats the Future for Conduit?

Future?

  • v1.0.0
    • More adapters
    • More how to docs
  • v2.0.0
    • Batch message processing
    • More GenStage

Questions?

Made with Slides.com