Messaging in Elixir with Conduit
Allen Madsen
@blatyo
github.com/blatyo
Cinch Financial
Your totally unbiased, comprehensive, personal CFO.
Conduit is a Framework for Working with Message Queues
What is a message queue?
enqueue
dequeue
name: user.created
async
Delivery Guarantees
- At most once delivery
- Immediately acknowledge messages
- Work may or may not get done
- At least once delivery
- Acknowledge messages after work is done
- Work is guaranteed to happen once or more
- Work needs to be idempotent
- Exactly once delivery
- Complex enough that I'm not going to cover it
Buffer
Backup
Push vs Pull
- Push - Queue sends messages based on quality of service configuration.
- Backpressure is handled by the message queue.
- More efficient. Only does something when there is a message.
- Pull - Consumer polls for messages.
- Backpressure is your job.
- Less efficient. Must check with message queue even where there are no messages.
Decoupling
- Simplified deploys
- Only needs to know how to talk to queue. Not many other apps.
- Can be stopped/started arbitrarily as long as setup for at least once guarantee.
- Choice of language
- Independent scaling
- Resiliancy
- Failure in one system doesn't cascade into others
- System can recover
Why Build Conduit?
Lots of libraries, not much in regards to a scalable architecture.
Why Build Conduit?
Consider a library that allows you to make a connection versus one that gives you a scalable OTP architecture.
Reusable messaging patterns packaged as plugs.
Why Build Conduit?
What is Conduit?
Conduit is a framework for...
Connecting to a Message Queue Through an Adapter
config :my_app, MyApp.Broker,
adapter: ConduitAMQP,
url: "amqp://my_app:secret@my-rabbit-host.com"
config :my_app, MyApp.Broker,
adapter: ConduitSQS,
access_key_id: [{:system, "AWS_ACCESS_KEY_ID"}, :instance_role],
secret_access_key: [{:system, "AWS_SECRET_ACCESS_KEY"}, :instance_role]
And eventually others
config :my_app, MyApp.Stomp,
adapter: ConduitStomp,
host: "localhost",
port: 61613,
login: "guest",
passcode: "guest"
Configuring Queues and Exchanges
defmodule MyApp.Broker do
use Conduit.Broker, otp_app: :my_app
configure do
exchange "my.topic", type: "topic", durable: true
queue "my.queue", from: ["#.created.user"], exchange: "amq.topic", durable: true
end
end
Send Messages with Pipelines
defmodule MyApp.Broker do
use Conduit.Broker, otp_app: :my_app
pipeline :out_tracking do
plug Conduit.Plug.CorrelationId
plug Conduit.Plug.CreatedBy, app: "my_app"
plug Conduit.Plug.CreatedAt
plug Conduit.Plug.LogOutgoing
end
pipeline :serialize do
plug Conduit.Plug.Format, content_type: "application/json"
plug Conduit.Plug.Encode
end
outgoing do
pipe_through [:out_tracking, :serialize]
publish :created_user, exchange: "amq.topic", to: "my_app.created.user"
end
end
import Conduit.Message
message = put_body(%Conduit.Message{}, %{"email" => "bob@gmail.com"})
MyApp.Broker.publish(message, :created_user)
Receive Messages with Pipelines
defmodule MyApp.Broker do
use Conduit.Broker, otp_app: :my_app
pipeline :in_tracking do
plug Conduit.Plug.CorrelationId
plug Conduit.Plug.LogIncoming
end
pipeline :error_handling do
plug Conduit.Plug.AckException
plug Conduit.Plug.DeadLetter, broker: MyApp.Broker, publish_to: :error
plug Conduit.Plug.Retry, attempts: 5
end
pipeline :deserialize do
plug Conduit.Plug.Decode
plug Conduit.Plug.Parse, content_type: "application/json"
end
incoming MyApp do
pipe_through [:in_tracking, :error_handling, :deserialize]
subscribe :user_created, SendWelcomeEmailSubscriber,
from: "my_app.created.user"
end
end
Receive Messages with Pipelines (cont)
defmodule MyApp.SendWelcomeEmailSubscriber do
use Conduit.Subscriber
def process(message, _) do
%{"email" => email} = message.body
# send email
message
end
end
How do I test with Conduit?
Use the Test Adapter
config :my_app, MyApp.Broker,
adapter: Conduit.TestAdapter
Calling a Subscriber
import Conduit.Message
message =
%Conduit.Message{}
|> put_body(%{"email" => "bob@gmail.com"})
|> put_correlation_id("123")
message = MyApp.Broker.receives(:user_created, message)
# assert properties about the message
# assert side effects
Through it's pipelines
import Conduit.Message
message =
%Conduit.Message{}
|> put_body(%{"email" => "bob@gmail.com"})
|> put_correlation_id("123")
message = MyApp.UserCreatedSubscriber.run(message)
# assert properties about the message
# assert side effects
Or directly
Sending a Message
import Conduit.Message
message =
%Conduit.Message{}
|> put_body(%{"email" => "bob@gmail.com"})
|> put_correlation_id("123")
MyApp.Broker.publish(:user_created, message)
# or something that calls publish
assert_message_published message
# assert properties about the message
Whats the Future for Conduit?
Future?
- v1.0.0
- More adapters
- More how to docs
- v2.0.0
- Batch message processing
- More GenStage
Questions?
deck
By blatyo
deck
- 3,270