Event Driven Architecture
We will understand...
- The need of Event Driven Architecture
- What exactly is Event Driven Approach
- Sagas Pattern
- Transactional Outbox Pattern
- Libraries: Debezium & Eventuate CDC
- Implementation with Debezium
- Event Sourcing from a high-level perspective
MICROSERVICES & Data Consistency
- Decoupled microservices have their own databases
- Microservices cannot directly access/change other service's data
- Some transactions span multiple microservices
- Data needs to be consistent across services
CAP THEOREM
In a distributed data store, you can get only 2 out of 3 guarantees:
- Consistency
- Availability
- Partition Tolerance
API COMPOSITION
API composer invokes multiple services and performs in-memory joins for the final query
PROS
- Simple way to query data in a microservice architecture
CONS
- Runtime coupling
- Low system availability
- Inefficient In-memory joins and aggregation of large data.
EVENT DRIVEN APPROACH
- Service fires domain events for its local transactions
- Concerned Services hook into events and perform the desired operation and may fire other events.
- Uses a messaging protocol to fire events
Sagas
(A long series of incidents happened over a period of time)
- Series of local atomic transactions on each involved component
- After a local transaction, another transaction is triggered through an event.
- If any local transaction fails, it triggers a compensating transactions that rollback all the changes done previously.
Sagas - ISSUES
- Reliably publishing events atomically (outbox pattern solves this)
- Asynchronous flow initiated by client, using synchronous HTTP request, needs to determine the outcome
- Wait until saga is completed, and respond with status
- Respond after initiating saga, client will poll to check status of saga.
- Respond after initiating saga, and send event to the client through web socket once the sage is completed
PROS
Data consistency across multiple services without using distributed transactions
CONS
Programming model is more complex that requires properly designed compensating transactions that explicitly undo changes made earlier in a saga.
SAGA COORDINATION
- Choreography
- Orchestration
CHOREOGRAPHY BASED SAGA
- No centralized point of control
- Each local transaction publishes events that trigger a local transaction in other services
PROS
- Good for simple workflows that require few participants and don't need a coordination logic.
- Doesn't require additional service
- Doesn't introduce a single point of failure
CONS
- Workflow can become confusing when adding new steps.
- There's a risk of cyclic dependency between saga participants
- Integration testing is difficult.
Choreography Based Sagas
ORCHESTRATION BASED SAGA
- A centralized controller tells the participants what local transactions to execute.
- requests, stores and interprets the states of each task, and handles failure recovery with compensating transactions
PROS
- Good for complex workflows involving many participants.
- Control over every participant, and the flow of activities.
- Doesn't introduce cyclic dependencies.
- Clear separation of concerns simplifies business logic.
CONS
- Requires an implementation of a coordination logic.
- Introduces additional point of failure, i.e. orchestrator
Orchestration Based Sagas
Transactional Outbox Pattern
MEssage RElay Patterns
- Transaction Log Tailing
- Polling Publisher
EVENTUATE CDC
PROS
Designed specially for outbox pattern, provides abstraction libraries along with CDC Service.
CONS
Not so flexible, and customizable configuration
Not everything is well documented
Relatively new, no major version released
DEBEZIUM OUTBOX ROUTER
PROS
Flexibility over columns and tables nomenclature
Well documented configuration, stable and reliable.
CONS
No client integration libraries for abstraction, available for Quarkus framework only.
COMPONENTS: OUTBOX IMPLEMENTATION
- Kafka Connect
- MySQL (binlogs enabled, ROW format)
- Debezium MySQL Connector
- Debezium Outbox event router Transform
THE Outbox Table
- id: contains the unique ID of the event. It is passed as header, can be used to remove duplicate messages.
- aggregate_type: used for routing message to corresponding Kafka topic.
-
aggregate_id: The event key, which provides an ID for the payload.
This is important for maintaining correct order in Kafka partitions.
- payload: A representation of the outbox change event.
- created_at: Timestamp for event, used as Kafka message timestamp, INDEXED for housekeeping queries.
TRANSACTIONAL OUTBOX ISSUES
- Guarantees at-least once delivery, a message can get published more than once. As a result, a message consumer must be idempotent.
Event Sourcing
- Store immutable events in an event store, instead of maintaining mutable application state in databases.
- Application state is derived by consuming the series of events.
PROS
- makes it possible to reliably publish events whenever state changes.
- provides 100% reliable audit log
- potential for temporal queries that determine the state at any point in time
CONS
- Different and unfamiliar style of programming and so there is a learning curve
- Difficult to query data, needs to implement another pattern CQRS
REferences
- BASE & ACID: https://queue.acm.org/detail.cfm?id=1394128
- EDA: https://docs.microsoft.com/en-us/azure/architecture/guide/architecture-styles/event-driven
- Saga Pattern: https://docs.microsoft.com/en-us/azure/architecture/reference-architectures/saga/saga
- Debezium Outbox Router: https://debezium.io/documentation/reference/1.9/transformations/outbox-event-router.html#basic-outbox-table
- Eventuate Tram CDC: https://eventuate.io/docs/manual/eventuate-tram/latest/cdc-configuration.html#_id_generation
- Event Sourcing: https://docs.microsoft.com/en-us/azure/architecture/patterns/event-sourcing
Event Driven Architecture
By Ruman Saleem
