Event Driven Architecture

We will understand...

  • The need of Event Driven Architecture
  • What exactly is Event Driven Approach
  • Sagas Pattern
  • Transactional Outbox Pattern
  • Libraries: Debezium & Eventuate CDC
  • Implementation with Debezium
  • Event Sourcing from a high-level perspective

MICROSERVICES & Data Consistency

  • Decoupled microservices have their own databases
  • Microservices cannot directly access/change other service's data
  • Some transactions span multiple microservices
  • Data needs to be consistent across services

CAP THEOREM

In a distributed data store, you can get only 2 out of 3 guarantees:

  • Consistency
  • Availability
  • Partition Tolerance

API COMPOSITION

API composer invokes multiple services and performs in-memory joins for the final query

PROS

  • Simple way to query data in a microservice architecture

CONS

  • Runtime coupling
  • Low system availability
  • Inefficient In-memory joins and aggregation of large data.

EVENT DRIVEN APPROACH

  • Service fires domain events for its local transactions
  • Concerned Services hook into events and perform the desired operation and may fire other events.
  • Uses a messaging protocol to fire events

Sagas

(A long series of incidents happened over a period of time)

  • Series of local atomic transactions on each involved component
  • After a local transaction, another transaction is triggered through an event.
  • If any local transaction fails, it triggers a compensating transactions that rollback all the changes done previously.

Sagas - ISSUES

  • Reliably publishing events atomically (outbox pattern solves this)
  • Asynchronous flow initiated by client, using synchronous HTTP request, needs to determine the outcome
    • Wait until saga is completed, and respond with status
    • Respond after initiating saga, client will poll to check status of saga.
    • Respond after initiating saga, and send event to the client through web socket once the sage is completed

PROS

Data consistency across multiple services without using distributed transactions

CONS

Programming model is more complex that requires properly designed compensating transactions that explicitly undo changes made earlier in a saga.

SAGA COORDINATION

  • Choreography
  • Orchestration

CHOREOGRAPHY BASED SAGA

  • No centralized point of control
  • Each local transaction publishes events that trigger a local transaction in other services

PROS

  • Good for simple workflows that require few participants and don't need a coordination logic.
  • Doesn't require additional service
  • Doesn't introduce a single point of failure

CONS

  • Workflow can become confusing when adding new steps.
  • There's a risk of cyclic dependency between saga participants
  • Integration testing is difficult.

Choreography Based Sagas

ORCHESTRATION BASED SAGA

  • A centralized controller tells the participants what local transactions to execute.
  • requests, stores and interprets the states of each task, and handles failure recovery with compensating transactions

PROS

  • Good for complex workflows involving many participants.
  • Control over every participant, and the flow of activities.
  • Doesn't introduce cyclic dependencies.
  • Clear separation of concerns simplifies business logic.

CONS

  • Requires an implementation of a coordination logic.
  • Introduces additional point of failure, i.e. orchestrator

Orchestration Based Sagas

Transactional Outbox Pattern

MEssage RElay Patterns

  • Transaction Log Tailing
  • Polling Publisher

EVENTUATE CDC

PROS

Designed specially for outbox pattern, provides abstraction libraries along with CDC Service.

CONS

Not so flexible, and customizable configuration

Not everything is well documented

Relatively new, no major version released

DEBEZIUM OUTBOX ROUTER

PROS

Flexibility over columns and tables nomenclature

Well documented configuration, stable and reliable.

CONS

No client integration libraries for abstraction, available for Quarkus framework only.

 

COMPONENTS: OUTBOX IMPLEMENTATION

  • Kafka Connect
  • MySQL (binlogs enabled, ROW format)
  • Debezium MySQL Connector
  • Debezium Outbox event router Transform

THE Outbox Table

  • id: contains the unique ID of the event. It is passed as header, can be used to remove duplicate messages.
  • aggregate_type: used for routing message to corresponding Kafka topic.
  • aggregate_id: The event key, which provides an ID for the payload.

    This is important for maintaining correct order in Kafka partitions.

  • payload: A representation of the outbox change event.
  • created_at: Timestamp for event, used as Kafka message timestamp, INDEXED for housekeeping queries.

TRANSACTIONAL OUTBOX ISSUES

  • Guarantees at-least once delivery, a message can get published more than once. As a result, a message consumer must be idempotent.

Event Sourcing

  • Store immutable events in an event store, instead of maintaining mutable application state in databases.
  • Application state is derived by consuming the series of events.

PROS

  • makes it possible to reliably publish events whenever state changes.
  • provides 100% reliable audit log
  • potential for temporal queries that determine the state at any point in time

CONS

  • Different and unfamiliar style of programming and so there is a learning curve
  • Difficult to query data, needs to implement another pattern CQRS

REferences

Made with Slides.com