Title Text

CQRS + EVENT SOURCING IN NODEJS

BY ROMAN SACHENKO

ABOUT ME

Software engineer and team lead at DA-14

CONTENTS

Theory

  • CQRS and Event Sourcing

 

How did we choose it?

  • project requirements and conditions

 

How it supposed to work?

  • flow ideas
  • technology suitability 
  • architecture solution

Problems and Solutions

  • difficult to update services
  • lack of suitable libraries
  • missing healthcheck
  • missing loger
  • growing event store

 

What would I recommend instead?

  • architecture solution

 

Other

  • best practices
  • helpful modules
  • articles

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING

Command Query Responsibility Segregation (CQRS)

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING

Command Query Responsibility Segregation (CQRS)

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING

Command Query Responsibility Segregation (CQRS)

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING

Command Query Responsibility Segregation (CQRS)

resources

product

warehouse

production

factory

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING

Command Query Responsibility Segregation (CQRS)

resources

product

warehouse

production

factory

Huge Load

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING

Command Query Responsibility Segregation (CQRS)

balancer

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING

Command Query Responsibility Segregation (CQRS)

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING

Command Query Responsibility Segregation (CQRS)

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING

Command Query Responsibility Segregation (CQRS)

balancer

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING

CQRS vs typical CRUD: benefits

  • scalability
    • separate interfaces
    • separate models
    • separate storages
  • more clean application parts
  • suitable architecture solution for Domain Driven Design (DDD)
  • good for big teams

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING

Event Sourcing

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING

Event Sourcing

start

finish

You, good guy

villian

villian

villian

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING

Event Sourcing

Voldemort

Aid Kit

Hannibal

Palpatine

Aid Kit

Start

Warehouse

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING

Event Sourcing

Voldemort

Aid Kit

Hannibal

Palpatine

Aid Kit

Start

Warehouse

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING

Event Sourcing

Voldemort

Aid Kit

Hannibal

Palpatine

Aid Kit

Start

Warehouse

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING

Event Sourcing

Voldemort

Aid Kit

Hannibal

Palpatine

Aid Kit

Start

Warehouse

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING

Event Sourcing

Voldemort

Aid Kit

Hannibal

Palpatine

Aid Kit

Start

Warehouse

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING

Event Sourcing

Voldemort

Aid Kit

Hannibal

Palpatine

Aid Kit

Start

Warehouse

Warehouse

Steps

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING

Event Sourcing

Voldemort

Aid Kit

Hannibal

Palpatine

Aid Kit

Start

Steps

Voldemort: fight

Warehouse

Warehouse

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING

Event Sourcing

Voldemort

Aid Kit

Hannibal

Palpatine

Aid Kit

Voldemort: fight

Voldemort: passed

Start

Warehouse

Warehouse

Steps

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING

Event Sourcing

Voldemort

Aid Kit #1

Hannibal

Palpatine

Aid Kit #2

Start

Voldemort: passed

Aid Kid#1: used

Steps

Voldemort: fight

Warehouse

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING

Event Sourcing

Voldemort

Aid Kit #1

Hannibal

Palpatine

Aid Kit #2

Start

Hannibal: fight

Aid Kit#1: used

Voldemort: passed

Voldemort: fight

Steps

Warehouse

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING

Event Sourcing

Voldemort

Aid Kit #1

Hannibal

Palpatine

Aid Kit #2

Start

Hannibal: failed

Aid Kit#1: used

Voldemort: passed

Voldemort: fight

Steps

Hannibal: fight

Warehouse

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING

Event Sourcing

Voldemort

Aid Kit #1

Hannibal

Palpatine

Aid Kit #2

Start

Warehouse

Hannibal: fight

Aid Kit#1: used

Voldemort: passed

Warehouse

Voldemort: fight

Steps

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING

Voldemort: passed

Aid Kit#1: used

Hannibal: fight

Hannibal: failed

Event Sourcing

Voldemort: fight

Warehouse

States or events results

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING

Event Sourcing

Hannibal: fight

shoot 3 times

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING

Event Sourcing

Hannibal: fight

event: shoot 3 times => event: { amo: { use: 3 } };

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING

Event Sourcing

Initial snapshot

Current state

LIVES: 0

AMO: 0

POINTS: 0

LIVES: 0

AMO: 0

POINTS: 0

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING

Event Sourcing

Warehouse

{ amo: { load: 200, use: 0 },  lives: { add: 100, deduct: 0 }, addPoints: 0 }

Initial snapshot

Current state

LIVES: 100

AMO: 200

POINTS: 0

LIVES: 0

AMO: 0

POINTS: 0

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING

Event Sourcing

Warehouse

Voldemort: fight

{ amo: { load: 200, use: 0 },  lives: { add: 100, deduct: 0 }, addPoints: 0 }

{ amo: { load: 0, use: 3 },  lives: { add: 0, deduct: 0 }, addPoints: 0 }

Current state

LIVES: 100

AMO: 197

POINTS: 0

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING

Event Sourcing

Voldemort: fight

Voldemort: passed

{ amo: { load: 200, use: 0 },  lives: { add: 100, deduct: 0 }, addPoints: 0 }

{ amo: { load: 0, use: 3 },  lives: { add: 0, deduct: 0 }, addPoints: 0 }

{ amo: { load: 0, use: 10 },  lives: { add: 0, deduct: 25 }, addPoints: 10 }

Warehouse

Current state

LIVES: 75

AMO: 187

POINTS: 10

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING

Event Sourcing

Voldemort: fight

Voldemort: passed

{ amo: { load: 200, use: 0 },  lives: { add: 100, deduct: 0 }, addPoints: 0 }

{ amo: { load: 0, use: 3 },  lives: { add: 0, deduct: 0 }, addPoints: 0 }

{ amo: { load: 0, use: 10 },  lives: { add: 0, deduct: 25 }, addPoints: 10 }

Warehouse

{ amo: { load: 0, use: 0 },  lives: { add: 15, deduct: 0 }, addPoints: 0 }

Aid Kit#1: used

Current state

LIVES: 90

AMO: 187

POINTS: 10

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING

Hannibal: fight

Hannibal: failed

Event Sourcing

Voldemort: fight

Voldemort: passed

{ amo: { load: 200, use: 0 },  lives: { add: 100, deduct: 0 }, addPoints: 0 }

{ amo: { load: 0, use: 3 },  lives: { add: 0, deduct: 0 }, addPoints: 0 }

{ amo: { load: 0, use: 10 },  lives: { add: 0, deduct: 25 }, addPoints: 10 }

Warehouse

{ amo: { load: 0, use: 0 },  lives: { add: 15, deduct: 0 }, addPoints: 0 }

Aid Kit#1: used

{ amo: { load: 0, use: 57 },  lives: { add: 0, deduct: 0 }, addPoints: 0 }

{ amo: { load: 0, use: 5 },  lives: { add: 0, deduct: 120 }, addPoints: 0 }

Current state

LIVES: -30

AMO: 125

POINTS: 10

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING

Event Sourcing

Events

Consequences

Hannibal: fight

Hannibal: failed

Voldemort: fight

Voldemort: passed

Warehouse

Aid Kit#1: used

{ amo: { load: 200, use: 0 },  lives: { add: 100, deduct: 0 }, addPoints: 0 }

{ amo: { load: 0, use: 3 },  lives: { add: 0, deduct: 0 }, addPoints: 0 }

{ amo: { load: 0, use: 10 },  lives: { add: 0, deduct: 25 }, addPoints: 10 }

{ amo: { load: 0, use: 0 },  lives: { add: 15, deduct: 0 }, addPoints: 0 }

{ amo: { load: 0, use: 57 },  lives: { add: 0, deduct: 0 }, addPoints: 0 }

{ amo: { load: 0, use: 5 },  lives: { add: 0, deduct: 120 }, addPoints: 0 }

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING

Event Sourcing: benefits

  • ability to 'jump' between states (do/undo)
  • ability to have everything logged
  • more easy to deal with corrupted data
  • easy to follow steps and reproduce system state

HOW DID WE CHOOSE THIS?

Conditions

  • recommended by a client

HOW DID WE CHOOSE THIS?

Conditions

  • we knew nothing about the flow

HOW DID WE CHOOSE THIS?

Conditions

  • requested to build a stable load protected system

HOW DID WE CHOOSE THIS?

Conditions

  • requested to build an action log tracking system

HOW DID WE CHOOSE THIS?

HOW IT SUPPOSED TO WORK?

Architecture Items

  • API Gateway
  • Microservices
  • Event bus
  • DB - Event store
  • DB - regular storage

HOW IT SUPPOSED TO WORK?

Client

API

Queries

Commands

MS0

MS1

MSn

Denormalizer

Event Bus

Event store

HOW IT SUPPOSED TO WORK?

  • NodeJS + ExpressJS
    • perfect for event-driven model
    • streams support in a core level

Stack

HOW IT SUPPOSED TO WORK?

  • MongoDB
    • schemaless
    • awesome for logging =>
      fast enough to save events
    • possible to setup records rotation
    • streams support
    • watch support

Stack

HOW IT SUPPOSED TO WORK?

  • Apache Kafka
    • scalable
    • durability
    • provides with high performance
    • streams support

Stack

HOW IT SUPPOSED TO WORK?

  • Kubernetes
    • auth-deployment
    • auth-launching
    • scalability

Stack

HOW IT SUPPOSED TO WORK?

commands

queries

HOW IT SUPPOSED TO WORK?

PROBLEMS: #0 HARD TO UPDATE FUNCTIONALITY

PROBLEMS: #0 HARD TO UPDATE FUNCTIONALITY

Problem

API gateway

Service (X)

Denormalizer

DB Model (X)

DB Model (X)

DB Model (X)

PROBLEMS: #0 HARD TO UPDATE FUNCTIONALITY

Solution

API gateway

Service (X)

Denormalizer

DB Model (X) Repo

PROBLEMS: #1 COULDN'T FIND SUITABLE LIBRARIES

PROBLEMS: #1 COULDN'T FIND SUITABLE LIBRARIES

Problem

  • Couldn't find a suitable Apache Kafka library to cover basic needs with built-in topic auto-creation functionality

PROBLEMS: #1 COULDN'T FIND SUITABLE LIBRARIES

Solution: make it easy - write a script

Service

Deployment script

Build

Kafka topics creation script

PROBLEMS: #2 HEALTHCHECK

PROBLEMS: #2 HEALTHCHECK

Simple schema

DOWN

UP

UP

HTTP API Gateway

Command

Success response

PROBLEMS: #2 HEALTHCHECK

Problem

  • Is my service available?
  • Service down or event bus down?

 

PROBLEMS: #2 HEALTHCHECK

DOWN

UP

UP

HTTP API Gateway

Command

Success response

Solution: Simple HTTP request is enough

HTTP

GET /status

HTTP

GET /status

HTTP

GET /status

PROBLEMS: #3 ERROR HANDLING

PROBLEMS: #3 ERROR HANDLING

Problem: there is no way to notify a client about errors due to the event-driven architecture

MS0

ERROR

Command

Success response

HTTP API Gateway

MS1

MSn

PROBLEMS: #3 ERROR HANDLING

Service

ERROR

Notification Service

PROBLEMS: #4 LOGGING

PROBLEMS: #4 LOGGING

Regular Service 0

Regular Service 1

Regular Service n

Loggin Service

HTTP Access

SSH Access

PROBLEMS: #4 LOGGING

Solution

  • use separate logging module for each service
    • clean logs
    • number of log items will grow up along with a number of services 
  • use shared logs passed via an event bus
    • easy to check logs for services in one place
    • a strategy to identify log items by service should be created

PROBLEMS: #5 GROWING EVENT LOG

PROBLEMS: #5 GROWING EVENT LOG

Timeline

PROBLEMS: #5 GROWING EVENT LOG

Timeline

PROBLEMS: #5 GROWING EVENT LOG

Timeline

PROBLEMS: #5 GROWING EVENT LOG

Timeline

PROBLEMS: #5 GROWING EVENT LOG

Solution: make snapshots

Event Store

Snapshot Store

event (n)

event (10)

...

event (9)

event (8)

event (6)

event (5)

event (4)

event (3)

event (2)

event (1)

snapshot (1)

snapshot (2)

snapshot (3)

snapshot (n)

...

...

...

...

...

...

...

PROBLEMS: #5 GROWING EVENT LOG

Solution

  • store initial object state
  • store middle objects/states (snapshots)
  • schedule clean-up service
  • apply events to an initial object

But

  • think about valuable data

WHAT WOULD I RECOMMEND INSTEAD?

WHAT WOULD I RECOMMEND INSTEAD? (IN MY CASE)

  • merge services in a monolythic system if possible
  • extract potentially loaded services 
  • use lambda functions not to maintain microservices

WHEN IT'S RECOMMENDED TO USE CQRS + EVENT SOURCING?

WHEN IT'S RECOMMENDED TO USE CQRS + EVENT SOURCING?

  • in case of implementing DDD approach
  • much of not CRUD-based functionality
  • version control is required to perform to/undo operations
  • services scalability is required
  • an event-driven system is required
  • a full transaction log is required

BE AWARE OF

BE AWARE OF

CQRS + Event Sourcing Cons

  • complex architecture solution
  • much time to jum into
  • high store usage (for event store)
  • eventual consistency

OFFTOP BUT PAY ATTENTION

OFFTOP BUT PAY ATTENTION

Event Sourcing is all about functional programming

  • do/redo/undo operations are actually functional operations
    • applying an event produces a state which can be used to produce a new state by applying another event
  • immutable data enables functional programming
    • events are immutable

BEST PRACTICES

BEST PRACTICES

  • think thrice before you use CQRS and Event Sourcing
  • write documentation at the beginning
  • create healthcheck flow at the beginning
  • create logging service at the beginning
  • don't store objects, but events
  • version your events
  • think about event log size

HELPFUL MODULES

HELPFUL MODULES

ARTICLES

ARTICLES

CQRS

 

Event Sourcing

 

CQRS + Event Sourcing

QUESTIONS

*and again I couldn't find a funny meme for this page

roman.sachenko@gmail.com

https://www.facebook.com/rsachenko

https://github.com/roman-sachenko

https://twitter.com/RSachenko

roman.sachenko

Made with Slides.com