Title Text




CQRS + EVENT SOURCING IN NODEJS
BY ROMAN




ABOUT ME
Software engineer and team lead at DA-14


CONTENTS
Theory
- CQRS and Event Sourcing
How did we choose it?
- project requirements and conditions
How it supposed to work?
- flow ideas
- technology suitability
- architecture solution
Problems and Solutions
- difficult to update services
- lack of suitable libraries
- missing healthcheck
- missing loger
- growing event store
What would I recommend instead?
- architecture solution
Other
- best practices
- helpful modules
- articles

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING


THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Command Query Responsibility Segregation (CQRS)


THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Command Query Responsibility Segregation (CQRS)



THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Command Query Responsibility Segregation (CQRS)




THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Command Query Responsibility Segregation (CQRS)





resources
product
warehouse
production
factory

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Command Query Responsibility Segregation (CQRS)





resources
product
warehouse
production
factory
Huge Load

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Command Query Responsibility Segregation (CQRS)








balancer

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING

Command Query Responsibility Segregation (CQRS)

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING


Command Query Responsibility Segregation (CQRS)





THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING


Command Query Responsibility Segregation (CQRS)






balancer

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
CQRS vs typical CRUD: benefits
- scalability
- separate interfaces
- separate models
- separate storages
- more clean application parts
- suitable architecture solution for Domain Driven Design (DDD)
- good for big teams

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Event Sourcing



THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Event Sourcing
start
finish
You, good guy
villian
villian
villian

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Event Sourcing




Voldemort
Aid Kit
Hannibal
Palpatine
Aid Kit
Start
Warehouse

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Event Sourcing




Voldemort
Aid Kit
Hannibal
Palpatine
Aid Kit
Start
Warehouse

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Event Sourcing




Voldemort
Aid Kit
Hannibal
Palpatine
Aid Kit
Start
Warehouse

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Event Sourcing




Voldemort
Aid Kit
Hannibal
Palpatine
Aid Kit
Start
Warehouse

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Event Sourcing




Voldemort
Aid Kit
Hannibal
Palpatine
Aid Kit
Start
Warehouse

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Event Sourcing



Voldemort
Aid Kit
Hannibal
Palpatine
Aid Kit

Start
Warehouse
Warehouse
Steps

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Event Sourcing




Voldemort
Aid Kit
Hannibal
Palpatine
Aid Kit
Start
Steps
Voldemort: fight

Warehouse
Warehouse

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Event Sourcing




Voldemort
Aid Kit
Hannibal
Palpatine
Aid Kit
Voldemort: fight
Voldemort: passed

Start
Warehouse
Warehouse
Steps

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Event Sourcing




Voldemort
Aid Kit #1
Hannibal
Palpatine
Aid Kit #2
Start
Voldemort: passed
Aid Kid#1: used

Steps
Voldemort: fight
Warehouse

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Event Sourcing




Voldemort
Aid Kit #1
Hannibal
Palpatine
Aid Kit #2
Start
Hannibal: fight
Aid Kit#1: used

Voldemort: passed
Voldemort: fight
Steps
Warehouse

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Event Sourcing




Voldemort
Aid Kit #1
Hannibal
Palpatine
Aid Kit #2
Start
Hannibal: failed

Aid Kit#1: used
Voldemort: passed
Voldemort: fight
Steps
Hannibal: fight
Warehouse

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Event Sourcing




Voldemort
Aid Kit #1
Hannibal
Palpatine
Aid Kit #2

Start
Warehouse
Hannibal: fight
Aid Kit#1: used
Voldemort: passed
Warehouse
Voldemort: fight
Steps

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Voldemort: passed
Aid Kit#1: used
Hannibal: fight
Hannibal: failed
Event Sourcing
Voldemort: fight
Warehouse
States or events results

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Event Sourcing
Hannibal: fight
shoot 3 times

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Event Sourcing
Hannibal: fight
event: shoot 3 times => event: { amo: { use: 3 } };

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Event Sourcing
Initial snapshot
Current state
LIVES: 0
AMO: 0
POINTS: 0
LIVES: 0
AMO: 0
POINTS: 0

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Event Sourcing
Warehouse
{ amo: { load: 200, use: 0 }, lives: { add: 100, deduct: 0 }, addPoints: 0 }
Initial snapshot
Current state
LIVES: 100
AMO: 200
POINTS: 0
LIVES: 0
AMO: 0
POINTS: 0

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Event Sourcing
Warehouse
Voldemort: fight
{ amo: { load: 200, use: 0 }, lives: { add: 100, deduct: 0 }, addPoints: 0 }
{ amo: { load: 0, use: 3 }, lives: { add: 0, deduct: 0 }, addPoints: 0 }
Current state
LIVES: 100
AMO: 197
POINTS: 0

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Event Sourcing
Voldemort: fight
Voldemort: passed
{ amo: { load: 200, use: 0 }, lives: { add: 100, deduct: 0 }, addPoints: 0 }
{ amo: { load: 0, use: 3 }, lives: { add: 0, deduct: 0 }, addPoints: 0 }
{ amo: { load: 0, use: 10 }, lives: { add: 0, deduct: 25 }, addPoints: 10 }
Warehouse
Current state
LIVES: 75
AMO: 187
POINTS: 10

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Event Sourcing
Voldemort: fight
Voldemort: passed
{ amo: { load: 200, use: 0 }, lives: { add: 100, deduct: 0 }, addPoints: 0 }
{ amo: { load: 0, use: 3 }, lives: { add: 0, deduct: 0 }, addPoints: 0 }
{ amo: { load: 0, use: 10 }, lives: { add: 0, deduct: 25 }, addPoints: 10 }
Warehouse
{ amo: { load: 0, use: 0 }, lives: { add: 15, deduct: 0 }, addPoints: 0 }
Aid Kit#1: used
Current state
LIVES: 90
AMO: 187
POINTS: 10

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Hannibal: fight
Hannibal: failed
Event Sourcing
Voldemort: fight
Voldemort: passed
{ amo: { load: 200, use: 0 }, lives: { add: 100, deduct: 0 }, addPoints: 0 }
{ amo: { load: 0, use: 3 }, lives: { add: 0, deduct: 0 }, addPoints: 0 }
{ amo: { load: 0, use: 10 }, lives: { add: 0, deduct: 25 }, addPoints: 10 }
Warehouse
{ amo: { load: 0, use: 0 }, lives: { add: 15, deduct: 0 }, addPoints: 0 }
Aid Kit#1: used
{ amo: { load: 0, use: 57 }, lives: { add: 0, deduct: 0 }, addPoints: 0 }
{ amo: { load: 0, use: 5 }, lives: { add: 0, deduct: 120 }, addPoints: 0 }
Current state
LIVES: -30
AMO: 125
POINTS: 10

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Event Sourcing
Events
Consequences
Hannibal: fight
Hannibal: failed
Voldemort: fight
Voldemort: passed
Warehouse
Aid Kit#1: used
{ amo: { load: 200, use: 0 }, lives: { add: 100, deduct: 0 }, addPoints: 0 }
{ amo: { load: 0, use: 3 }, lives: { add: 0, deduct: 0 }, addPoints: 0 }
{ amo: { load: 0, use: 10 }, lives: { add: 0, deduct: 25 }, addPoints: 10 }
{ amo: { load: 0, use: 0 }, lives: { add: 15, deduct: 0 }, addPoints: 0 }
{ amo: { load: 0, use: 57 }, lives: { add: 0, deduct: 0 }, addPoints: 0 }
{ amo: { load: 0, use: 5 }, lives: { add: 0, deduct: 120 }, addPoints: 0 }

THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Event Sourcing: benefits
- ability to 'jump' between states (do/undo)
- ability to have everything logged
- more easy to deal with corrupted data
- easy to follow steps and reproduce system state

HOW DID WE CHOOSE THIS?


Conditions

- recommended by a client
HOW DID WE CHOOSE THIS?

Conditions
- we knew nothing about the flow

HOW DID WE CHOOSE THIS?

Conditions
- requested to build a stable load protected system

HOW DID WE CHOOSE THIS?

Conditions
- requested to build an action log tracking system

HOW DID WE CHOOSE THIS?

HOW IT SUPPOSED TO WORK?


Architecture Items
- API Gateway
- Microservices
- Event bus
- DB - Event store
- DB - regular storage
HOW IT SUPPOSED TO WORK?




Client
API
Queries
Commands
MS0
MS1
MSn
Denormalizer

Event Bus
Event store

HOW IT SUPPOSED TO WORK?

-
NodeJS + ExpressJS
- perfect for event-driven model
- streams support in a core level
Stack

HOW IT SUPPOSED TO WORK?

- MongoDB
- schemaless
- awesome for logging =>
fast enough to save events - possible to setup records rotation
- streams support
- watch support

Stack
HOW IT SUPPOSED TO WORK?

- Apache Kafka
- scalable
- durability
- provides with high performance
- streams support
Stack

HOW IT SUPPOSED TO WORK?

-
Kubernetes
- auth-deployment
- auth-launching
- scalability
Stack
HOW IT SUPPOSED TO WORK?

















commands
queries
HOW IT SUPPOSED TO WORK?

PROBLEMS: #0 HARD TO UPDATE FUNCTIONALITY


PROBLEMS: #0 HARD TO UPDATE FUNCTIONALITY
Problem
API gateway
Service (X)
Denormalizer
DB Model (X)
DB Model (X)
DB Model (X)


PROBLEMS: #0 HARD TO UPDATE FUNCTIONALITY
Solution
API gateway
Service (X)
Denormalizer
DB Model (X) Repo


PROBLEMS: #1 COULDN'T FIND SUITABLE LIBRARIES


PROBLEMS: #1 COULDN'T FIND SUITABLE LIBRARIES
Problem
- Couldn't find a suitable Apache Kafka library to cover basic needs with built-in topic auto-creation functionality

PROBLEMS: #1 COULDN'T FIND SUITABLE LIBRARIES
Solution: make it easy - write a script
Service
Deployment script
Build
Kafka topics creation script

PROBLEMS: #2 HEALTHCHECK


PROBLEMS: #2 HEALTHCHECK
Simple schema

DOWN
UP
UP
HTTP API Gateway
Command
Success response

PROBLEMS: #2 HEALTHCHECK
Problem
- Is my service available?
- Service down or event bus down?

PROBLEMS: #2 HEALTHCHECK

DOWN
UP
UP
HTTP API Gateway
Command
Success response
Solution: Simple HTTP request is enough
HTTP
GET /status
HTTP
GET /status
HTTP
GET /status

PROBLEMS: #3 ERROR HANDLING


PROBLEMS: #3 ERROR HANDLING
Problem: there is no way to notify a client about errors due to the event-driven architecture

MS0
ERROR
Command
Success response

HTTP API Gateway
MS1
MSn

PROBLEMS: #3 ERROR HANDLING
Service
ERROR


Notification Service

PROBLEMS: #4 LOGGING


PROBLEMS: #4 LOGGING
Regular Service 0
Regular Service 1
Regular Service n

Loggin Service
HTTP Access


SSH Access

PROBLEMS: #4 LOGGING
Solution
- use separate logging module for each service
- clean logs
- number of log items will grow up along with a number of services
- use shared logs passed via an event bus
- easy to check logs for services in one place
- a strategy to identify log items by service should be created

PROBLEMS: #5 GROWING EVENT LOG


PROBLEMS: #5 GROWING EVENT LOG

Timeline

PROBLEMS: #5 GROWING EVENT LOG


Timeline

PROBLEMS: #5 GROWING EVENT LOG



Timeline

PROBLEMS: #5 GROWING EVENT LOG




Timeline

PROBLEMS: #5 GROWING EVENT LOG
Solution: make snapshots
Event Store
Snapshot Store
event (n)
event (10)
...
event (9)
event (8)
event (6)
event (5)
event (4)
event (3)
event (2)
event (1)
snapshot (1)
snapshot (2)
snapshot (3)
snapshot (n)
...
...
...
...
...
...
...

PROBLEMS: #5 GROWING EVENT LOG
Solution
- store initial object state
- store middle objects/states (snapshots)
- schedule clean-up service
- apply events to an initial object
But
- think about valuable data

WHAT WOULD I RECOMMEND INSTEAD?


WHAT WOULD I RECOMMEND INSTEAD? (IN MY CASE)
- merge services in a monolythic system if possible
- extract potentially loaded services
- use lambda functions not to maintain microservices

WHEN IT'S RECOMMENDED TO USE CQRS + EVENT SOURCING?


WHEN IT'S RECOMMENDED TO USE CQRS + EVENT SOURCING?
- in case of implementing DDD approach
- much of not CRUD-based functionality
- version control is required to perform to/undo operations
- services scalability is required
- an event-driven system is required
- a full transaction log is required

BE AWARE OF


BE AWARE OF
CQRS + Event Sourcing Cons
- complex architecture solution
- much time to jum into
- high store usage (for event store)
- eventual consistency

OFFTOP BUT PAY ATTENTION


OFFTOP BUT PAY ATTENTION
Event Sourcing is all about functional programming
- do/redo/undo operations are actually functional operations
- applying an event produces a state which can be used to produce a new state by applying another event
- immutable data enables functional programming
- events are immutable

BEST PRACTICES


BEST PRACTICES
- think thrice before you use CQRS and Event Sourcing
- write documentation at the beginning
- create healthcheck flow at the beginning
- create logging service at the beginning
- don't store objects, but events
- version your events
- think about event log size

HELPFUL MODULES


HELPFUL MODULES

ARTICLES


ARTICLES
CQRS
- https://martinfowler.com/bliki/CQRS.html
- https://blog.risingstack.com/cqrs-explained-node-js-at-scale/
- https://lostechies.com/jimmybogard/2012/08/23/cqrs-and-user-experience/
Event Sourcing
CQRS + Event Sourcing

QUESTIONS
*and again I couldn't find a funny meme for this page

roman.sachenko@gmail.com
https://www.facebook.com/rsachenko
https://github.com/roman-sachenko
https://twitter.com/RSachenko
roman.sachenko
NodeJS + CQRS + Event Sourcing
By Roman Sachenko
NodeJS + CQRS + Event Sourcing
Why do I do this? I kept asking myself this question while working on the next project and building microservices-based architecture with CQRS and Event Sourcing. What does this system do and why is it so complicated? I'll tell you about mistakes, which I’ve made, issues, which I've faced with, and solutions, which shouldn’t be applied. I’ll share my own experience and tell you about those things, I wish I’d known before I started working on the project.
- 3,432