Title Text
CQRS + EVENT SOURCING IN NODEJS
BY ROMAN
ABOUT ME
Software engineer and team lead at DA-14
CONTENTS
Theory
- CQRS and Event Sourcing
How did we choose it?
- project requirements and conditions
How it supposed to work?
- flow ideas
- technology suitability
- architecture solution
Problems and Solutions
- difficult to update services
- lack of suitable libraries
- missing healthcheck
- missing loger
- growing event store
What would I recommend instead?
- architecture solution
Other
- best practices
- helpful modules
- articles
THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Command Query Responsibility Segregation (CQRS)
THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Command Query Responsibility Segregation (CQRS)
THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Command Query Responsibility Segregation (CQRS)
THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Command Query Responsibility Segregation (CQRS)
resources
product
warehouse
production
factory
THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Command Query Responsibility Segregation (CQRS)
resources
product
warehouse
production
factory
Huge Load
THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Command Query Responsibility Segregation (CQRS)
balancer
THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Command Query Responsibility Segregation (CQRS)
THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Command Query Responsibility Segregation (CQRS)
THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Command Query Responsibility Segregation (CQRS)
balancer
THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
CQRS vs typical CRUD: benefits
- scalability
- separate interfaces
- separate models
- separate storages
- more clean application parts
- suitable architecture solution for Domain Driven Design (DDD)
- good for big teams
THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Event Sourcing
THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Event Sourcing
start
finish
You, good guy
villian
villian
villian
THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Event Sourcing
Voldemort
Aid Kit
Hannibal
Palpatine
Aid Kit
Start
Warehouse
THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Event Sourcing
Voldemort
Aid Kit
Hannibal
Palpatine
Aid Kit
Start
Warehouse
THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Event Sourcing
Voldemort
Aid Kit
Hannibal
Palpatine
Aid Kit
Start
Warehouse
THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Event Sourcing
Voldemort
Aid Kit
Hannibal
Palpatine
Aid Kit
Start
Warehouse
THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Event Sourcing
Voldemort
Aid Kit
Hannibal
Palpatine
Aid Kit
Start
Warehouse
THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Event Sourcing
Voldemort
Aid Kit
Hannibal
Palpatine
Aid Kit
Start
Warehouse
Warehouse
Steps
THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Event Sourcing
Voldemort
Aid Kit
Hannibal
Palpatine
Aid Kit
Start
Steps
Voldemort: fight
Warehouse
Warehouse
THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Event Sourcing
Voldemort
Aid Kit
Hannibal
Palpatine
Aid Kit
Voldemort: fight
Voldemort: passed
Start
Warehouse
Warehouse
Steps
THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Event Sourcing
Voldemort
Aid Kit #1
Hannibal
Palpatine
Aid Kit #2
Start
Voldemort: passed
Aid Kid#1: used
Steps
Voldemort: fight
Warehouse
THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Event Sourcing
Voldemort
Aid Kit #1
Hannibal
Palpatine
Aid Kit #2
Start
Hannibal: fight
Aid Kit#1: used
Voldemort: passed
Voldemort: fight
Steps
Warehouse
THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Event Sourcing
Voldemort
Aid Kit #1
Hannibal
Palpatine
Aid Kit #2
Start
Hannibal: failed
Aid Kit#1: used
Voldemort: passed
Voldemort: fight
Steps
Hannibal: fight
Warehouse
THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Event Sourcing
Voldemort
Aid Kit #1
Hannibal
Palpatine
Aid Kit #2
Start
Warehouse
Hannibal: fight
Aid Kit#1: used
Voldemort: passed
Warehouse
Voldemort: fight
Steps
THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Voldemort: passed
Aid Kit#1: used
Hannibal: fight
Hannibal: failed
Event Sourcing
Voldemort: fight
Warehouse
States or events results
THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Event Sourcing
Hannibal: fight
shoot 3 times
THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Event Sourcing
Hannibal: fight
event: shoot 3 times => event: { amo: { use: 3 } };
THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Event Sourcing
Initial snapshot
Current state
LIVES: 0
AMO: 0
POINTS: 0
LIVES: 0
AMO: 0
POINTS: 0
THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Event Sourcing
Warehouse
{ amo: { load: 200, use: 0 }, lives: { add: 100, deduct: 0 }, addPoints: 0 }
Initial snapshot
Current state
LIVES: 100
AMO: 200
POINTS: 0
LIVES: 0
AMO: 0
POINTS: 0
THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Event Sourcing
Warehouse
Voldemort: fight
{ amo: { load: 200, use: 0 }, lives: { add: 100, deduct: 0 }, addPoints: 0 }
{ amo: { load: 0, use: 3 }, lives: { add: 0, deduct: 0 }, addPoints: 0 }
Current state
LIVES: 100
AMO: 197
POINTS: 0
THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Event Sourcing
Voldemort: fight
Voldemort: passed
{ amo: { load: 200, use: 0 }, lives: { add: 100, deduct: 0 }, addPoints: 0 }
{ amo: { load: 0, use: 3 }, lives: { add: 0, deduct: 0 }, addPoints: 0 }
{ amo: { load: 0, use: 10 }, lives: { add: 0, deduct: 25 }, addPoints: 10 }
Warehouse
Current state
LIVES: 75
AMO: 187
POINTS: 10
THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Event Sourcing
Voldemort: fight
Voldemort: passed
{ amo: { load: 200, use: 0 }, lives: { add: 100, deduct: 0 }, addPoints: 0 }
{ amo: { load: 0, use: 3 }, lives: { add: 0, deduct: 0 }, addPoints: 0 }
{ amo: { load: 0, use: 10 }, lives: { add: 0, deduct: 25 }, addPoints: 10 }
Warehouse
{ amo: { load: 0, use: 0 }, lives: { add: 15, deduct: 0 }, addPoints: 0 }
Aid Kit#1: used
Current state
LIVES: 90
AMO: 187
POINTS: 10
THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Hannibal: fight
Hannibal: failed
Event Sourcing
Voldemort: fight
Voldemort: passed
{ amo: { load: 200, use: 0 }, lives: { add: 100, deduct: 0 }, addPoints: 0 }
{ amo: { load: 0, use: 3 }, lives: { add: 0, deduct: 0 }, addPoints: 0 }
{ amo: { load: 0, use: 10 }, lives: { add: 0, deduct: 25 }, addPoints: 10 }
Warehouse
{ amo: { load: 0, use: 0 }, lives: { add: 15, deduct: 0 }, addPoints: 0 }
Aid Kit#1: used
{ amo: { load: 0, use: 57 }, lives: { add: 0, deduct: 0 }, addPoints: 0 }
{ amo: { load: 0, use: 5 }, lives: { add: 0, deduct: 120 }, addPoints: 0 }
Current state
LIVES: -30
AMO: 125
POINTS: 10
THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Event Sourcing
Events
Consequences
Hannibal: fight
Hannibal: failed
Voldemort: fight
Voldemort: passed
Warehouse
Aid Kit#1: used
{ amo: { load: 200, use: 0 }, lives: { add: 100, deduct: 0 }, addPoints: 0 }
{ amo: { load: 0, use: 3 }, lives: { add: 0, deduct: 0 }, addPoints: 0 }
{ amo: { load: 0, use: 10 }, lives: { add: 0, deduct: 25 }, addPoints: 10 }
{ amo: { load: 0, use: 0 }, lives: { add: 15, deduct: 0 }, addPoints: 0 }
{ amo: { load: 0, use: 57 }, lives: { add: 0, deduct: 0 }, addPoints: 0 }
{ amo: { load: 0, use: 5 }, lives: { add: 0, deduct: 120 }, addPoints: 0 }
THEORY FOR BEGINNERS: CQRS AND EVENT SOURCING
Event Sourcing: benefits
- ability to 'jump' between states (do/undo)
- ability to have everything logged
- more easy to deal with corrupted data
- easy to follow steps and reproduce system state
HOW DID WE CHOOSE THIS?
Conditions
- recommended by a client
HOW DID WE CHOOSE THIS?
Conditions
- we knew nothing about the flow
HOW DID WE CHOOSE THIS?
Conditions
- requested to build a stable load protected system
HOW DID WE CHOOSE THIS?
Conditions
- requested to build an action log tracking system
HOW DID WE CHOOSE THIS?
HOW IT SUPPOSED TO WORK?
Architecture Items
- API Gateway
- Microservices
- Event bus
- DB - Event store
- DB - regular storage
HOW IT SUPPOSED TO WORK?
Client
API
Queries
Commands
MS0
MS1
MSn
Denormalizer
Event Bus
Event store
HOW IT SUPPOSED TO WORK?
-
NodeJS + ExpressJS
- perfect for event-driven model
- streams support in a core level
Stack
HOW IT SUPPOSED TO WORK?
- MongoDB
- schemaless
- awesome for logging =>
fast enough to save events - possible to setup records rotation
- streams support
- watch support
Stack
HOW IT SUPPOSED TO WORK?
- Apache Kafka
- scalable
- durability
- provides with high performance
- streams support
Stack
HOW IT SUPPOSED TO WORK?
-
Kubernetes
- auth-deployment
- auth-launching
- scalability
Stack
HOW IT SUPPOSED TO WORK?
commands
queries
HOW IT SUPPOSED TO WORK?
PROBLEMS: #0 HARD TO UPDATE FUNCTIONALITY
PROBLEMS: #0 HARD TO UPDATE FUNCTIONALITY
Problem
API gateway
Service (X)
Denormalizer
DB Model (X)
DB Model (X)
DB Model (X)
PROBLEMS: #0 HARD TO UPDATE FUNCTIONALITY
Solution
API gateway
Service (X)
Denormalizer
DB Model (X) Repo
PROBLEMS: #1 COULDN'T FIND SUITABLE LIBRARIES
PROBLEMS: #1 COULDN'T FIND SUITABLE LIBRARIES
Problem
- Couldn't find a suitable Apache Kafka library to cover basic needs with built-in topic auto-creation functionality
PROBLEMS: #1 COULDN'T FIND SUITABLE LIBRARIES
Solution: make it easy - write a script
Service
Deployment script
Build
Kafka topics creation script
PROBLEMS: #2 HEALTHCHECK
PROBLEMS: #2 HEALTHCHECK
Simple schema
DOWN
UP
UP
HTTP API Gateway
Command
Success response
PROBLEMS: #2 HEALTHCHECK
Problem
- Is my service available?
- Service down or event bus down?
PROBLEMS: #2 HEALTHCHECK
DOWN
UP
UP
HTTP API Gateway
Command
Success response
Solution: Simple HTTP request is enough
HTTP
GET /status
HTTP
GET /status
HTTP
GET /status
PROBLEMS: #3 ERROR HANDLING
PROBLEMS: #3 ERROR HANDLING
Problem: there is no way to notify a client about errors due to the event-driven architecture
MS0
ERROR
Command
Success response
HTTP API Gateway
MS1
MSn
PROBLEMS: #3 ERROR HANDLING
Service
ERROR
Notification Service
PROBLEMS: #4 LOGGING
PROBLEMS: #4 LOGGING
Regular Service 0
Regular Service 1
Regular Service n
Loggin Service
HTTP Access
SSH Access
PROBLEMS: #4 LOGGING
Solution
- use separate logging module for each service
- clean logs
- number of log items will grow up along with a number of services
- use shared logs passed via an event bus
- easy to check logs for services in one place
- a strategy to identify log items by service should be created
PROBLEMS: #5 GROWING EVENT LOG
PROBLEMS: #5 GROWING EVENT LOG
Timeline
PROBLEMS: #5 GROWING EVENT LOG
Timeline
PROBLEMS: #5 GROWING EVENT LOG
Timeline
PROBLEMS: #5 GROWING EVENT LOG
Timeline
PROBLEMS: #5 GROWING EVENT LOG
Solution: make snapshots
Event Store
Snapshot Store
event (n)
event (10)
...
event (9)
event (8)
event (6)
event (5)
event (4)
event (3)
event (2)
event (1)
snapshot (1)
snapshot (2)
snapshot (3)
snapshot (n)
...
...
...
...
...
...
...
PROBLEMS: #5 GROWING EVENT LOG
Solution
- store initial object state
- store middle objects/states (snapshots)
- schedule clean-up service
- apply events to an initial object
But
- think about valuable data
WHAT WOULD I RECOMMEND INSTEAD?
WHAT WOULD I RECOMMEND INSTEAD? (IN MY CASE)
- merge services in a monolythic system if possible
- extract potentially loaded services
- use lambda functions not to maintain microservices
WHEN IT'S RECOMMENDED TO USE CQRS + EVENT SOURCING?
WHEN IT'S RECOMMENDED TO USE CQRS + EVENT SOURCING?
- in case of implementing DDD approach
- much of not CRUD-based functionality
- version control is required to perform to/undo operations
- services scalability is required
- an event-driven system is required
- a full transaction log is required
BE AWARE OF
BE AWARE OF
CQRS + Event Sourcing Cons
- complex architecture solution
- much time to jum into
- high store usage (for event store)
- eventual consistency
OFFTOP BUT PAY ATTENTION
OFFTOP BUT PAY ATTENTION
Event Sourcing is all about functional programming
- do/redo/undo operations are actually functional operations
- applying an event produces a state which can be used to produce a new state by applying another event
- immutable data enables functional programming
- events are immutable
BEST PRACTICES
BEST PRACTICES
- think thrice before you use CQRS and Event Sourcing
- write documentation at the beginning
- create healthcheck flow at the beginning
- create logging service at the beginning
- don't store objects, but events
- version your events
- think about event log size
HELPFUL MODULES
HELPFUL MODULES
ARTICLES
ARTICLES
CQRS
- https://martinfowler.com/bliki/CQRS.html
- https://blog.risingstack.com/cqrs-explained-node-js-at-scale/
- https://lostechies.com/jimmybogard/2012/08/23/cqrs-and-user-experience/
Event Sourcing
CQRS + Event Sourcing
QUESTIONS
*and again I couldn't find a funny meme for this page
roman.sachenko@gmail.com
https://www.facebook.com/rsachenko
https://github.com/roman-sachenko
https://twitter.com/RSachenko
roman.sachenko
NodeJS + CQRS + Event Sourcing
By Roman Sachenko
NodeJS + CQRS + Event Sourcing
Why do I do this? I kept asking myself this question while working on the next project and building microservices-based architecture with CQRS and Event Sourcing. What does this system do and why is it so complicated? I'll tell you about mistakes, which I’ve made, issues, which I've faced with, and solutions, which shouldn’t be applied. I’ll share my own experience and tell you about those things, I wish I’d known before I started working on the project.
- 3,326