Event Processing
with
Event Store

@zmerta

It's all about
events
and event streams

Event is

something that happened
in the past
and has a business meaning

Events are persisted
in event streams

Event stream is
a time ordered sequence
of events in time

Conceptually
it's append only

Event Sourcing

In traditional systems
we persist the current state
of an object

In event sourced systems
we persist all changes
that lead to the current state
of an object

Every change is
an immutable event.

Object state is restored
by replaying
the entire
stream of events

Benefits

Business benefits
Performance
Scalable
Flexible

Complex Event
Processing

Complex event is
an event that happen
if lots of other events happened

In CEP an event means
a record of an activity
in a system

Event features

Form
Significance
Relativity

Relationships between events

Time
Cause
Aggregation

CEP as a technique helps discover complex events by analyzing and correlating other events

The goal of CEP is to identify meaningful events
(such as opportunities or threats)

CEP relies on number of techniques

Event-pattern detection

Event abstraction

Event filtering

Event aggregation and transformation

Modeling event hierarchies

Detecting relationships between events

Abstracting event-driven processes

Use cases

Financial services
Fraud detection
Business activity monitoring
Security monitoring

The functional database
for storing
and processing events

Open source

The BSD 3-Clause License
(commercial support available)

Multiplatform

Windows
Linux
Mac OS X 

Scalable

Replication
Clustering

Replication

Master node - responsible for write coordination

Slave nodes - replicas

Clustering

Two modes:
Manager and database nodes (Commercial)

​Database nodes (Open Source / Commercial)

Nodes discovery:
DNS
List of other nodes' addresses
 

Administration

Web based UI

Event Store basics

Event

Consists of event data and some system data

eventId - event identifier (could be generated by db)
eventType - defines type of event
data - custom event data
metadata - event metadata

Event example

{
    "eventId": "8cfedd64-7e40-47ee-a16c-e57e2987783b",
    "eventType": "TemperatureMeasured",
    "data": {
        "zone": "Ireland",
        "server": "web1",
        "temperature": 64
    }
}

Event metadata

Additional custom or system data

$correlationId - application level correlation id
$causationId - application level causation id

(both propagated to any new events produced internally)

Event stream

Used to store events
The partition point of the system

Stream category

Streams could be categorized
Category is resolved from stream name (after character "-")

[stream name]-[category]

examples:
temperatures_by_zone-Ireland
temperatures_by_server-web1

Stream metadata

$maxAge - maximum age of events in a stream
$maxCount - maximum number of events in a stream
$cacheControl - controls the cache of the head of a stream
$acl - access control list

Stream ACL

Stream ACL example

{
  "$acl" : {
    "$w"  : "john",
    "$r"  : ["john", "thomas"],
    "$d"  : "$admins",
    "$mw" : "$admins",
    "$mr" : "$admins"
   }
}

API

HTTP
TCP

HTTP

Event streams are exposed as AtomPub feeds
(Atom 1.0. specification)
Restful API

TCP

Asynchronous TCP protocol

(Google Protocol Buffers)
 

clients for C#, Java, Scala, Erlang, Haskell, ...

Which API to use?

HTTP

Scalable for large number of subscribers
(streams are cacheable)

Supported in nearly any environment

TCP

If you need really high performance
(many times faster than HTTP)

Basic stream operations

Append to a stream
Read a stream
Delete stream
Stream subscriptions

Append to a stream

Single event write
Batch write
Transactions (TCP API)

Read a stream

Depends on API

HTTP - AtomPub feed, paging, conditional get
TCP - StreamEventSlice/AllEventsSlice

Delete a stream

Soft delete - stream could be recreated later

Hard delete - stream couldn't be recreated later

Concurrency & Idempotency

All writes are idempotent
Idempotency is based on the EventId and stream
Also depends on ExpectedVersion

Optimistic locking
ExpectedVersion - normally a number representing the version of the stream

Stream subscriptions

Clients can subscribe to the stream and be notified when new events are written to that stream

Volatile subscriptions
Catch-Up subscriptions
Persistent subscriptions

Projections

JavaScript query language
for event processing

Allows you to react
to events ...

Build state,
emit new events
or link to existing events

Concept of
continuous queries

Functional principles

transform(f3(f2(f1(initial(), e1), e2), e3)

f(state, event) => state

f is run over the series of events

transform(state) => result

transform can transform the state to the form of result
you want to receive

initial() => state

initial returns the initial state

fromAll()
fromStream(stream)
fromStreams(stream1, stream2, ..., streamn)
fromCategory(category)

Event selection

Event matching

when([
​  [SomePatternMatch]: function(state, event) {
    return new state;
  },

  [OtherPatternMatch]: function(state, event) {
    return new state;
  }

  ...
]);

Special event matchers

when([
​  $init: function(state, event) {
    return initial state;
  },

  $any: function(state, event) {
    return new state;
  }
]);

Code

Event indexing

linkTo(newStream, event)

Event indexing

Code

Event emitting

emit(stream, eventType, eventData)

Code

Questions?