Polyglot Persistence

applied with Marten

Tim Cools ( @timcoolsnet )

Expensive Bugs

440.000.000 $

Capitalist money

Knight Capital Group

Expensive Bugs

Expensive Bugs

500.000.000 $

Ariane 5 Crash

> 7.000.000.000 $
Development cost

Expensive Bugs

896 BTC

Flexcoin

401.438 €

Customers money

Expensive Bugs

By sending thousands of simultaneous requests, the attacker was able to "move" coins from one user account to another until the sending account was overdrawn, before balances were updated.

Flexcoin

Expensive Bugs

Data consistency is YOUR responsibility!

NOT of the technology you use...

Expensive Bugs

Polyglot Persistence

- Relational databases

- Distributed databases

- Pittfalls

 

Marten

- Document database & event Store

Content

Data storage used to be (more) simple

RDBMS

Shared database integration

RDBMS

ACID

RDBMS

ACID

TRANSACTIONS

RDBMS

ACID

Atomicity

Consistency

Isolation

Durability

RDBMS

All operations in a transaction are fully completed, or not completed at all.

Atomicity

ACID 

The data updated by a transaction should keep the database in a valid state.

Consistency

ACID 

Transactions are processed as if they run one-by-one after each other.

Isolation

ACI

A 750 €

Isolation

ACI

B 1500 €

750 €

-500 €

Transfer 500€ from A to B

1000 €

+500 €

A 250 €
B 1000 €

single transaction

commit

A 750 €

Isolation

ACI

B 1500 €

750

-500

Transfer 500€ from A to B

1000

+500

A 250 €
B 1000 €

750

-350

Transfer 350€ from A to C

C 2350 €
C 2350 €

+350

2000

A 400 €

concurrent transactions - serialization non-lock

commit

commit

FAILED

Isolation

ACI

Isolation Levels

Serializable

Repeatable reads

Read commited

Read uncommited

Isolation

ACI

Isolation

ACI

When a transaction is completed successfuly the result is stored permanently.

Durability

ACID 

Durability

ACID 

Transaction

Write Transaction Log

Commit

Update buffer

Transaction Log
INSERT
UPDATE
Data Page
A ...
B ...
C ...

File

File

Sync

Async

Index
...
...

Data storage is more complicate these days

Database revolution

Scalability Issues

Database revolution

Development productivity

Key/value

Database revolution

Document

Event Log

Search

Blockchain

Graph

And many more...

Use the database that is best suited for the job

Polyglot Persistence

Polyglot Persistence

Distributed data

CAP Theorem

Consistency

Availability

Partitions

Distributed data

CAP Theorem

Consistency

All data is consistent on all the nodes

CAP Theorem

Strong

Eventual

Weak

CAP Theorem

High availability

of that data (for updates)

CAP Theorem

Partitions

Tolerance to network partitions

CAP Theorem

Only 2 out of 3

C

A

P

CAP Theorem

CA

C

A

RDBMS

Single-node
NoSql

CAP Theorem

CP

C

P

MongoDB

BigTable

Redis

MemcachedDB

CAP Theorem

AP

A

P

Cassandra

Riak

CouchDB

SimpleDB

CAP Theorem

The modern CAP goal should be to maximize combinations of consistency and availability that make sense for the specific application.

Developer Productivity

Developer Productivity

Aggregate Oriented Database

public class Route
{
    private readonly List<Stop> _stops = new List<Stop>();

    public Guid Id { get; private set; }
    public RouteStatus Status { get; private set; }
    public DateTime? Date { get; private set; }

    public IEnumerable Stops => _stops;

    public void Plan(DateTime date)
    {
        if (date < DateTime.Today.AddDays(1))
        {
            throw new InvalidOperationException("Route can only plan from tomorrow.");
        }

        Status = RouteStatus.Planned;
        Date = date;
    }

    ...

Developer Productivity

Aggregate Oriented Database

{  
   "ID":"3f86959a-6970-485c-9a1a-ba168cb77212",
   "Date":"2016-05-26T00:00:00",
   "Stops":{  
      "$type":"System.Collections.Generic.List`1[[Soloco.Talks.PolyglotPersistence.Stop, Soloco.Talks.PolyglotPersistence]], mscorlib",
      "$values":[  
         {  
            "Name":"Home",
            "Position":{ "Latitude":4.481736, "Longitude":51.197894 }
         },
         {  
            "Name":"WooRank",
            "Position":{ "Latitude":4.400963, "Longitude":50.828417 }
         },
         {  
            "Name":"Home",
            "Position":{ "Latitude":4.481736, "Longitude":51.197894 }
         }
      ]
   },
   "Status":"Planned"
}

Polyglot Persistence

Each database system added is added complexity

Polyglot Persistence

Many databases are not mature yet

Polyglot Persistence

Integration between data-stores is hard to get right

Polyglot Persistence

Shared database integration

Integration

Application database

Integration

Integration

Services

Event Driven Architecture

Integration

CQRS

synchronous

Integration

CQRS

synchronous

Integration

CQRS

synchronous

Integration

CQRS

asynchronous

Integration

CQRS+ES

(a)synchronyous

Single Log

Integration

Great power comes with great responsibility

Distributed data

Marten

Marten + PostgreSQL

Relational

Event Store

Document Database

Key/value

(hstore)

* Not Marten

*

*

PostgreSQL

You are in good company

PostgreSQL

And many more...

Marten

Document database

Event Store

> Demo

Marten

Based on proven technlogy

ACID Support

Scalability options

Developer productivity

Marten

Code: Github

Chat: Gitter

Package: Nuget

Questions?

Tim Cools ( @timcoolsnet )

2016-11-10 Polyglot Persistence & Marten #visug

By Tim Cools

2016-11-10 Polyglot Persistence & Marten #visug

Demo marten: https://github.com/tim-cools/soloco-talks

  • 633
Loading comments...

More from Tim Cools