Road to

Polyglot Persistence

 

Tim Cools ( @timcoolsnet )

@BuildStuff

Expensive Bugs

440.000.000 $

Capitalist money

Knight Capital Group

Expensive Bugs

Expensive Bugs

500.000.000 $

Ariane 5 Crash

> 7.000.000.000 $
Development cost

Expensive Bugs

896 BTC

Flexcoin

401.438 €

Customers money

Expensive Bugs

Polyglot Persistence

- Relational databases

- Distributed databases

- Pittfalls & integration

Content

Shared database integration

RDBMS

ACID

RDBMS

ACID

TRANSACTIONS

RDBMS

ACID

Atomicity

Consistency

Isolation

Durability

RDBMS

All operations in a transaction are fully completed, or not completed at all.

Atomicity

ACID 

The data updated by a transaction should keep the database in a valid state.

Consistency

ACID 

Transactions are processed as if they run one-by-one after each other.

Isolation

ACI

A 750 €

Isolation

ACI

B 1500 €

750 €

-500 €

Transfer 500€ from A to B

1000 €

+500 €

A 250 €
B 1000 €

single transaction

commit

A 750 €

Isolation

ACI

B 1500 €

750

-500

Transfer 500€ from A to B

1000

+500

A 250 €
B 1000 €

750

-350

Transfer 350€ from A to C

C 2350 €
C 2350 €

+350

2000

A 400 €

concurrent transactions - serialization non-lock

commit

commit

FAILED

Isolation

ACI

Isolation Levels

Serializable

Repeatable reads

Read commited

Read uncommited

Isolation

ACI

Isolation

ACI

When a transaction is completed successfuly the result is stored permanently.

Durability

ACID 

Durability

ACID 

Transaction

Write Transaction Log

Commit

Update buffer

Transaction Log
INSERT
UPDATE
Table Data
A ...
B ...
C ...

Log Files

Data Files

Sync

Async

Index Data
...
...
Data Pages

Scalability Issues

Database revolution

Development productivity

Key/value

Database revolution

Document

Event Log

Search

Blockchain

Graph

And many more...

Tim Cools ( @timcoolsnet )

Use the database that is best suited for the job

Polyglot Persistence

Polyglot Persistence

Distributed data

CAP Theorem

Consistency

Availability

Partitions

Distributed data

CAP Theorem

Consistency

All data is consistent on all the nodes

CAP Theorem

Strong

Eventual

Weak

CAP Theorem

High availability

of that data (for updates)

CAP Theorem

Partitions

Tolerance to network partitions

CAP Theorem

Only 2 out of 3

C

A

P

CAP Theorem

CA

C

A

RDBMS

Single-node
NoSql

CAP Theorem

CA

C

A

RDBMS

Single-node
NoSql

CAP Theorem

CP

C

P

MongoDB

BigTable

Redis

MemcachedDB

CAP Theorem

AP

A

P

Cassandra

Riak

CouchDB

SimpleDB

CAP Theorem

The modern CAP goal should be to maximize combinations of consistency and availability that make sense for the specific application.

Developer Productivity

Developer Productivity

Model

Tables

Mapping

ORM

Developer Productivity

Model

public class Route
{
    private readonly List<Stop> _stops = new List<Stop>();

    public Guid Id { get; private set; }
    public RouteStatus Status { get; private set; }
    public DateTime? Date { get; private set; }

    public IEnumerable Stops => _stops;

    public void Plan(DateTime date)
    {
        if (date < DateTime.Today.AddDays(1))
        {
            throw new InvalidOperationException("Route can only plan from tomorrow.");
        }

        Status = RouteStatus.Planned;
        Date = date;
    }

    ...

Developer Productivity

Document Oriented Database

{  
   "ID":"3f86959a-6970-485c-9a1a-ba168cb77212",
   "Date":"2016-05-26T00:00:00",
   "Stops": [  
     {  
       "Name":"Home",
       "Position":{ "Latitude":4.481736, "Longitude":51.197894 }
     },
     {  
       "Name":"Build Stuff",
       "Position":{ "Latitude":4.400963, "Longitude":50.828417 }
     },
     {  
       "Name":"Home",
       "Position":{ "Latitude":4.481736, "Longitude":51.197894 }
     }
   ],
   "Status":"Planned"
}

Polyglot Persistence

Tim Cools ( @timcoolsnet )

Each database system added is added complexity

Polyglot Persistence

Tim Cools ( @timcoolsnet )

Many databases are not mature yet

Polyglot Persistence

Tim Cools ( @timcoolsnet )

Integration between data-stores is hard to get right

Polyglot Persistence

Tim Cools ( @timcoolsnet )

Shared database integration

Integration

Application database

Integration

Integration

Services

Event Driven Architecture

Integration

CQRS

synchronous

Integration

CQRS

synchronous

Integration

CQRS

synchronous

Integration

CQRS

asynchronous

Integration

CQRS+ES

(a)synchronyous

Single Log

Integration

Great power comes with great responsibility

Polyglot Persistence

Be responsible

Learn new shiny databases

Learn your current databases

 

Learn how to integrate them well

Tim Cools ( @timcoolsnet )

Marten + PostgreSQL

Document

Database

Event Store

Open source .NET library

Tim Cools ( @timcoolsnet )

Thank You

Tim Cools ( @timcoolsnet )

Feedback Form

Questions?

Tim Cools ( @timcoolsnet )