Road to
Polyglot Persistence
Tim Cools ( @timcoolsnet )
@BuildStuff
Expensive Bugs
440.000.000 $
Capitalist money
Knight Capital Group
Expensive Bugs
Expensive Bugs
500.000.000 $
Ariane 5 Crash
> 7.000.000.000 $
Development cost
Expensive Bugs
896 BTC
Flexcoin
401.438 €
Customers money
Expensive Bugs
Polyglot Persistence
- Relational databases
- Distributed databases
- Pittfalls & integration
Content
Shared database integration
RDBMS
ACID
RDBMS
ACID
TRANSACTIONS
RDBMS
ACID
Atomicity
Consistency
Isolation
Durability
RDBMS
All operations in a transaction are fully completed, or not completed at all.
Atomicity
ACID
The data updated by a transaction should keep the database in a valid state.
Consistency
ACID
Transactions are processed as if they run one-by-one after each other.
Isolation
ACID
A | 750 € |
---|
Isolation
ACID
B | 1500 € |
---|
750 €
-500 €
Transfer 500€ from A to B
1000 €
+500 €
A | 250 € |
---|
B | 1000 € |
---|
single transaction
commit
A | 750 € |
---|
Isolation
ACID
B | 1500 € |
---|
750
-500
Transfer 500€ from A to B
1000
+500
A | 250 € |
---|
B | 1000 € |
---|
750
-350
Transfer 350€ from A to C
C | 2350 € |
---|
C | 2350 € |
---|
+350
2000
A | 400 € |
---|
concurrent transactions - serialization non-lock
commit
commit
FAILED
Isolation
ACID
Isolation Levels
Serializable
Repeatable reads
Read commited
Read uncommited
Isolation
ACID
Isolation
ACID
When a transaction is completed successfuly the result is stored permanently.
Durability
ACID
Durability
ACID
Transaction
Write Transaction Log
Commit
Update buffer
Transaction Log |
---|
INSERT |
UPDATE |
Table Data | |
---|---|
A | ... |
B | ... |
C | ... |
Log Files
Data Files
Sync
Async
Index Data |
---|
... |
... |
Data Pages |
---|
Scalability Issues
Database revolution
Development productivity
Key/value
Database revolution
Document
Event Log
Search
Blockchain
Graph
And many more...
Tim Cools ( @timcoolsnet )
Use the database that is best suited for the job
Polyglot Persistence
Polyglot Persistence
Distributed data
CAP Theorem
Consistency
Availability
Partitions
Distributed data
CAP Theorem
Consistency
All data is consistent on all the nodes
CAP Theorem
Strong
Eventual
Weak
CAP Theorem
High availability
of that data (for updates)
CAP Theorem
Partitions
Tolerance to network partitions
CAP Theorem
Only 2 out of 3
C
A
P
CAP Theorem
CA
C
A
RDBMS
Single-node
NoSql
CAP Theorem
CA
C
A
RDBMS
Single-node
NoSql
CAP Theorem
CP
C
P
MongoDB
BigTable
Redis
MemcachedDB
CAP Theorem
AP
A
P
Cassandra
Riak
CouchDB
SimpleDB
CAP Theorem
The modern CAP goal should be to maximize combinations of consistency and availability that make sense for the specific application.
Developer Productivity
Developer Productivity
Model
Tables
Mapping
ORM
Developer Productivity
Model
public class Route
{
private readonly List<Stop> _stops = new List<Stop>();
public Guid Id { get; private set; }
public RouteStatus Status { get; private set; }
public DateTime? Date { get; private set; }
public IEnumerable Stops => _stops;
public void Plan(DateTime date)
{
if (date < DateTime.Today.AddDays(1))
{
throw new InvalidOperationException("Route can only plan from tomorrow.");
}
Status = RouteStatus.Planned;
Date = date;
}
...
Developer Productivity
Document Oriented Database
{
"ID":"3f86959a-6970-485c-9a1a-ba168cb77212",
"Date":"2016-05-26T00:00:00",
"Stops": [
{
"Name":"Home",
"Position":{ "Latitude":4.481736, "Longitude":51.197894 }
},
{
"Name":"Build Stuff",
"Position":{ "Latitude":4.400963, "Longitude":50.828417 }
},
{
"Name":"Home",
"Position":{ "Latitude":4.481736, "Longitude":51.197894 }
}
],
"Status":"Planned"
}
Polyglot Persistence
Tim Cools ( @timcoolsnet )
Each database system added is added complexity
Polyglot Persistence
Tim Cools ( @timcoolsnet )
Many databases are not mature yet
Polyglot Persistence
Tim Cools ( @timcoolsnet )
Integration between data-stores is hard to get right
Polyglot Persistence
Tim Cools ( @timcoolsnet )
Shared database integration
Integration
Application database
Integration
Integration
Services
Event Driven Architecture
Integration
CQRS
synchronous
Integration
CQRS
synchronous
Integration
CQRS
synchronous
Integration
CQRS
asynchronous
Integration
CQRS+ES
(a)synchronyous
Single Log
Integration
Great power comes with great responsibility
Polyglot Persistence
Be responsible
Learn new shiny databases
Learn your current databases
Learn how to integrate them well
Tim Cools ( @timcoolsnet )
Marten + PostgreSQL
Document
Database
Event Store
Open source .NET library
Tim Cools ( @timcoolsnet )
Thank You
Tim Cools ( @timcoolsnet )
Feedback Form
Questions?
Tim Cools ( @timcoolsnet )
2016-11 Polyglot Persistence #buildstuff
By Tim Cools
2016-11 Polyglot Persistence #buildstuff
- 1,307