and other new ideas

Andrey Sitnik
Evil Martians

CRDT

for client-server communication

evilmartians.com

I am working at

@sitnikcode

Autoprefixer

PostCSS

Browserslist

The creator of

@sitnikcode

Autoprefixer was started in 2013

twitter.com/andrey_sitnik/status/321655851565850624

@sitnikcode

Now Autoprefixer and PostCSS grew up

NPM Trends

@sitnikcode

They don’t need new features

Image: CLM BBDO Paris

It is a time for а new adventure

Part 1. What is next?

GraphQL is cool

@sitnikcode

But it is the present, not the future

Create the idea

Create API

Solve production problems

Polish the code

Promote

@sitnikcode

What’s the next big idea?

RPC

SOAP

REST

GraphQL

?

@sitnikcode

And then I saw CRDT talk by Victor Grishenko

Swarm.js

What I believe in

RPC

SOAP

REST

GraphQL

CRDT

¯\_(ツ)_/¯
not 100% sure

@sitnikcode

A way fo revolution

  1. New non-standard problem
  2. Bring back old forgotten theory
  3. Find it fitting for other things

@sitnikcode

America

  1. New non-standard problem
  2. Bring back old forgotten theory
  3. Find it fitting for other things

Indian spices

Round Earth theory

America

@sitnikcode

React

  1. New non-standard problem
  2. Bring back old forgotten theory
  3. Find it fitting for other things

React

Functional programming

Facebook notifications

@sitnikcode

Redux

  1. New non-standard problem
  2. Bring back old forgotten theory
  3. Find it fitting for other things

React Hot Loader

Elm

Redux

@sitnikcode

Part 2. Non-standard problem

A very standard problem

Buy a milk

0

Buy a break

0

Buy a whiskey

1

New to-do

Buy a whiskey

1

Leave a comment

You were prepared for this

@sitnikcode

Vuex

const action = {
  type: 'addTask',
  name: 'Buy a sugar',
  likes: 0,
  comment: ''
}

store.dispatch(action)

@sitnikcode

commit('addStarted'})
fetch('/tasks', opts)
  .then(() => {
    commit('addComplete'})
  })
  .catch(() => {
    commit('addFailed'})
  })

AJAX and loaders

@sitnikcode

  methods: {
    add() {
      // loading
      this.$apollo.mutate(mutation)
        .then(() => …)
        .catch(() => …)
    }
  }

GraphQL and loaders

@sitnikcode

You finished the project

@sitnikcode

Small detail from client

I forgot a small detail.
Multiple users should edit the same task at the same moment.

Should not be that hard to add?

@sitnikcode

Should we ask user to merge conflict?

@sitnikcode

Buy a whiskey

Buy a vodka

Another user change the same task.
Please choose the right revision:

We need automaic edit conflict resolution

@sitnikcode

Buy a whiskey

Buy a vodka

Another user change the same task.
Please choose the right revision:

People shouldn’t suffer. Machines should suffer

@sitnikcode

We can add hacks around GraphQL …

@sitnikcode

… or you can go to the dungeons of CS

@sitnikcode

Distributed systems, 1982

@sitnikcode

Part 3. Distributed systems

Requests

GraphQL/AJAX request

The things
we care about

Don’t care

@sitnikcode

Distributed system

Sync

@sitnikcode

Conflicts resolution

{
  name: 'Водка',
  likes: 50
}
{
  name: 'Виски',
  likes: 51
}

?

@sitnikcode

“Bad programmers worry about the code.
Good programmers worry about
data structures”

— Linus Torvalds

Source: https://lwn.net/Articles/193245/

Sending the whole state is a problem

@sitnikcode

Event sourcing

set likes=50
set name='Виски'
set likes=50
set name='Виски'

@sitnikcode

Event = Redux action

set likes=50
{ type: 'SET_LIKES', value: 50 }

@sitnikcode

Important part

No AJAX requests

showLoader()
fetch('/profile', {
  credentials: 'include',
  method: 'POST',
  form
}).then(r => {
  hideLoadaer()
}).catch(e => {
  …
})

@sitnikcode

No mutations or sagas

const ADD_TODO = gql`
  mutation addTodo($type: String!) {
    addTodo(type: $type) {
      id
      type
    }
  }
`;


this.$apollo.mutate({
  …
})
const createTask = write.bind(
  null, taskList, taskList.push,
  createTaskFailed)

function* watchCreateTask() {
  while (true) {
    let { payload } = yield take(CREATE_TASK);
    yield fork(createTask, payload.task);
  }
}

@sitnikcode

No loaders

Continue to work

Save

@sitnikcode

Keep WebSockets

WS

@sitnikcode

Send Redux actions to server in background

Buy a break

Buy a break

dispatch({ type: 'DONE', taskId: 12 })
{ type: 'DONE', taskId: 12 }

WS

@sitnikcode

Optimistic UI, 2010

Continue to work

Save

Continue to work

Save

First mention: Google Wave Operational Transformation

@sitnikcode

Server re-send actions to other clients

{ type: 'DONE', taskId: 12 }

WS

WS

@sitnikcode

Server could create Redux actions too

{ type: 'TRIAL_END' }

WS

@sitnikcode

What if you need a loader?

Pessimistic UI

@sitnikcode

Use two actions for Pessimistic UI

{ type: 'PAY', … }
{ type: 'PAID' }

Offline?

@sitnikcode

Show global warning

Buy a milk

0

Buy a break

0

Buy a whiskey

1

Offline

@sitnikcode

Vuex is ready to keep actions in memory

@sitnikcode

DB error?

@sitnikcode

Vuex already has action undo in DevTools

@sitnikcode

Server can ask a client to undo wrong action

Name A

Rename to B

Name B

Rename to B

Undo

Name A

Part 4. The time

Distributed systems are not so easy

Step 1 Create an idea

Step2 Implement it

Every client could create action

Action

Client 1

Client 2

Action

Action

Action

@sitnikcode

Actions order problem

Rename to A

Final: B

Rename to B

Rename to B

Final: A

Rename to A

@sitnikcode

 Time to maintain the same action order?

dispatch({ …, time: Date.now() })
actions.sortBy(i => i.time)

@sitnikcode

Time: not unique

Date.now() //=> 153247098000
Date.now() //=> 153247098000

@sitnikcode

Time: no consistency growth

Date.now() //=> 153247000011

// sudo ntpdate -s time.nist.gov

Date.now() //=> 153247000005

@sitnikcode

Time: no synchroniziation across clients

13:15:00 UTC

12:10:00 UTC

1970-01-01 01:15:00

@sitnikcode

Return to distributed systems dungeons

@sitnikcode

No time = no problem

@sitnikcode

Logical clock, 1978

let counter = 0

function now () {
  counter += 1
  return counter + ':' + uniqueClientId
}

Update counter after receiving actions

1:clientA

2:clientB

3:clientA

3:clientB

@sitnikcode

Part 5. Edit conflicts

Easy to merge changes in different fields

{
  type: 'setLikes',
  taskId: 1,
  likes: 50
}
{
  type: 'changeName',
  taskId: 1,
  name: 'Buy a bread'
}

@sitnikcode

But how to merge changes of the same field?

{
  type: 'setLikes',
  taskId: 1,
  likes: 50
}
{
  type: 'setLikes',
  taskId: 1,
  likes: 50
}

?

@sitnikcode

Collaborative text editing?

@sitnikcode

How to make actions atomic?

{
  ? // What put here
}

@sitnikcode

Return to distributed systems dungeons

@sitnikcode

CRDT, 2009

hal.upmc.fr/inria-00555588/document

@sitnikcode

One of them after my talk in Porto

Conflict-free replicated data types

  • Op-based counter
  • G-Counter
  • PN-Counter
  • Non-negative Counter
  • LWW-Register
  • MV-Register
  • G-Set
  • 2P-Set
  • LWW-element-Set
  • PN-Set
  • OR-Set
  • Add-only monotonic DAG
  • Add-remove Partial Order
  • Replicated Growable Array
  • Continuous sequence

…

@sitnikcode

Types are different

Op-based

State-based

With distributed time

Without timer

@sitnikcode

Every type has

  • Limits (last write wins)
  • Operations (set key, remove key)

@sitnikcode

P2P: every node could create event

Name = A, time: 2

Name = B, time: 1

@sitnikcode

No master

Same final state after synchronization

Name = A

Name = A

Name = A

Name = A

Name = A

Name = A

Name = A

Sorry, last write wins

@sitnikcode

Op-based counter

{
  type: 'setLikes',
  taskId: 1,
  likes: 51
}
{
  type: 'addLike',
  taskId: 1
},
{
  type: 'removeLike',
  taskId: 1
}

@sitnikcode

Action for changing every symbol in text?

c a t
1 2 3

delete symbol at index 1
cat → at

add symbol h after index 1
cat → chat

String:

Indexes:

@sitnikcode

Operations change symbol indexes

c a t
1 2 3

delete symbol at index 1

String:

Indexes:

a t
1 2

String:

Indexes:

@sitnikcode

Conflicts after merging

delete 1

add h after 1
cat → aht

delete symbol at index 1
cat → at

add symbol h after index 1
cat → chat

c a t
1 2 3

String:

Indexes:

@sitnikcode

Persistent indexes

c a t
1 2 3

delete symbol at index 1

String:

Indexes:

a t
2 3

String:

Indexes:

add symbol h after index 1

c a t
1 2 3

String:

Indexes:

c  h  a  t
1 1.5 2  3

String:

Indexes:

@sitnikcode

Better merging

delete 1

add h to 1.5
cat → hat

delete symbol at index 1
cat → at

add symbol h to index 1.5
cat → chat

c a t
1 2 3

String:

Indexes:

@sitnikcode

CRDT is not a silver bullet

CRDT

Real apps

@sitnikcode

CRDT is inspiration for your custom types

Part 6. The result

Overview

  1. Vuex, Redux, etc
  2. action.time = distributedTime()
  3. Sync actions between server and clients
  4. Time travel for undo and inserting in the middle of the history
  5. CRDT to make actions atomic

@sitnikcode

We made a collaborative ToDo app

Buy a milk

0

Buy a break

0

Buy a whiskey

1

New to-do

Buy a whiskey

1

Leave a comment

But maybe we made something bigger?

New non-standard problem

Bring back old forgotten theory

Find it fitting for other things

@sitnikcode

Click to turn off

Click to turn on

Final state on

Save on to DB

Save off to DB

Use case: AJAX, GraphQL

Wrong order because of the network

@sitnikcode

Save on to DB

Ignore

Turn off, time: 1

Final state on

Use case: CRDT

Turn on, time: 2

@sitnikcode

AJAX & GraphQL were designed for

Image:  Nikita Prokopov

@sitnikcode

CRDT was designed for

Image:  Nikita Prokopov

@sitnikcode

40 % packet lost for Western websites

@sitnikcode

No stable Internet in NY subway

Image: Mr. Robot

Next billion users will not have good Internet

Image: John Stanmeyer

Feature 1

  1. Bad connection support
  2. More clients
  3. More money

@sitnikcode

Bad connection support because CRDT is P2P

@sitnikcode

Option: Multi-master servers

@sitnikcode

Option: Cross-tab sync

Created by Patrick Brentanofrom the Noun Project

Only leader tab keeps WS

@sitnikcode

Option: Mesh

@sitnikcode

Feature 2

  1. P2P-protocol by design
  2. More options
  3. Better scalability

@sitnikcode

Optimistic UI

Optimistic UI

Appling changes before server response

Offline support

Predictable conflict resolution

CRDT

@sitnikcode

Optimistic UI and live updates by design

@sitnikcode

Feature 3

  1. Subscriptions by design
  2. Always latest state
  3. Users will encounter less mistakes

@sitnikcode

“Every 100ms of additional latency costs Amazon 1% in profit”

More cases: pwastats.com

Performance → money

@sitnikcode

Feature 4

  1. Optimistic UI by design
  2. Faster UI
  3. More money

@sitnikcode

Atom Teletype

Visual Studio Live Share

CRDT in Redis

CRDT is a trend

SoundCloud

@sitnikcode

CRDT for regular apps

What I believe in

RPC

SOAP

REST

GraphQL

CRDT

¯\_(ツ)_/¯
not 100% sure

@sitnikcode

  • Swarm.js
  • Yjs
  • OrbitDB
  • automerge
  • …

Many solutions with some of ideas

@sitnikcode

Logux: My CRDT framework

@sitnikcode

VPN ¯\_(ツ)_/¯

Thanks

VPN ¯\_(ツ)_/¯

VPN ¯\_(ツ)_/¯