and other new ideas

Andrey Sitnik
Evil Martians

CRDT

for client-server communication

evilmartians.com

I am working at

@sitnikcode

Consultancy boutique working with

@sitnikcode

Autoprefixer

PostCSS

Browserslist

Our known DevTools

@sitnikcode

Logux

AnyCable

Our new DevTools

@sitnikcode

WebSockets

Distributed

CRDT

Why CRDT is cool?

@sitnikcode

Why CRDT is not so scary?

@sitnikcode

Part 1. Good technology?

A way to revolution

  1. New non-standard problem
  2. Old forgotten theory
  3. Find it fitting for other things

@sitnikcode

React

React

Functional programming

Facebook notifications

@sitnikcode

  1. New non-standard problem
  2. Old forgotten theory
  3. Find it fitting for other things

Redux

React Hot Loader

Elm

Redux

@sitnikcode

  1. New non-standard problem
  2. Old forgotten theory
  3. Find it fitting for other things

Part 2. Non-standard problem

A very standard problem

Buy milk

0

Buy break

0

Buy whiskey

1

New to-do

Buy whiskey

1

Leave a comment

You were prepared for this

@sitnikcode

Redux

const action = {
  type: 'ADD_TASK',
  name: 'Buy sugar',
  likes: 0,
  comment: ''
}

store.dispatch(action)
const state2 = { tasks: [X, Y] }

const state3 = reducers(state2, action)

state3 //=> { tasks: [X, Y, Z] }

@sitnikcode

dispatch({ type: 'ADD_STARTED' })
fetch('/tasks', opts)
  .then(() => {
    dispatch({ type: 'ADD_COMPLETE' })
  })
  .catch(() => {
    dispatch({ type: 'ADD_FAILED' })
  })

AJAX and loaders

@sitnikcode

<Mutation mutation={ADD_TODO}>
  {(addTodo, { loading, error }) => (
    !loading && <Form onSubmit={addTodo} />
    { loading && <Loading /> }
    { error && <Error /> }
  )}
</Mutation>

GraphQL and loaders

@sitnikcode

You finished the project

@sitnikcode

Small detail from client

I forgot a small detail.
Multiple users should edit the same task at the same moment.

Would that be so hard to add?

@sitnikcode

Collaboration is a popular feature

@sitnikcode

Especially with

remote work

Should we ask user to resolve conflict?

@sitnikcode

Buy whiskey

Buy vodka

Another user edited the same task.
Please choose the right revision:

Automatic edit conflict resolution

@sitnikcode

Buy whiskey

Buy vodka

Another user change the same task.
Please choose the right revision:

People shouldn’t suffer
Machines should suffer

@sitnikcode

We can add hacks around GraphQL…

@sitnikcode

… or you can go to the dungeons of CS

@sitnikcode

Distributed systems, 1982

@sitnikcode

Part 3. Distributed systems

Requests

GraphQL/AJAX request

The things
we care about

Don’t care

@sitnikcode

Distributed system

Sync

@sitnikcode

Conflicts resolution

{
  name: '伏特加',
  likes: 50
}
{
  name: '白兰地',
  likes: 51
}

?

@sitnikcode

“Bad programmers worry about the code.
Good programmers worry about
data structures”

— Linus Torvalds

Source: https://lwn.net/Articles/193245/

Sending the whole state is a problem

@sitnikcode

Event sourcing

set likes=50
set name='白兰地'
set likes=50
set name='白兰地'

@sitnikcode

Event = Redux action

set likes=50
{ type: 'SET_LIKES', value: 50 }

@sitnikcode

Important part

No AJAX requests

showLoader()
fetch('/profile', {
  credentials: 'include',
  method: 'POST',
  form
}).then(r => {
  hideLoadaer()
}).catch(e => {
  …
})

@sitnikcode

No mutations or sagas

const ADD_TODO = gql`
  mutation addTodo($type: String!) {
    addTodo(type: $type) {
      id
      type
    }
  }
`;

const AddTodo = () => {
  return (
    <Mutation mutation={ADD_TODO}>
      {addTodo => (
        <button onClick={() => {
          addTodo({ variables })
        }}>
      )}
    </Mutation>
  )
}
const createTask = write.bind(
  null, taskList, taskList.push,
  createTaskFailed)

function* watchCreateTask() {
  while (true) {
    let { payload } = yield take(CREATE_TASK);
    yield fork(createTask, payload.task);
  }
}

@sitnikcode

No loaders

Continue to work

Save

@sitnikcode

Keep WebSockets

WS

@sitnikcode

Send actions to server in background

Buy break

Buy break

dispatch({ type: 'DONE', taskId: 12 })
{ type: 'DONE', taskId: 12 }

WS

@sitnikcode

Optimistic UI, 2010

Continue to work

Save

Continue to work

Save

First mention: Google Wave Operational Transformation

@sitnikcode

Server re-sends actions to other clients

{ type: 'DONE', taskId: 12 }

WS

WS

@sitnikcode

Server could create Redux actions too

{ type: 'TRIAL_END' }

WS

@sitnikcode

Offline?

@sitnikcode

Show global warning

Buy milk

0

Buy bread

0

Buy whiskey

1

Offline

@sitnikcode

Redux already keeps actions in memory

@sitnikcode

DB error?

@sitnikcode

Redux already has action undo

@sitnikcode

Server can ask client to undo a bad action

Name A

Rename to B

Name B

Rename to B

Undo

Name A

Part 4. The time

Distributed systems are not so easy

Step 1 Create an idea

Step2 Implement it

Every client creates actions

Action

Client 1

Client 2

Action

Action

Action

@sitnikcode

Actions order problem

Rename to A

Final: B

Rename to B

Rename to B

Final: A

Rename to A

@sitnikcode

Can we use time to order actions?

dispatch({ …, time: Date.now() })
actions.sortBy(i => i.time)

@sitnikcode

Time: not unique

Date.now() //=> 153247098000
Date.now() //=> 153247098000

@sitnikcode

Time: no consistency growth

Date.now() //=> 153247000011

// sudo ntpdate -s time.nist.gov

Date.now() //=> 153247000005

@sitnikcode

Time: no synchroniziation across clients

13:15:00 UTC

12:10:00 UTC

1970-01-01 01:15:00

@sitnikcode

Back to distributed systems dungeons

@sitnikcode

No time = no problem

@sitnikcode

Logical clock, 1978

let counter = 0

function now () {
  counter += 1
  return counter + ':' + uniqueClientId
}

Part 5. Edit conflicts

Merge changes in different fields

{
  type: 'SET_LIKES',
  taskId: 1,
  likes: 50
}
{
  type: 'CHANGE_NAME',
  taskId: 1,
  name: 'Buy bread'
}

@sitnikcode

But how to merge the same field?

{
  type: 'SET_LIKES',
  taskId: 1,
  likes: 50
}
{
  type: 'SET_LIKES',
  taskId: 1,
  likes: 50
}

?

@sitnikcode

Collaborative text editing?

@sitnikcode

How to make actions atomic?

{
  ? // What put here
}

@sitnikcode

@sitnikcode

Back to distributed systems dungeons

CRDT, 2009

hal.upmc.fr/inria-00555588/document

@sitnikcode

One of them after my talk in Porto

Conflict-free replicated data types

  • Op-based counter
  • G-Counter
  • PN-Counter
  • Non-negative Counter
  • LWW-Register
  • MV-Register
  • G-Set
  • 2P-Set
  • LWW-element-Set
  • PN-Set
  • OR-Set
  • Add-only monotonic DAG
  • Add-remove Partial Order
  • Replicated Growable Array
  • Continuous sequence

…

@sitnikcode

Every type has

  • Limits (last write wins)
  • Operations (set key, remove key)

@sitnikcode

P2P: every node could create event

Name = A, time: 2

Name = B, time: 1

@sitnikcode

No master

Same final state after synchronization

Name = A

Sorry, last write wins

@sitnikcode

Name = A

Name = A

Name = A

Name = A

Name = A

Name = A

Op-based counter

{
  type: 'SET_LIKES',
  taskId: 1,
  likes: 51
}
{
  type: 'ADD_LIKE',
  taskId: 1
},
{
  type: 'REMOVE_LIKE',
  taskId: 1
}

@sitnikcode

Conflicts after merging

delete 1

add h after 1
cat → aht

delete symbol at index 1
cat → at

add symbol h after index 1
cat → chat

c a t
1 2 3

String:

Indexes:

@sitnikcode

Linked list

@sitnikcode

'cat' → 'c'—'a'—'t'
                 ↓
{ type: 'ADD', id: 1, symbol: 'c', after: null },
{ type: 'ADD', id: 2, symbol: 'a', after: 1 },
{ type: 'ADD', id: 3, symbol: 't', after: 2 }

Better merging

delete symbol
cat → at

add symbol h to
cat → chat

@sitnikcode

{ type: 'ADD',  id: 1, symbol: 'c', after: null },
{ type: 'ADD',  id: 2, symbol: 'a', after: 1 },
{ type: 'ADD',  id: 3, symbol: 't', after: 2 }
{ type: 'HIDE', id: A }
{ type: 'ADD', id: A, symbol: 'h', after: A }

Better merging

cat → hat

delete symbol
cat → at

add symbol h to
cat → chat

@sitnikcode

{ type: 'ADD',  id: 1, symbol: 'c', after: null },
{ type: 'ADD',  id: 2, symbol: 'a', after: 1 },
{ type: 'ADD',  id: 3, symbol: 't', after: 2 },
{ type: 'HIDE', id: A },
{ type: 'ADD',  id: A, symbol: 'h', after: A }

CRDT is not a silver bullet

CRDT

Real apps

@sitnikcode

CRDT is an inspiration for your custom types

Part 6. The result

Overview

  1. Change state by actions
  2. action.time = distributedTime()
  3. Sync actions log between server and clients
  4. Time travel for undo and inserting in the middle of the history
  5. CRDT to make actions atomic

@sitnikcode

We made a collaborative ToDo app

Buy milk

0

Buy bread

0

Buy whiskey

1

New to-do

Buy whiskey

1

Leave a comment

But maybe we made something bigger?

New non-standard problem

Bring back old forgotten theory

Find it fitting for other things

@sitnikcode

Click to turn off

Click to turn on

Final state on

Save on to DB

Save off to DB

Use case: AJAX, GraphQL

Wrong order because of the network

@sitnikcode

Save on to DB

Ignore

Turn off, time: 1

Final state on

Use case: CRDT

Turn on, time: 2

@sitnikcode

AJAX & GraphQL were designed for

Image:  Nikita Prokopov

@sitnikcode

CRDT was designed for

Image:  Nikita Prokopov

@sitnikcode

40 % packet loss in China

@sitnikcode

No stable Internet in NYC subway

Image: Mr. Robot

Next billion users will not have good Internet

Image: John Stanmeyer

Feature 1

  1. Works with poor connection
  2. More clients
  3. More money

@sitnikcode

CRDT supports also P2P

@sitnikcode

Option: Multi-master servers

@sitnikcode

Option: Mesh

@sitnikcode

Feature 2

  1. P2P-protocol by design
  2. More options
  3. Better scalability

@sitnikcode

Optimistic UI

Optimistic UI

Appling changes before server response

Offline support

Predictable conflict resolution

CRDT

@sitnikcode

Optimistic UI and live updates by design

@sitnikcode

“Every 100ms of additional latency costs Amazon 1% in profit”

More cases: pwastats.com

Optimistic UI → Performance → Money

@sitnikcode

Feature 3

  1. Optimistic UI by design
  2. Faster UI
  3. More money

@sitnikcode

Visual Studio

Collaborative editing is a trend

@sitnikcode

Figma

Trello

Local First

@sitnikcode

You own your data

@sitnikcode

Thanks