and other new ideas

Andrey Sitnik
Evil Martians

CRDT

for client-server communication

evilmartians.com

I am working at

@sitnikcode

Consultancy boutique working with

@sitnikcode

Autoprefixer

PostCSS

Browserslist

Our known DevTools

@sitnikcode

Logux

AnyCable

Our new DevTools

@sitnikcode

WebSockets

Distributed

CRDT

Why CRDT is cool?

@sitnikcode

Why CRDT is not so scary?

@sitnikcode

Part 1. Good technology?

A way to revolution

  1. New non-standard problem
  2. Old forgotten theory
  3. Find it fitting for other things

@sitnikcode

React

React

Functional programming

Facebook notifications

@sitnikcode

  1. New non-standard problem
  2. Old forgotten theory
  3. Find it fitting for other things

Redux

React Hot Loader

Elm

Redux

@sitnikcode

  1. New non-standard problem
  2. Old forgotten theory
  3. Find it fitting for other things

Part 2. Non-standard problem

A very standard problem

Buy milk

0

Buy break

0

Buy whiskey

1

New to-do

Buy whiskey

1

Leave a comment

You were prepared for this

@sitnikcode

Redux

const action = {
  type: 'ADD_TASK',
  name: 'Buy sugar',
  likes: 0,
  comment: ''
}

store.dispatch(action)
const state2 = { tasks: [X, Y] }

const state3 = reducers(state2, action)

state3 //=> { tasks: [X, Y, Z] }

@sitnikcode

dispatch({ type: 'ADD_STARTED' })
fetch('/tasks', opts)
  .then(() => {
    dispatch({ type: 'ADD_COMPLETE' })
  })
  .catch(() => {
    dispatch({ type: 'ADD_FAILED' })
  })

AJAX and loaders

@sitnikcode

<Mutation mutation={ADD_TODO}>
  {(addTodo, { loading, error }) => (
    !loading && <Form onSubmit={addTodo} />
    { loading && <Loading /> }
    { error && <Error /> }
  )}
</Mutation>

GraphQL and loaders

@sitnikcode

You finished the project

@sitnikcode

Small detail from client

I forgot a small detail.
Multiple users should edit the same task at the same moment.

Would that be so hard to add?

@sitnikcode

Collaboration is a popular feature

@sitnikcode

Especially with

remote work

Should we ask user to resolve conflict?

@sitnikcode

Buy whiskey

Buy vodka

Another user edited the same task.
Please choose the right revision:

Automatic edit conflict resolution

@sitnikcode

Buy whiskey

Buy vodka

Another user change the same task.
Please choose the right revision:

People shouldn’t suffer
Machines should suffer

@sitnikcode

We can add hacks around GraphQL…

@sitnikcode

… or you can go to the dungeons of CS

@sitnikcode

Distributed systems, 1982

@sitnikcode

Part 3. Distributed systems

Requests

GraphQL/AJAX request

The things
we care about

Don’t care

@sitnikcode

Distributed system

Sync

@sitnikcode

Conflicts resolution

{
  name: '伏特加',
  likes: 50
}
{
  name: '白兰地',
  likes: 51
}

?

@sitnikcode

“Bad programmers worry about the code.
Good programmers worry about
data structures”

— Linus Torvalds

Source: https://lwn.net/Articles/193245/

Sending the whole state is a problem

@sitnikcode

Event sourcing

set likes=50
set name='白兰地'
set likes=50
set name='白兰地'

@sitnikcode

Event = Redux action

set likes=50
{ type: 'SET_LIKES', value: 50 }

@sitnikcode

Important part

No AJAX requests

showLoader()
fetch('/profile', {
  credentials: 'include',
  method: 'POST',
  form
}).then(r => {
  hideLoadaer()
}).catch(e => {
  …
})

@sitnikcode

No mutations or sagas

const ADD_TODO = gql`
  mutation addTodo($type: String!) {
    addTodo(type: $type) {
      id
      type
    }
  }
`;

const AddTodo = () => {
  return (
    <Mutation mutation={ADD_TODO}>
      {addTodo => (
        <button onClick={() => {
          addTodo({ variables })
        }}>
      )}
    </Mutation>
  )
}
const createTask = write.bind(
  null, taskList, taskList.push,
  createTaskFailed)

function* watchCreateTask() {
  while (true) {
    let { payload } = yield take(CREATE_TASK);
    yield fork(createTask, payload.task);
  }
}

@sitnikcode

No loaders

Continue to work

Save

@sitnikcode

Keep WebSockets

WS

@sitnikcode

Send actions to server in background

Buy break

Buy break

dispatch({ type: 'DONE', taskId: 12 })
{ type: 'DONE', taskId: 12 }

WS

@sitnikcode

Optimistic UI, 2010

Continue to work

Save

Continue to work

Save

First mention: Google Wave Operational Transformation

@sitnikcode

Server re-sends actions to other clients

{ type: 'DONE', taskId: 12 }

WS

WS

@sitnikcode

Server could create Redux actions too

{ type: 'TRIAL_END' }

WS

@sitnikcode

Offline?

@sitnikcode

Show global warning

Buy milk

0

Buy bread

0

Buy whiskey

1

Offline

@sitnikcode

Redux already keeps actions in memory

@sitnikcode

DB error?

@sitnikcode

Redux already has action undo

@sitnikcode

Server can ask client to undo a bad action

Name A

Rename to B

Name B

Rename to B

Undo

Name A

Part 4. The time

Distributed systems are not so easy

Step 1 Create an idea

Step2 Implement it

Every client creates actions

Action

Client 1

Client 2

Action

Action

Action

@sitnikcode

Actions order problem

Rename to A

Final: B

Rename to B

Rename to B

Final: A

Rename to A

@sitnikcode

Can we use time to order actions?

dispatch({ …, time: Date.now() })
actions.sortBy(i => i.time)

@sitnikcode

Time: not unique

Date.now() //=> 153247098000
Date.now() //=> 153247098000

@sitnikcode

Time: no consistency growth

Date.now() //=> 153247000011

// sudo ntpdate -s time.nist.gov

Date.now() //=> 153247000005

@sitnikcode

Time: no synchroniziation across clients

13:15:00 UTC

12:10:00 UTC

1970-01-01 01:15:00

@sitnikcode

Back to distributed systems dungeons

@sitnikcode

No time = no problem

@sitnikcode

Logical clock, 1978

let counter = 0

function now () {
  counter += 1
  return counter + ':' + uniqueClientId
}

Part 5. Edit conflicts

Merge changes in different fields

{
  type: 'SET_LIKES',
  taskId: 1,
  likes: 50
}
{
  type: 'CHANGE_NAME',
  taskId: 1,
  name: 'Buy bread'
}

@sitnikcode

But how to merge the same field?

{
  type: 'SET_LIKES',
  taskId: 1,
  likes: 50
}
{
  type: 'SET_LIKES',
  taskId: 1,
  likes: 50
}

?

@sitnikcode

Collaborative text editing?

@sitnikcode

How to make actions atomic?

{
  ? // What put here
}

@sitnikcode

@sitnikcode

Back to distributed systems dungeons

CRDT, 2009

hal.upmc.fr/inria-00555588/document

@sitnikcode

One of them after my talk in Porto

Conflict-free replicated data types

  • Op-based counter
  • G-Counter
  • PN-Counter
  • Non-negative Counter
  • LWW-Register
  • MV-Register
  • G-Set
  • 2P-Set
  • LWW-element-Set
  • PN-Set
  • OR-Set
  • Add-only monotonic DAG
  • Add-remove Partial Order
  • Replicated Growable Array
  • Continuous sequence

…

@sitnikcode

Every type has

  • Limits (last write wins)
  • Operations (set key, remove key)

@sitnikcode

P2P: every node could create event

Name = A, time: 2

Name = B, time: 1

@sitnikcode

No master

Same final state after synchronization

Name = A

Sorry, last write wins

@sitnikcode

Name = A

Name = A

Name = A

Name = A

Name = A

Name = A

Op-based counter

{
  type: 'SET_LIKES',
  taskId: 1,
  likes: 51
}
{
  type: 'ADD_LIKE',
  taskId: 1
},
{
  type: 'REMOVE_LIKE',
  taskId: 1
}

@sitnikcode

Conflicts after merging

delete 1

add h after 1
cat → aht

delete symbol at index 1
cat → at

add symbol h after index 1
cat → chat

c a t
1 2 3

String:

Indexes:

@sitnikcode

Linked list

@sitnikcode

'cat' → 'c'—'a'—'t'
                 ↓
{ type: 'ADD', id: 1, symbol: 'c', after: null },
{ type: 'ADD', id: 2, symbol: 'a', after: 1 },
{ type: 'ADD', id: 3, symbol: 't', after: 2 }

Better merging

delete symbol
cat → at

add symbol h to
cat → chat

@sitnikcode

{ type: 'ADD',  id: 1, symbol: 'c', after: null },
{ type: 'ADD',  id: 2, symbol: 'a', after: 1 },
{ type: 'ADD',  id: 3, symbol: 't', after: 2 }
{ type: 'HIDE', id: A }
{ type: 'ADD', id: A, symbol: 'h', after: A }

Better merging

cat → hat

delete symbol
cat → at

add symbol h to
cat → chat

@sitnikcode

{ type: 'ADD',  id: 1, symbol: 'c', after: null },
{ type: 'ADD',  id: 2, symbol: 'a', after: 1 },
{ type: 'ADD',  id: 3, symbol: 't', after: 2 },
{ type: 'HIDE', id: A },
{ type: 'ADD',  id: A, symbol: 'h', after: A }

CRDT is not a silver bullet

CRDT

Real apps

@sitnikcode

CRDT is an inspiration for your custom types

Part 6. The result

Overview

  1. Change state by actions
  2. action.time = distributedTime()
  3. Sync actions log between server and clients
  4. Time travel for undo and inserting in the middle of the history
  5. CRDT to make actions atomic

@sitnikcode

We made a collaborative ToDo app

Buy milk

0

Buy bread

0

Buy whiskey

1

New to-do

Buy whiskey

1

Leave a comment

But maybe we made something bigger?

New non-standard problem

Bring back old forgotten theory

Find it fitting for other things

@sitnikcode

Click to turn off

Click to turn on

Final state on

Save on to DB

Save off to DB

Use case: AJAX, GraphQL

Wrong order because of the network

@sitnikcode

Save on to DB

Ignore

Turn off, time: 1

Final state on

Use case: CRDT

Turn on, time: 2

@sitnikcode

AJAX & GraphQL were designed for

Image:  Nikita Prokopov

@sitnikcode

CRDT was designed for

Image:  Nikita Prokopov

@sitnikcode

40 % packet loss in China

@sitnikcode

No stable Internet in NYC subway

Image: Mr. Robot

Next billion users will not have good Internet

Image: John Stanmeyer

Feature 1

  1. Works with poor connection
  2. More clients
  3. More money

@sitnikcode

CRDT supports also P2P

@sitnikcode

Option: Multi-master servers

@sitnikcode

Option: Mesh

@sitnikcode

Feature 2

  1. P2P-protocol by design
  2. More options
  3. Better scalability

@sitnikcode

Optimistic UI

Optimistic UI

Appling changes before server response

Offline support

Predictable conflict resolution

CRDT

@sitnikcode

Optimistic UI and live updates by design

@sitnikcode

“Every 100ms of additional latency costs Amazon 1% in profit”

More cases: pwastats.com

Optimistic UI → Performance → Money

@sitnikcode

Feature 3

  1. Optimistic UI by design
  2. Faster UI
  3. More money

@sitnikcode

Visual Studio

Collaborative editing is a trend

@sitnikcode

Figma

Trello

Local First

@sitnikcode

You own your data

@sitnikcode

Thanks

CRDT and other new ideas for client-server communication

By Andrey Sitnik

CRDT and other new ideas for client-server communication

Right now we have great frameworks and technologies to create websites and mobile apps. We have great languages and databases for server back-end. But there are no many changes in a connection between client and server. Many application stops to work with a slow or unstable connection. Loaders block our UX on every small step. Most of the application doesn’t have offline support and live updates. We are not ready for next billion users with slow Internet. But also we are not ready for the unstable Internet in metro or low-signal LTE. Andrey Sitnik, the creator of PostCSS and Autoprefixer, will speak about new ideas in client-server communications. How the mix of ideas behind CRDT, distributed computing, and Elm/Redux could change the UX in most of applications.

  • 16,707

More from Andrey Sitnik