and other new ideas

Andrey Sitnik
Evil Martians

CRDT

for client-server communication

evilmartians.com

I am working at

@sitnikcode

Autoprefixer

PostCSS

Browserslist

The creator of

@sitnikcode

Autoprefixer was started in 2013

twitter.com/andrey_sitnik/status/321655851565850624

@sitnikcode

Now Autoprefixer and PostCSS grew up

NPM Trends

@sitnikcode

They don’t need new features

Image: CLM BBDO Paris

It is a time for а new adventure

Part 1. What is next?

GraphQL is cool

@sitnikcode

But it is the present, not the future

Create the idea

Create API

Solve production problems

Polish the code

Promote

@sitnikcode

What’s the next big idea?

RPC

SOAP

REST

GraphQL

?

@sitnikcode

And then I saw CRDT talk by Victor Grishenko

Swarm.js

What I believe in

RPC

SOAP

REST

GraphQL

CRDT

¯\_(ツ)_/¯
not 100% sure

@sitnikcode

A way fo revolution

  1. New non-standard problem
  2. Bring back old forgotten theory
  3. Find it fitting for other things

@sitnikcode

America

  1. New non-standard problem
  2. Bring back old forgotten theory
  3. Find it fitting for other things

Indian spices

Round Earth theory

America

@sitnikcode

React

  1. New non-standard problem
  2. Bring back old forgotten theory
  3. Find it fitting for other things

React

Functional programming

Facebook notifications

@sitnikcode

Redux

  1. New non-standard problem
  2. Bring back old forgotten theory
  3. Find it fitting for other things

React Hot Loader

Elm

Redux

@sitnikcode

Part 2. Non-standard problem

A very standard problem

Buy a milk

0

Buy a break

0

Buy a whiskey

1

New to-do

Buy a whiskey

1

Leave a comment

You were prepared for this

@sitnikcode

Vuex

const action = {
  type: 'addTask',
  name: 'Buy a sugar',
  likes: 0,
  comment: ''
}

store.dispatch(action)

@sitnikcode

commit('addStarted'})
fetch('/tasks', opts)
  .then(() => {
    commit('addComplete'})
  })
  .catch(() => {
    commit('addFailed'})
  })

AJAX and loaders

@sitnikcode

  methods: {
    add() {
      // loading
      this.$apollo.mutate(mutation)
        .then(() => …)
        .catch(() => …)
    }
  }

GraphQL and loaders

@sitnikcode

You finished the project

@sitnikcode

Small detail from client

I forgot a small detail.
Multiple users should edit the same task at the same moment.

Should not be that hard to add?

@sitnikcode

Should we ask user to merge conflict?

@sitnikcode

Buy a whiskey

Buy a vodka

Another user change the same task.
Please choose the right revision:

We need automaic edit conflict resolution

@sitnikcode

Buy a whiskey

Buy a vodka

Another user change the same task.
Please choose the right revision:

People shouldn’t suffer. Machines should suffer

@sitnikcode

We can add hacks around GraphQL …

@sitnikcode

… or you can go to the dungeons of CS

@sitnikcode

Distributed systems, 1982

@sitnikcode

Part 3. Distributed systems

Requests

GraphQL/AJAX request

The things
we care about

Don’t care

@sitnikcode

Distributed system

Sync

@sitnikcode

Conflicts resolution

{
  name: 'Водка',
  likes: 50
}
{
  name: 'Виски',
  likes: 51
}

?

@sitnikcode

“Bad programmers worry about the code.
Good programmers worry about
data structures”

— Linus Torvalds

Source: https://lwn.net/Articles/193245/

Sending the whole state is a problem

@sitnikcode

Event sourcing

set likes=50
set name='Виски'
set likes=50
set name='Виски'

@sitnikcode

Event = Redux action

set likes=50
{ type: 'SET_LIKES', value: 50 }

@sitnikcode

Important part

No AJAX requests

showLoader()
fetch('/profile', {
  credentials: 'include',
  method: 'POST',
  form
}).then(r => {
  hideLoadaer()
}).catch(e => {
  …
})

@sitnikcode

No mutations or sagas

const ADD_TODO = gql`
  mutation addTodo($type: String!) {
    addTodo(type: $type) {
      id
      type
    }
  }
`;


this.$apollo.mutate({
  …
})
const createTask = write.bind(
  null, taskList, taskList.push,
  createTaskFailed)

function* watchCreateTask() {
  while (true) {
    let { payload } = yield take(CREATE_TASK);
    yield fork(createTask, payload.task);
  }
}

@sitnikcode

No loaders

Continue to work

Save

@sitnikcode

Keep WebSockets

WS

@sitnikcode

Send Redux actions to server in background

Buy a break

Buy a break

dispatch({ type: 'DONE', taskId: 12 })
{ type: 'DONE', taskId: 12 }

WS

@sitnikcode

Optimistic UI, 2010

Continue to work

Save

Continue to work

Save

First mention: Google Wave Operational Transformation

@sitnikcode

Server re-send actions to other clients

{ type: 'DONE', taskId: 12 }

WS

WS

@sitnikcode

Server could create Redux actions too

{ type: 'TRIAL_END' }

WS

@sitnikcode

What if you need a loader?

Pessimistic UI

@sitnikcode

Use two actions for Pessimistic UI

{ type: 'PAY', … }
{ type: 'PAID' }

Offline?

@sitnikcode

Show global warning

Buy a milk

0

Buy a break

0

Buy a whiskey

1

Offline

@sitnikcode

Vuex is ready to keep actions in memory

@sitnikcode

DB error?

@sitnikcode

Vuex already has action undo in DevTools

@sitnikcode

Server can ask a client to undo wrong action

Name A

Rename to B

Name B

Rename to B

Undo

Name A

Part 4. The time

Distributed systems are not so easy

Step 1 Create an idea

Step2 Implement it

Every client could create action

Action

Client 1

Client 2

Action

Action

Action

@sitnikcode

Actions order problem

Rename to A

Final: B

Rename to B

Rename to B

Final: A

Rename to A

@sitnikcode

 Time to maintain the same action order?

dispatch({ …, time: Date.now() })
actions.sortBy(i => i.time)

@sitnikcode

Time: not unique

Date.now() //=> 153247098000
Date.now() //=> 153247098000

@sitnikcode

Time: no consistency growth

Date.now() //=> 153247000011

// sudo ntpdate -s time.nist.gov

Date.now() //=> 153247000005

@sitnikcode

Time: no synchroniziation across clients

13:15:00 UTC

12:10:00 UTC

1970-01-01 01:15:00

@sitnikcode

Return to distributed systems dungeons

@sitnikcode

No time = no problem

@sitnikcode

Logical clock, 1978

let counter = 0

function now () {
  counter += 1
  return counter + ':' + uniqueClientId
}

Update counter after receiving actions

1:clientA

2:clientB

3:clientA

3:clientB

@sitnikcode

Part 5. Edit conflicts

Easy to merge changes in different fields

{
  type: 'setLikes',
  taskId: 1,
  likes: 50
}
{
  type: 'changeName',
  taskId: 1,
  name: 'Buy a bread'
}

@sitnikcode

But how to merge changes of the same field?

{
  type: 'setLikes',
  taskId: 1,
  likes: 50
}
{
  type: 'setLikes',
  taskId: 1,
  likes: 50
}

?

@sitnikcode

Collaborative text editing?

@sitnikcode

How to make actions atomic?

{
  ? // What put here
}

@sitnikcode

Return to distributed systems dungeons

@sitnikcode

CRDT, 2009

hal.upmc.fr/inria-00555588/document

@sitnikcode

One of them after my talk in Porto

Conflict-free replicated data types

  • Op-based counter
  • G-Counter
  • PN-Counter
  • Non-negative Counter
  • LWW-Register
  • MV-Register
  • G-Set
  • 2P-Set
  • LWW-element-Set
  • PN-Set
  • OR-Set
  • Add-only monotonic DAG
  • Add-remove Partial Order
  • Replicated Growable Array
  • Continuous sequence

…

@sitnikcode

Types are different

Op-based

State-based

With distributed time

Without timer

@sitnikcode

Every type has

  • Limits (last write wins)
  • Operations (set key, remove key)

@sitnikcode

P2P: every node could create event

Name = A, time: 2

Name = B, time: 1

@sitnikcode

No master

Same final state after synchronization

Name = A

Name = A

Name = A

Name = A

Name = A

Name = A

Name = A

Sorry, last write wins

@sitnikcode

Op-based counter

{
  type: 'setLikes',
  taskId: 1,
  likes: 51
}
{
  type: 'addLike',
  taskId: 1
},
{
  type: 'removeLike',
  taskId: 1
}

@sitnikcode

Action for changing every symbol in text?

c a t
1 2 3

delete symbol at index 1
cat → at

add symbol h after index 1
cat → chat

String:

Indexes:

@sitnikcode

Operations change symbol indexes

c a t
1 2 3

delete symbol at index 1

String:

Indexes:

a t
1 2

String:

Indexes:

@sitnikcode

Conflicts after merging

delete 1

add h after 1
cat → aht

delete symbol at index 1
cat → at

add symbol h after index 1
cat → chat

c a t
1 2 3

String:

Indexes:

@sitnikcode

Persistent indexes

c a t
1 2 3

delete symbol at index 1

String:

Indexes:

a t
2 3

String:

Indexes:

add symbol h after index 1

c a t
1 2 3

String:

Indexes:

c  h  a  t
1 1.5 2  3

String:

Indexes:

@sitnikcode

Better merging

delete 1

add h to 1.5
cat → hat

delete symbol at index 1
cat → at

add symbol h to index 1.5
cat → chat

c a t
1 2 3

String:

Indexes:

@sitnikcode

CRDT is not a silver bullet

CRDT

Real apps

@sitnikcode

CRDT is inspiration for your custom types

Part 6. The result

Overview

  1. Vuex, Redux, etc
  2. action.time = distributedTime()
  3. Sync actions between server and clients
  4. Time travel for undo and inserting in the middle of the history
  5. CRDT to make actions atomic

@sitnikcode

We made a collaborative ToDo app

Buy a milk

0

Buy a break

0

Buy a whiskey

1

New to-do

Buy a whiskey

1

Leave a comment

But maybe we made something bigger?

New non-standard problem

Bring back old forgotten theory

Find it fitting for other things

@sitnikcode

Click to turn off

Click to turn on

Final state on

Save on to DB

Save off to DB

Use case: AJAX, GraphQL

Wrong order because of the network

@sitnikcode

Save on to DB

Ignore

Turn off, time: 1

Final state on

Use case: CRDT

Turn on, time: 2

@sitnikcode

AJAX & GraphQL were designed for

Image:  Nikita Prokopov

@sitnikcode

CRDT was designed for

Image:  Nikita Prokopov

@sitnikcode

40 % packet lost for Western websites

@sitnikcode

No stable Internet in NY subway

Image: Mr. Robot

Next billion users will not have good Internet

Image: John Stanmeyer

Feature 1

  1. Bad connection support
  2. More clients
  3. More money

@sitnikcode

Bad connection support because CRDT is P2P

@sitnikcode

Option: Multi-master servers

@sitnikcode

Option: Cross-tab sync

Created by Patrick Brentanofrom the Noun Project

Only leader tab keeps WS

@sitnikcode

Option: Mesh

@sitnikcode

Feature 2

  1. P2P-protocol by design
  2. More options
  3. Better scalability

@sitnikcode

Optimistic UI

Optimistic UI

Appling changes before server response

Offline support

Predictable conflict resolution

CRDT

@sitnikcode

Optimistic UI and live updates by design

@sitnikcode

Feature 3

  1. Subscriptions by design
  2. Always latest state
  3. Users will encounter less mistakes

@sitnikcode

“Every 100ms of additional latency costs Amazon 1% in profit”

More cases: pwastats.com

Performance → money

@sitnikcode

Feature 4

  1. Optimistic UI by design
  2. Faster UI
  3. More money

@sitnikcode

Atom Teletype

Visual Studio Live Share

CRDT in Redis

CRDT is a trend

SoundCloud

@sitnikcode

CRDT for regular apps

What I believe in

RPC

SOAP

REST

GraphQL

CRDT

¯\_(ツ)_/¯
not 100% sure

@sitnikcode

  • Swarm.js
  • Yjs
  • OrbitDB
  • automerge
  • …

Many solutions with some of ideas

@sitnikcode

Logux: My CRDT framework

@sitnikcode

VPN ¯\_(ツ)_/¯

Thanks

VPN ¯\_(ツ)_/¯

VPN ¯\_(ツ)_/¯

CRDT and other new ideas for client-server communication

By Andrey Sitnik

CRDT and other new ideas for client-server communication

Right now we have great frameworks and technologies to create websites and mobile apps. We have great languages and databases for server back-end. But there are no many changes in a connection between client and server. Many application stops to work with a slow or unstable connection. Loaders block our UX on every small step. Most of the application doesn’t have offline support and live updates. We are not ready for next billion users with slow Internet. But also we are not ready for the unstable Internet in metro or low-signal LTE. Andrey Sitnik, the creator of PostCSS and Autoprefixer, will speak about new ideas in client-server communications. How the mix of ideas behind CRDT, distributed computing, and Elm/Redux could change the UX in most of applications.

  • 5,100

More from Andrey Sitnik