A new way of client-server conversation

Andrey Sitnik, Evil Martians

Russia is fun

Evil Martians front-end open source

Animakit

Autoprefixer

PostCSS

AnyCable

My first pull request …

… was not that successful

Revenge from Autoprefixer

An issue for Rework …

… was not successful either

Rework is dead

Open source advice

Maintainer — be polite

Contributor — don’t be afraid to fail

Front-end architecture …

State

Reducers

Components

… is great

State

Reducers

Components

image/svg+xml

Back-end …

State

Reducers

Components

image/svg+xml

View

Controller

Model

… is great too

State

Reducers

Components

image/svg+xml

View

Controller

Model

image/svg+xml

But the problem lies in between

State

Reducers

Components

image/svg+xml

View

Controller

Model

image/svg+xml

Communication

AJAX Requests

Chapter 1

Websites

Static pages

DHTML & jQuery

SPA

Live updates

Offline

New synchronization

AJAX

Two-field form

<form>
  <input name="name"…>
  <input name="color"…>
</form>

Server synchronization code

import { Actions } from 'flummox'
import api from '../api/labels-api'
export default class LabelsActions extends Actions {
  async add(project, data) {
    return await api.add(project, data).then( id => {
      return [id, { ...data, id, project }]
    })
  }
  async update(pr, id, data) {
    return await api.update(pr, id, data).then(() => {
      return [id, { ...data, id, project }]
    })
  }
  async remove(project, id) {
    return await api.remove(project, id).then(() => id )
  }
}
import { OrderedMap } from 'immutable'
import { Store }      from 'flummox'
export default class LabelsStore extends Store {
  constructor(flux, data) {
    super()
    this.state = {
      labels: OrderedMap(data.map(i => [parseInt(i.id), i]))
    }
    let actions = flux.getActionIds('labels')
    this.register(actions.add,    this.update)
    this.register(actions.update, this.update)
    this.register(actions.remove, this.remove)
  }
  remove(id) {
    this.setState({
      labels: this.state.labels.delete(parseInt(id))
    })
  }
  update([id, data]) {
    this.setState({
      labels: this.state.labels.set(parseInt(id), data)
    })
  }
}
fetch('/comments.json').then(r => r.json()).then(list => {
  setComments(list)
})

Simple fetch

showLoader()
fetch('/comments.json').then(r => r.json()).then(list => {
  hideLoader()
  setComments(list)
})

Fetch + loader

showLoader()
fetch('/comments.json').then(r => r.json()).then(list => {
  hideLoader()
  setComments(list)
}).catch(err => {
  if (isNoNetwork(err)) {
    showNetworkError(err)
  } else {
    showServerError(err)
  }
})

Fetch + loader + error handling

Local connection in development

Image author: Nikita Prokopov

Real connection in production

Image author: Nikita Prokopov

Complexity curve with AJAX

Complexity

Features

More complicated problems

  • Deduplication
  • Relevance
  • Protocol version validation
  • Conflicts

Better world

SPA

Live Updates

Offline

Logux goals

  1. Reduce code size
  2. Support current ecosystems
  3. Live updates out of the box
  4. Optimistic UI & Offline-first

Idea

Chapter 2

Saint Petersburg — best bars in Russia

Conflict

Images author: Nikita Prokopov

{
  title: '伏特加',
  likes: 50
}
{
  title: '白兰地',
  likes: 51
}

?

Solution

“Bad programmers worry about the code.
Good programmers worry about
data structures”

Linus Torvalds

Event sourcing

Images author: Nikita Prokopov

['post:title', '伏特加']
['likes:set', 51]

['likes:remove', 1]
['post:title', '伏特加']
['likes:set', 51]

['post:title', '白兰地']

CRDT, 2009

Conflict-free Replicated Data Types

  • Op-based counter
  • G-Counter
  • PN-Counter
  • Non-negative Counter
  • LWW-Register
  • MV-Register
  • G-Set
  • 2P-Set
  • LWW-element-Set
  • PN-Set
  • OR-Set
  • Add-only monotonic DAG
  • Add-remove Partial Order
  • Replicated Growable Array
  • Continuous sequence

CRDT in JS

Swarm.js

@gritzko

Event sourcing is everywhere

Redux
DevTools
log

CRDT
log

Backend DB
oplog

Log must be a first-class citizen

Logux inspiration

Redux

Swarm.js

+

Actions log on client

{
  type: 'CHANGE_NAME',
  user: 13,
  name: 'New name'
}

Images author: Nikita Prokopov

Actions log on server

Images author: Nikita Prokopov

Synchronize logs

Images author: Nikita Prokopov

WebSocket

Synchronization is simple

1

2

3

4

5

Images author: Nikita Prokopov

— Give me new actions since “4”

Server actions

Images author: Nikita Prokopov

Between clients

Images author: Nikita Prokopov

Problems and Solutions

Chapter 3

How to write open source

Step 1 Create an idea

Step2 Implement it

Problem 1: order of actions

A

A

B

B

State: B

State: A

Images author: Nikita Prokopov

Sort by time?

{ action }, { time }

Distributed time is сomplicated

Questions about time: 1

const time1 = Date.now()
const time2 = Date.now()

time2 !== time1

A millisecond is a lot of time

Millisecond

Millisecond

Actions

Questions about time: 2

const time1 = Date.now()
const time2 = Date.now()

time2 >= time1

CloudFlare leap second problem

Questions about time: 3

const client = await client.askTime()
const server = Date.now()

diff(client, server) < HOUR

Wrong time zone

Solution 1: action ID

id: [1489328408677, '10:bfj58s0w', 0]

Milliseconds

Unique Node ID

User ID

Random string

Timestamp always increases

if (now < prevNow) {
  now = prevNow
}
prevNow = now

Sequence number

[1489328408677, '10:…', 0]
[1489328408677, '10:…', 1]
[1489328408678, '10:…', 0]

Same milliseconds

Server sends current time

[
  'connected',
  [0, 0],          // Protocol version
  'server:h4vjdl', // Server unique ID
  [
    1475316481050, // "connect" message was received
    1475316482879  // "connected" message was sent
  ]
]

Client calculates time difference

const roundTrip = now - startTime - authTime
const timeFix = startTime -
                receivedTime +
                roundTrip / 2

Client applies time difference

{ // Received
  id,
  time: id[0] + timeFix
}

{ // Sent
  id,
  time: id[0] - timeFix
}

Chinese Internet

40% packet loss

Problem 2: disconnect detection

ws.onclose = function () {
  // Could be never called
}

Solution 2: ping command

> ['ping', 120]
< ['pong', 130]

Live connection status

if (loguxClient.connected) {
  // Connection was tested
  // at least 5 second ago
}

Problem 3: tabs conflict

Images author: Sebastien Gabriel & Nikita Prokopov

Duplicate

Action

Saved

I hate programming
I hate programming
I hate programming
It works!
I love programming

Solution 3: only the leader tab keeps WS

Images author: Sebastien Gabriel & Nikita Prokopov

leader

follower

Synchronized state between tabs

Modern front-end

Expectation

Reality

Other problems of distributed systems

Problems
Firewalls and WS
Log cleaning

Scaling
Outdated clients
Deduplication

Solutions
Force WSS
Action’s reasons of life

Peer-to-peer protocol
Subprotocol verification
Action ID

Logux API

Chapter 4

API

“Talk is cheap. Show me the code.”

Linus Torvalds

Client: we care about client JS size

logux + redux ≈
12 KB

Client: Redux compatible API

import { createLoguxStore } from 'logux-redux'

const store = createLoguxStore(reducers, 
  preloadedState, undefined, {
  url:         'wss://logux.example.com/',
  userId:       user.id,
  subprotocol: '1.1.5',
  credentials:  user.token
})

Client: for “your friend” without Redux

import Client from 'logux-client/client'

const logux = new Client({
  url:         'wss://logux.example.com/',
  userId:       user.id,
  subprotocol: '1.1.5',
  credentials:  user.token
})

Client: local actions

dispatch({ type: 'MENU_OPEN' })

Client: cross-tab actions

add({
  type: 'READ_NOTIFICATION',
  id: notification
}, {
  reasons
})

Client: server actions

add({
  type: 'CHANGE_NAME',
  user, name
}, {
  sync: true,
  reasons
})

Client: instead of AJAX

showLoader()
fetch('/profile', {
  credentials: 'include',
  method: 'POST',
  form
}).then(r => {
  hideLoadaer()
}).catch(e => {
  …
})
add({
  type: 'CHANGE_NAME',
  user, name
}, {
  sync: true,
  reasons
})

Client: Optimistic UI by default

case 'CHANGE_NAME':
  return { name: action.name, ...user }

Optimistic UI

Image author: Denys Mishunov

Client: server could undo action on error

{
  type: 'logux/undo',
  id: [1489328408677, '10:bfj58s0w', 0],
  reason: 'error'
}

Logux Status: widget

Logux Status: favicon

Normal

Offline

Error

Logux Status: warning

Client: action storage

Local

Cached data

Not synchronized yet

Images author: Nikita Prokopov

Client: Pessimistic UI

Client: ask server for changes

add({ type: 'ASK_PAY' })

…

case 'ASK_PAY':
  return { payment: 'waiting' }

Client: wait for server answer

case 'PAID':
  return { payment: 'paid' }

Server: Node.js framework

import { Server } from 'logux-server'

const app = new Server({
  subprotocol: '1.2.0',
  supports: '1.x',
  root: __dirname
})

Server: actions handling

app.type('CHANGE_NAME', {
  access(action, meta, userId) {
    return userId === action.user
  }
  process(action, meta) {
    …
  }
})

Server: use any DB or send any requests

process(action, meta) {
  return db.user.id(action.user).then(u => {
    if (isFirstOlder(u.lastChange, meta)) {
      return u.changeName(action.name)
    }
  })
}

Server: send actions to client at any time

app.log.add({
  type: 'CHANGE_NAME',
  name: 'Looser',
  user: user.id
})

RESTful vs. Logux

Logux

RESTful

URL, HTTP method

action.type

POST form, GET query

Action object

Request happened now

Action could be

created in the past

Wrap legacy back-end (PHP, Ruby)

process({ user, name }) {
  return send('POST',
              `user/${ user }/change.php`,
              { name })
}

Logux Protocol is open

Backend languages plan

What is Logux?

Replaces AJAX and RESTful


Synchronizes logs (like Redux actions)
between client and server

Why Logux?

  • Less code
  • Live updates out of the box
  • Optimistic UI & Offline-first
  • Redux API on the front-end
  • Similar to RESTful on the back-end

Trade-offs

Chapter 4

Problem 1: Complexity

Complexity

App

Net

Logux

Cleaning

App

Right case for Logux

Complexity

App

Net

Logux

Cleaning

App

Problem 2: under construction

0.1 preview

protocol, core components

end-user API

channels, docs

0.2

autosubscribe, scaling

0.3

Amplifr already uses it in production

30 contributors

Want to reduce the code now?

Relay

Relay-like API for Logux is planned

Want to be Offline-first now?

Firebase

PouchDB

Gun.js

Want CRDT now?

Swarm.js

Questions?