Centrifugo – building language-agnostic real-time messaging server. And one more Centrifuge

GowayFest

Alexander Emelin, 10.2018

https://slides.com/fz/cc

Centrifugo – building language-agnostic real-time messaging server in Go. And one more Centrifuge

Centrifugo

https://github.com/centrifugal/centrifugo

Real-time messages

Real-time solutions

https://www.leggetter.co.uk/real-time-web-technologies-guide/

https://deepstreamhub.com/blog/realtime-framework-overview/

Primus

Firebase

Meteor

Phoenix

pusher.com

pubnub.com

crossbar.io

Atmosphere

socket.io

Faye

Horizon

deepstreamhub

emitter.io

feathers.js

SignalR

Pushpin

SocketCluster

Nchan

Initial motivation

Simplified Centrifugo scheme

Features

Language-agnostic
Decoupled from application backend
JWT authentication with expiration support
Simple integration with existing application
Scale horizontally with Redis
Solid performance
Message recovery after short disconnects
Information about active subscribers in channel
Cross platform (Linux, MacOS, Windows)
MIT license

Works everywhere (browser, mobile)
Works over same ports as HTTP
Low-overhead
Bidirectional

Websocket advantages

Websocket

Actually not that bad anymore

xhr-streaming 👍
eventsource 👍
iframe-eventsource
iframe-htmlfile
xhr-polling
iframe-xhr-polling
jsonp-polling

SockJS fallbacks

Websocket and HTTP/2

https://tools.ietf.org/html/draft-ietf-httpbis-h2-websockets-07

Running the WebSocket Protocol (RFC 6455) over a single stream of an HTTP/2 connection:

Internals

Client-to-server transports

Websocket – https://github.com/gorilla/websocket

SockJS – https://github.com/igm/sockjs-go

~~GRPC bidirectional streaming~~

Websocket vs GRPC streaming:

	Websocket	GRPC streaming
Server memory usage with 10k clients	~ 500 mb	~ 2000 mb
Cpu usage under the same load	~ 30%	~ 85%
Traffic over interface in same scenario	~ 17 mb	~ 19 mb

Protocol

Custom
Defined in protocol buffers schema
Looks similar to JSON RPC 2.0 on high level
Similar to MQTT on business level (but no QOS)
Two serialization formats: JSON and Protobuf

Protobuf vs JSON

BenchmarkJsonMarshal-8            500000    2980 ns/op    1223 B/op   9 allocs/op
BenchmarkJsonUnmarshal-8          500000    3120 ns/op    463 B/op    7 allocs/op


BenchmarkProtobufMarshal-8        2000000    901 ns/op    200 B/op    7 allocs/op
BenchmarkProtobufUnmarshal-8      2000000    692 ns/op    192 B/op   10 allocs/op

https://github.com/alecthomas/go_serialization_benchmarks

JSON: {"type":"ping","time":123456789} = 32 bytes

Protobuf: 0A 05 08 95 9A EF 3A = 7 bytes

Engine

Responsible for PUB/SUB mechanics
Manages history cache
Manages presence information

Built-in engines

In memory
Redis Engine – thanks again Gary Burd

Message delivery model

At most once

+ with recovery feature

Optimizations

Gogoprotobuf library for protocol buffers

Gogoprotobuf

BenchmarkJsonMarshal-8            500000    2980 ns/op    1223 B/op   9 allocs/op
BenchmarkJsonUnmarshal-8          500000    3120 ns/op    463 B/op    7 allocs/op


BenchmarkProtobufMarshal-8        2000000    901 ns/op    200 B/op    7 allocs/op
BenchmarkProtobufUnmarshal-8      2000000    692 ns/op    192 B/op   10 allocs/op


BenchmarkGogoprotobufMarshal-8    10000000   152 ns/op    64 B/op     1 allocs/op
BenchmarkGogoprotobufUnmarshal-8  10000000   221 ns/op    96 B/op     3 allocs/op

https://github.com/gogo/protobuf

Optimizations

Gogoprotobuf library for protocol buffers
Reduce syscalls by merging messages

BenchmarkMerge-4     20000    75249 ns/op    69170 B/op    63 allocs/op
BenchmarkNoMerge-4   10000   193548 ns/op    63568 B/op    213 allocs/op

Reducing syscalls

http://www.sureshjoshi.com/development/streaming-protocol-buffers/

https://en.wikipedia.org/wiki/JSON_streaming

{"id": 1, "method": "subscribe", "params": {"channel": "ch1"}}
{"id": 2, "method": "subscribe", "params": {"channel": "ch2"}}

JSON streaming

Varint-prefixed Protobuf

Optimizations

Gogoprotobuf library for protocol buffers
Reduce syscalls by merging messages
sync.Pool to reuse buffers
Redis pipelining with smart batching technique

Smart batching

	subCh := make(chan string, 128)
	for i := 0; i < 128; i++ {
		subCh <- "channel"
	}

	maxBatchSize := 50

	for {
		select {
		case channel := <-subCh:
			batch := []string{channel}
		loop:
			for len(batch) < maxBatchSize {
				select {
				case channel := <-subCh:
					batch = append(batch, channel)
				default:
					break loop
				}
			}
			println(len(batch))
		}
	}

https://play.golang.org/p/4yOib4Hq6Z1

> go run main.go
50
50
28

Optimizations

Gogoprotobuf library for protocol buffers
Reduce syscalls by merging messages
sync.Pool to reuse buffers
Redis pipelining with smart batching technique
Save RTT amount with Redis using Lua scripts
Batch messages on client side

Prometheus

(+ export to Graphite)

Metrics

Dependency management

dep – https://github.com/golang/dep

Goreleaser

https://github.com/goreleaser/goreleaser

Release management

Centrifuge library

https://github.com/centrifugal/centrifuge

Beyond Centrifugo

Native authentication with middleware
Tight integration with business logic
Bidirectional messaging
Built-in RPC calls
More freedom in channel permission management

package main

import (
    "context"
    "log"
    "net/http"

    cent "github.com/centrifugal/centrifuge"
)

func main() {
    cfg := centrifuge.DefaultConfig
    cfg.ClientInsecure = true
    cfg.Publish = true

    node, _ := centrifuge.New(cfg)

    node.On().Connect(func(ctx context.Context, cl *cent.Client, e cent.ConnectEvent) cent.ConnectReply {
        cl.On().Subscribe(func(e cent.SubscribeEvent) cent.SubscribeReply {
            log.Printf("client subscribes on channel %s", e.Channel)
            return cent.SubscribeReply{}
        })

        cl.On().Publish(func(e cent.PublishEvent) cent.PublishReply {
            log.Printf("client publishes into channel %s: %s", e.Channel, string(e.Data))
            return cent.PublishReply{}
        })

        cl.On().Disconnect(func(e cent.DisconnectEvent) cent.DisconnectReply {
            log.Printf("client disconnected")
            return cent.DisconnectReply{}
        })

        log.Println("client connected")
        return cent.ConnectReply{}
    })

    if err := node.Run(); err != nil {
        panic(err)
    }

    http.Handle("/connection/websocket", cent.NewWebsocketHandler(node, cent.WebsocketConfig{}))

    go func() {
        if err := http.ListenAndServe(":8000", nil); err != nil {
            panic(err)
        }
    }()
}

How it feels

node.On().Connect(func(
       ctx context.Context,
       client *Client, 
       e ConnectEvent
   ) ConnectReply {

   client.On().Subscribe(func(e SubscribeEvent) SubscribeReply {
       log.Printf("client subscribes on channel %s", e.Channel)
       return SubscribeReply{}
   })

   client.On().Publish(func(e PublishEvent) PublishReply {
       log.Printf("client publishes into channel %s: %s", e.Channel, e.Data)
       return PublishReply{}
   })
 
   client.On().Disconnect(func(e DisconnectEvent) DisconnectReply {
       log.Printf("client disconnected")
       return DisconnectReply{}
   })

   log.Println("client connected")
   return ConnectReply{}
})

How it feels with zoom

Clients

Some examples

PUB/SUB

var centrifuge = new Centrifuge("ws://example.com/websocket");

centrifuge.subscribe("channel", function(message){
    console.log(message);
});

centrifuge.connect();

Bidirectional messaging

var centrifuge = new Centrifuge("ws://example.com/websocket");

centrifuge.on("connect", function(){
    centrifuge.send({"text": "hello"});
});

centrifuge.on("message", function(data) {
    console.log(data);
});

centrifuge.connect();

RPC

var centrifuge = new Centrifuge("ws://example.com/websocket");

centrifuge.rpc({"method": "say_hi"}).then(function(data){
    console.log(data);
});

centrifuge.connect();

Links

Questions?