GowayFest

Alexander Emelin, 10.2018

https://slides.com/fz/cc

Centrifugo – building language-agnostic real-time messaging server in Go. And one more Centrifuge

Centrifugo

https://github.com/centrifugal/centrifugo

Real-time messages

Real-time solutions

https://www.leggetter.co.uk/real-time-web-technologies-guide/

https://deepstreamhub.com/blog/realtime-framework-overview/

Primus

Firebase

Meteor

Phoenix

pusher.com

pubnub.com

crossbar.io

Atmosphere

socket.io

Faye

Horizon

deepstreamhub

emitter.io

feathers.js

SignalR

Pushpin

SocketCluster

Nchan

Initial motivation

Simplified Centrifugo scheme

Features

  • Language-agnostic
  • Decoupled from application backend
  • JWT authentication with expiration support
  • Simple integration with existing application
  • Scale horizontally with Redis
  • Solid performance
  • Message recovery after short disconnects
  • Information about active subscribers in channel
  • Cross platform (Linux, MacOS, Windows)
  • MIT license
  • Works everywhere (browser, mobile)
  • Works over same ports as HTTP
  • Low-overhead
  • Bidirectional

Websocket advantages

Websocket

Actually not that bad anymore

  • xhr-streaming 👍
  • eventsource 👍
  • iframe-eventsource
  • iframe-htmlfile
  • xhr-polling
  • iframe-xhr-polling
  • jsonp-polling

SockJS fallbacks

js

Websocket and HTTP/2

https://tools.ietf.org/html/draft-ietf-httpbis-h2-websockets-07

Running the WebSocket Protocol (RFC 6455) over a single stream of an HTTP/2 connection:

Internals

Client-to-server transports

Websocket – https://github.com/gorilla/websocket

SockJS – https://github.com/igm/sockjs-go

GRPC bidirectional streaming

Websocket vs GRPC streaming:

Websocket GRPC streaming
Server memory usage with 10k clients ~ 500 mb ~ 2000 mb
Cpu usage under the same load ~ 30% ~ 85%
Traffic over interface in same scenario ~ 17 mb ~ 19 mb

Protocol

  • Custom
  • Defined in protocol buffers schema
  • Looks similar to JSON RPC 2.0 on high level
  • Similar to MQTT on business level (but no QOS)
  • Two serialization formats: JSON and Protobuf

Protobuf vs JSON

BenchmarkJsonMarshal-8            500000    2980 ns/op    1223 B/op   9 allocs/op
BenchmarkJsonUnmarshal-8          500000    3120 ns/op    463 B/op    7 allocs/op


BenchmarkProtobufMarshal-8        2000000    901 ns/op    200 B/op    7 allocs/op
BenchmarkProtobufUnmarshal-8      2000000    692 ns/op    192 B/op   10 allocs/op

https://github.com/alecthomas/go_serialization_benchmarks

JSON: {"type":"ping","time":123456789} = 32 bytes

Protobuf: 0A 05 08 95 9A EF 3A = 7 bytes

Engine

  • Responsible for PUB/SUB mechanics
  • Manages history cache
  • Manages presence information

Built-in engines

  • In memory
  • Redis Engine – thanks again Gary Burd

Message delivery model

At most once

+ with recovery feature

Optimizations

  • Gogoprotobuf library for protocol buffers

Gogoprotobuf

BenchmarkJsonMarshal-8            500000    2980 ns/op    1223 B/op   9 allocs/op
BenchmarkJsonUnmarshal-8          500000    3120 ns/op    463 B/op    7 allocs/op


BenchmarkProtobufMarshal-8        2000000    901 ns/op    200 B/op    7 allocs/op
BenchmarkProtobufUnmarshal-8      2000000    692 ns/op    192 B/op   10 allocs/op


BenchmarkGogoprotobufMarshal-8    10000000   152 ns/op    64 B/op     1 allocs/op
BenchmarkGogoprotobufUnmarshal-8  10000000   221 ns/op    96 B/op     3 allocs/op

https://github.com/gogo/protobuf

Optimizations

  • Gogoprotobuf library for protocol buffers
  • Reduce syscalls by merging messages 
BenchmarkMerge-4     20000    75249 ns/op    69170 B/op    63 allocs/op
BenchmarkNoMerge-4   10000   193548 ns/op    63568 B/op    213 allocs/op

Reducing syscalls

http://www.sureshjoshi.com/development/streaming-protocol-buffers/

https://en.wikipedia.org/wiki/JSON_streaming

{"id": 1, "method": "subscribe", "params": {"channel": "ch1"}}
{"id": 2, "method": "subscribe", "params": {"channel": "ch2"}}

JSON streaming

Varint-prefixed Protobuf

Optimizations

  • Gogoprotobuf library for protocol buffers
  • Reduce syscalls by merging messages 
  • sync.Pool to reuse buffers
  • Redis pipelining with smart batching technique

Smart batching

	subCh := make(chan string, 128)
	for i := 0; i < 128; i++ {
		subCh <- "channel"
	}

	maxBatchSize := 50

	for {
		select {
		case channel := <-subCh:
			batch := []string{channel}
		loop:
			for len(batch) < maxBatchSize {
				select {
				case channel := <-subCh:
					batch = append(batch, channel)
				default:
					break loop
				}
			}
			println(len(batch))
		}
	}

https://play.golang.org/p/4yOib4Hq6Z1

> go run main.go
50
50
28

Optimizations

  • Gogoprotobuf library for protocol buffers
  • Reduce syscalls by merging messages 
  • sync.Pool to reuse buffers
  • Redis pipelining with smart batching technique
  • Save RTT amount with Redis using Lua scripts
  • Batch messages on client side

Prometheus

(+ export to Graphite)

Metrics

Dependency management

dep – https://github.com/golang/dep

Goreleaser

https://github.com/goreleaser/goreleaser

Release management

Centrifuge library

https://github.com/centrifugal/centrifuge

Beyond Centrifugo

  • Native authentication with middleware
  • Tight integration with business logic
  • Bidirectional messaging
  • Built-in RPC calls
  • More freedom in channel permission management 
package main

import (
    "context"
    "log"
    "net/http"

    cent "github.com/centrifugal/centrifuge"
)

func main() {
    cfg := centrifuge.DefaultConfig
    cfg.ClientInsecure = true
    cfg.Publish = true

    node, _ := centrifuge.New(cfg)

    node.On().Connect(func(ctx context.Context, cl *cent.Client, e cent.ConnectEvent) cent.ConnectReply {
        cl.On().Subscribe(func(e cent.SubscribeEvent) cent.SubscribeReply {
            log.Printf("client subscribes on channel %s", e.Channel)
            return cent.SubscribeReply{}
        })

        cl.On().Publish(func(e cent.PublishEvent) cent.PublishReply {
            log.Printf("client publishes into channel %s: %s", e.Channel, string(e.Data))
            return cent.PublishReply{}
        })

        cl.On().Disconnect(func(e cent.DisconnectEvent) cent.DisconnectReply {
            log.Printf("client disconnected")
            return cent.DisconnectReply{}
        })

        log.Println("client connected")
        return cent.ConnectReply{}
    })

    if err := node.Run(); err != nil {
        panic(err)
    }

    http.Handle("/connection/websocket", cent.NewWebsocketHandler(node, cent.WebsocketConfig{}))

    go func() {
        if err := http.ListenAndServe(":8000", nil); err != nil {
            panic(err)
        }
    }()
}

How it feels

node.On().Connect(func(
       ctx context.Context,
       client *Client, 
       e ConnectEvent
   ) ConnectReply {

   client.On().Subscribe(func(e SubscribeEvent) SubscribeReply {
       log.Printf("client subscribes on channel %s", e.Channel)
       return SubscribeReply{}
   })

   client.On().Publish(func(e PublishEvent) PublishReply {
       log.Printf("client publishes into channel %s: %s", e.Channel, e.Data)
       return PublishReply{}
   })
 
   client.On().Disconnect(func(e DisconnectEvent) DisconnectReply {
       log.Printf("client disconnected")
       return DisconnectReply{}
   })

   log.Println("client connected")
   return ConnectReply{}
})

How it feels with zoom

Clients

Some examples

PUB/SUB

var centrifuge = new Centrifuge("ws://example.com/websocket");

centrifuge.subscribe("channel", function(message){
    console.log(message);
});

centrifuge.connect();

Bidirectional messaging

var centrifuge = new Centrifuge("ws://example.com/websocket");

centrifuge.on("connect", function(){
    centrifuge.send({"text": "hello"});
});

centrifuge.on("message", function(data) {
    console.log(data);
});

centrifuge.connect();

RPC

var centrifuge = new Centrifuge("ws://example.com/websocket");

centrifuge.rpc({"method": "say_hi"}).then(function(data){
    console.log(data);
});

centrifuge.connect();

Links

Questions?

Centrifugo – building language-agnostic real-time messaging server. And one more Centrifuge

By Emelin Alexander

Made with Slides.com

Centrifugo – building language-agnostic real-time messaging server. And one more Centrifuge

  • 2,733

More from Emelin Alexander