Messaging

The current state of component communication

  • core/ml: python bindings
  • core leader/follower: database with driver 
  •  core/edge: nodes js plugin, rpc calls through the node js plugin
  • core/argus: writing files on disk, sending metrics through rpc
  • leader c++/python: named pipe

=> Custom solution for each, complex, not scalable, hard to deploy

A possible solution: messaging

  • Decouples each components
    • They just need to worry about ins, outs and transform
    • Can all be written in different languages
    • Easy to test, monitor, modify, scale
    • Each component can run anywhere they want, just need a connection to the broker
  • Performance of those more than enough
  • One way to rule them (almost) all, simplifies maintenance
  • Needs broker process/machine(s)

Example workflow

SCT (no more node JS plugin)

Poi

Edge

Standalone

Broker

Example workflow

MCT (no more JS plugin, no more two leader processes)

Poi

Edge

Standalone

Broker
+
data

Poi

Standalone

Poi

Poi

Standalone

Leader

Poi

Poi

Poi

Nats (https://nats.io/)

  • Open source (licence Apache 2)
  • Lightweight and performant
  • Simple to use and setup (both for developers and ops)
  • Drivers in the languages we need (Node Js, C, Python, and more)
  • Comes out of the box with all the tools we need:
    • Transmit bytes
    • Any number of publishers and subscribers
    • Secure connection to server
  • And that we will need:
    • Scale
    • Persist data
    • ... 

Messaging

By technology

Made with Slides.com

Messaging

  • 51

More from technology