Story time 

Stock Exchange & Data processing

• Collect data and metrics from devices and other markets
• Analyze the collected data and provide information for customers
• Serve real time data to the customers.

Our Requirements

(General goals)

Scalable

• Scale up and down based on work load
• Scale up based on latency 

Fault tolerance 

• Against network failures
• Against network partitions
• Against human errors
• Against hardware failures
• etc

Consistancy

• Reliable and correct information
• Serve most recent data or nothing

CAP theorom and availability 

User's decision

Extendable 

• Easy to understand
• Easy to reason about
• Maintainable 
• As Simple as possible

Couple of other products with the same structure

That got me thinking...

Implementation

Current Implementation

HellHound = Clients + Web Cluster

Clients

• Client as a component of system
• It's different than other components
• Re-frame application
• Websocket connection 
• Fallback to ajax ( removed )
• A re-frame event which populates the app-db
• Lots of subscribers

Why Websockets ?

• Low latency
• Small packet size
• Browser support
• Tooling and libraries

Why Not ?

Asset caching

Websocket Alternatives

• QUIC
• HTTP 2.0
• SSE
• SPDY
• WebRTC data channels

Immutant Workers

• Should be Stateless ( websocket )
• Ring + Immutant
•  Immutant websocket server
• System 
• Event Router

Kafka

• Task Streams
• Info Steams

Onyx Worker

• Stateless
• Lightweight

HellHound

Another web framework ????

Seriously ???

Why bother ? 

• Extract a framework or library from the product
• Improve the framework for future products

Ultimate goals

• FullStack
• Async
• Distributed
• Stream Processing
• Collection of smaller subsystems
• As simple as possible

Server Side

Problems & Improvements

• Messaging Abstraction ( Manifold )
• Workflow
• Communication Abstraction
• Server <-> Client events and event router
• Back Pressure
• Erlang Style Error handling per node
• Environments

Changes

• Aleph
• Manifold

Other stuff which I tested

• Pedestal
• Yada

Client Side

Problems & Improvements

• Data router  ( Ditch the reducers )
• Query Language
• App-db index 
• Server <-> Client events
• Service Workers
• Web Worker support
• Caching 
• Preact and other minor tweaks 
• Environments