Web and Cloud Computing

Architecture Overview

Group 13

Contents

  1. Architectural Overview
  2. The Seascape Wave Agent
  3. The central elixir application
  4. Conclusion

High-Level Architecture

Architecture Overview

  1.  Agent 'Seascape Wave'.
  2.  Ingest listening/computing nodes.
  3.  Analytics-focused distributed database (ElasticSearch).
  4.  Web user-interaction nodes.

Four key components:

Architecture Diagram

"Seascape Wave" Agent

Contents

  1. Architectural Overview
  2. The Seascape Wave Agent
  3. The central elixir application
  4. Conclusion

Agent: 'Seascape Wave'

Agent: 'Seascape Wave'

  • Runs on a centralized machine where metrics can be aggregated.
  • We want to construct the system to easily allow adding other agents in the future (might run inside environments without access to the host machine).
    • Start with saltstack
    • Allow for further expansion

Agent: 'Seascape Wave'

  • Built using Python/SaltStack
    • Built-in cryptography
      • Verified identities
      • Encrypted communication
    • Built-in message queue
  • Runs on the 'Salt master' which manages the container hosts (minions):
    • Minions can be configured by the agent to send metrics to the master. The agent can then format and send them to the ingest nodes.
    • Agent must have access to the machine hosting the containers.

Elixir Systems

Contents

  1. Architectural Overview
  2. The Seascape Wave Agent
  3. The central elixir application
  4. Conclusion

Ingest nodes

  • Built using Elixir.
  • Part of the BEAM cluster (see later slide).
  • Horizontally scalable by running many behind a load-balancer (distributing sessions in a round-robin fashion).
  • Listens for a data from an Agent.

Analytics-focused distributed database

  • For this task we've chosen to use Elasticsearch.
    Reasons:
    • Built to work with metric-based data like what we have in this project.
    • Built to scale to large mountains amounts of data.
    • Built to refine the interpretation of data (e.g what kind of queries you might want to do) later.

Web nodes: construction

  • Built using Elixir/Phoenix LiveView.
    • A user connects to it with their browser.
    • LiveView serves a static HTML page containing the SPA on first visit which then starts up a WebSocket connection through which all later interaction takes place:
      • DOM-diffing happens server-side, simplifying implementation.
    • Does not interfere with SEO, for example
  • Part of the BEAM cluster (see later slide).

Web nodes: scaling

  • Part of the BEAM cluster (see later slide).
  • Horizontally scalable by running many behind a load-balancer (distributing sessions in a round-robin fashion).
  • Requests data from the database whenever required, caches data locally.

The BEAM cluster: actors

  • Elixir works using the Actor model, allowing for a scalable and fault-tolerant implementation of concurrent services that is comprehensible and therefore maintainable.

The BEAM cluster: actors

  • Ingest and Web nodes are connected to the same Erlang/BEAM cluster, enabling:
    • Transparent message-passing, in particular:
    • Transparent pub-sub for real-time data updates which can be sent without going through the database.
    • CRDT-based real-time information about what agents (e.g. the user's clusters and containers within them) are currently 'up' and their health.

The BEAM cluster

  • On top of this, we run the distributed database Mnesia on all nodes in the cluster, to enable:
    • Persistent user sessions (without complicating the load balancers).
    • Storage of ephemeral (e.g. current 'real-time' data).

Conclusion

Contents

  1. Architectural Overview
  2. The Seascape Wave Agent
  3. The central elixir application
  4. Conclusion

General Remarks

  • For our particular project, there is no functionality we can provide to the user when they are offline. (Except telling them that they are disconnected and that the current metrics/graphs thus are stale.
    • This in part influenced our design decisions.
  • We are not adding extra 'message queues' (like e.g. RabbitMQ) to our stack, besides the message queue that is part of the Agent (internally in Saltstack in our case).
    • Of course, all Elixir processes (actors) have their own internal message queue ('mailbox').
      Fun fact: RabbitMQ is itself written in Erlang and thus uses these techniques under the hood
  • We are not adding external caches/message brokers (like e.g. Redis) to our application. Instead, we use direct message passing instead of an external message broker and Mnesia for persistent user sessions and ephemeral caching.

Web and Cloud Computing: Group 13

By qqwy

Web and Cloud Computing: Group 13

  • 254
Loading comments...

More from qqwy