Externalizing the cluster membership
Baosong@SR-Intelligence
2023-01-17
Agenda
- Background/Purpose
- Requirements
- The proposed architecture/deployment
- Roadmap
- Demo
Background/Purpose
- It is the very step to make the server stateless.
- Each node talks to other nodes for state synchronization resulting in O(n2) complexity, which does not scale.
- Each Library calls each I-server to synchronize cluster membership to the Library. The consolidation logic in Library is complicated and fragile.
What is cluster membership?
- The nodes within the cluster
- For each node
- The node state
- The job count
- The session count
- The loaded projects
- For each project
- The project state
- The users who logged in to the project
Requirements
- Single truth of cluster state.
-
Simplify the synchronization between server nodes.
-
The server nodes should join automatically when they start up.
-
Capable of switching between the two modes - cluster state externalized vs. cluster state within each server node.
The proposed architecture
Roadmap
- Finish stage 1 work by
- Merge the server-side code change
- Work with PL team to deploy the Cluster State Service in the containerized environment
- Move from stage1 to stage2
- Externalizing the session state
- Externalizing the cube index
- ...