Membership Changes

Evolving the cluster

Current State

How we handle peer changes.

PD

  • (Remove) Receives a command from pd-ctl.
  • (Add) Receives a command from a TiKV peer.
  • Sends instructions to each TiKV/Raft node.
  • If a new node is added, it is inserted as a learner.
message Message {
    MessageType msg_type = 1;
    ...
    repeated Entry entries = 7;
    ...
}

enum EntryType {
    EntryNormal = 0;
    EntryConfChange = 1;
}

TiKV

  • Receives instruction from pd.
  • Informs tooling/instrumentation as needed.
  • Relays message to raft.

Raft

  • Receives command via TiKV.
  • Forcibly takes action without coordination.
/// Takes the conf change and applies it.
pub fn apply_conf_change(&mut self, cc: &ConfChange) -> ConfState {
    // ...
    match cc.get_change_type() {
        ConfChangeType::AddNode => self.raft.add_node(nid),
        ConfChangeType::AddLearnerNode => self.raft.add_learner(nid),
        ConfChangeType::RemoveNode => self.raft.remove_node(nid),
    }
    // ...
}

pub fn remove_node(&mut self, id: u64) {
    self.mut_prs().remove(id);
    // ...
}

Not
Joint
Consensus

Not Good Enough:

How the current system can fail.

Example of How It Fails

Add, then Remove:

 

IDC 1 fails before Remove, quorum fails, cluster pauses until IDC 1 is back up.

Remove, then Add:

 

IDC 2 fails before Add, leaving only C. Quorum fails, cluster pauses.

Nodes:  A  B  C  D    Current: ABC
IDCS:   1  2  3  1    To Replace: A with D

There are others...

Lurking in the shadows.

We can stop them.

Joint Consensus

Problem

Adding and removing Raft peers offers several potential problems.

 

Primarily:

How can pauses in the cluster be prevented?

Solution

The Raft paper describes  a process called Joint Consensus.

 

TL;DR:

It involves using a union of both the old peer set and the new peer set temporarily.

A Corner with a Joint

We allow the leader to receive a command to change the peer set.

 

It enters the union state, then distributes the log message to followers.

Each of them then enter the union state as they learn of it.

 

After the union is committed (and learners caught up) the new configuration is applied the same way.

C_{(old)} \rightarrow C_{(old \bigcup new) } \rightarrow C_{(new)}
C(old)→C(old⋃new)→C(new)C_{(old)} \rightarrow C_{(old \bigcup new) } \rightarrow C_{(new)}

The Joint Consensus State

Log entries are still replicated to all servers.

 

Agreement for both elections and entry commits requires separate majorities from both C(new) and C(old) clusters.

Still services requests.

 

Individual peers transition at different times.

 

No leadership restrictions

Other Problems

Solved by the Joint

Catching Up New Nodes

If a node is added without any log entries it may take quite some time before it catches up, which could potentially render the cluster unable to make progress.

 

This is resolved via the learner state, when nodes in this state are not considered for majorities.

 

Leader not in C(new)

It is possible that the peer which is currently leader in C(old) will not be part of C(new). This means at some point during C(old, new) the leader will be managing a cluster it is not a part of until the C(new) entry is committed.

 

During this time it replicates log entries, but does not count itself in majorities. When C(new) is committed a leadership transfer can occur.

 

Removed Peers are Disruptive

When a server is removed they will time out, and try to start a new election with a new term. Though this will cause a new election.

 

In order to prevent this, peers will not service RequestVote RPCs they receive within the minimum election timeout of hearing from their current leader.

Moving Forward

Joint Consensus for Us

Raft

  • Add a new variant: ConfChangeType::SetNodes.
  • Have ConfChange hold a ConfState.
  • Add logic to calculate C(old, new) from C(old) and C(new).
  • Add mechanism to append C(new) when a C(old, new) is committed and any added learners are promoted.
  • Add way to remember the intended C(new).
  • Design way to react to ConfChange at the appropriate time.

Raft

message ConfState {
    repeated uint64 nodes = 1;
    repeated uint64 learners = 2;
}

enum ConfChangeType {
    AddNode    = 0;
    RemoveNode = 1;
    AddLearnerNode = 2;
    SetNodes = 3;
}

message ConfChange {
    uint64 id = 1;
    ConfChangeType change_type = 2;
    // Used for add/remove
    uint64 node_id = 3;
    bytes context = 4;
    // Used for set
    ConfState new_state = 4;
}

TiKV

  • Validate new Raft commands are not obstructed.
  • Add new metrics.
  • (Optional) Add configuration change to tikv-ctl.

PD

  • Migrate to calling new SetNodes command instead of AddNode and RemoveNode.
Made with Slides.com