Consensus algorithms in distributed systems

Piotr Grzesik

What is consensus ?

Reaching an agreement between processes and proposed values. 

 

Consensus is fundamental for solving problems like leader election, commiting a transaction to database or performing clock synchronisation.

Potential problems

  • Network partitions
  • Lost or duplicated messages
  • Processes crashing
  • Byzantine failures
  • Storage failure

Properties of consensus protocols

  • Agreement - final decision of all non-faulty processes must be the same  
  • Termination - each non-faulty process must eventually decide on a value
  • Validity - final decision must be one of the values proposed by the processes (no default values)

Consensus protocols

  • 2 phase commit
  • 3 phase commit
  • Paxos
  • Raft

2 Phase commit

Simple approach to solving consensus problem. Constitutes of two distinct phases: Propose and Commit/abort

2 Phase commit - propose

2 Phase commit - commit

Potential failures

  • Coordinator fails before propose phase
  • Coordinator fails during propose phase
  • Participant fails during propose phase
  • Participant fails during commit phase
  • Coordinator fails during commit phase

3 Phase commit

  • Improvement over 2PC at the cost of extra messages
  • Splits commit phase into prepare-to-commit and commit phases
  • Recovery node can take over in cases of coordinator crashing
  • Can sustain failure of a participant that commited/aborted transaction (all other participants know the outcome)
  • Prone to network partitioning

3 Phase commit

https://en.wikipedia.org/wiki/Three-phase_commit_protocol#/media/File:Three-phase_commit_diagram.png

Raft

  • Proposed by Diego Ongaro in his paper "In Search of an Understandable Consensus Algorithm"
  • Built, as easier to understand alternative to (multi)Paxos
  • Manages replicated log (replicated state machine)
  • Separates leader election and log replication
  • To operate needs at least n/2 +1 nodes (for 3 node cluster, can tolerate failure of 1 node)
  • Time divided into terms (change of leader -> change of term)

Replicated state machine

Possible server states

  • Leader - clients communicate with leader, responsible for log replication
  • Follower - accept replicated logs from leader
  • Candidate - candidate for a new leader during election

Used RPCs

  • RequestVote - used by candidates during election
  • AppendEntries - used by leaders to replicate log entries or serves as a heartbeat
  • (optional) InstallSnapshot - used by a leader to sent log snapshot to follower

Leader election

  • Increment term
  • Switch to Candidate
  • Vote for itself
  • Send RequestVote RPC call to all other participants
  • Become leader if received majority of votes
  • Become follower if received AppendEntries from valid leader
  • Start new election if no leader emerges (split vote)

Log

  • Each participant stores log entries, which consist of log index, term and value (command)
  • Logs are stored in a durable way
  • Entry is commited if it was durably stored on majority of the participants
  • No empty fields in log - if a given entry is commited all preceding entries are also commited

Log replication

  • Client communicates with a leader(or follower and gets redirected to leader) and sends a command
  • Leader appends the command to it's own log
  • Leader sends AppendEntries RPC call to all followers
  • On receiving responses from majority, leader passes command to it's state machine and returns result to the client
  • Leader notifies followers of commited entries
  • Followers pass command to their's state machines

Safety

  • If log entry command has been applied to state machine, no other state machine can apply different command for that log entry
  • Entries must be commited before applying to state machine
  • Leader never overwrite entries in its log
  • If entry is commited, it will be present in all future leader's logs
  • Leader might order follower to overwrite its entries to 'fix' log

Usage

  • Consul (https://www.consul.io/)
  • etcd (https://github.com/etcd-io/etcd)
  • rqlite (https://github.com/rqlite/rqlite)
  • Apache Ratis (http://ratis.incubator.apache.org/)

Demo

Sources

  1. https://www.the-paper-trail.org/post/2008-11-27-consensus-protocols-two-phase-commit/
  2. In Search of an Understandable Consensus Algorithm https://raft.github.io/raft.pdf
  3. https://www.the-paper-trail.org/post/2008-11-29-consensus-protocols-three-phase-commit/
  4. An Introduction to Raft (CoreOS Fest 2015) - https://www.youtube.com/watch?v=6bBggO6KN_k
  5. http://thesecretlivesofdata.com/raft
Made with Slides.com