P2P: Consistent Hashing

• Need to be able to add and remove nodes without downtime
• Solution: Map each node to a range in the hash's output space
• Each node is assigned a random location on the ring
  • Handles all keys up to the next node in the ring

P2P: Virtual Nodes

• Problems:
  • Not usually a uniform distribution, without LLN
  • Fails to support heterogeneity

• Solution: Virtual Nodes
  • Each physical machine is treated as a configurable number of virtual nodes
  • Distributes the key space handled by a physical node
    • Great for failures

P2P: REPLICATION

• Data is replicated across N successor nodes
  • This forms the "preference list"
• Effectively, each node is actually responsible for the range of keys belonging to all N-1 predecessors 

• Problem: Replication needs to occur across physical not virtual nodes
  • Preference list is constructed by skipping nodes to ensure physical replication

Failures: TEMPORARY

• Nodes fail all the time in a cloud environment, for a variety of reasons
  • Oftentimes are only temporarily offline
• When a node is unavailable, another node can accept the write
  • Enables "always-writeable" system
    • (Assuming 1+ nodes!)
• The write is stored in a local buffer
  • Applied once the network heals

Failures: PERMANENT

• Problem: Need to know when a node goes offline
  • Heartbeats are expensive (n^2!)
• Gossip Protocol
  • Every node tracks the overall cluster state
  • At every tick, randomly choose another node in the ring
    • Two nodes sync their cluster states
    • Leads to eventual failure propagation

CONFLICT RESOLUTION

• For an "always-writeable" system, conflict resolution cannot be done at write-time.
  • Dynamo uses read-time reconciliation instead
• Weak consistency guarantees -> divergent object versions

CONFLICT RESOLUTION: Vector CLocks

• Dynamo uses vector clocks
  • Avoids walk-clock skew issues
• VC is a list of tuples:
  • [node, counter]
• Each object version is immutable
• "Application-assisted conflict resolution"

CONFLICT RESOLUTION: Replica SYNCHRONIZATION

• Problem: Recovering from a permanent failure or a network partition
  • Can be very expensive with significant divergence
• Merkel trees simplify the process of figuring out which parts of a key-range are different.
  • Each leaf is a hash of a key
  • Minimizes data exchange, too!
• Used to sync divergent replicas in the background.