Outline

• Goals
• Solutions available
• How and Why we chose the solution we did?
• Overview of Redis Sentinel
• Customized tunings for partition tolerance

Old Redis HA Setup

• We use master-slave replication for HA.
• Manual failover for redis master, if master goes down.

Problem we wanted to solve

• Automated failover of redis master (Redis master HA)
• Partition tolerance

Things to keep in mind for Redis HA

• Redis async replication - No strong consistency
• Automated switching is not built-in

Solutions explored

• Vanilla Redis cluster
• Vanilla Redis sentinel
• Haproxy + Smitty + Twemproxy + Redis Sentinel
• Redis sentinel with haproxy

Vanilla Redis cluster

• Supports automatic failover with builtin sharding
• Builtin redis support
• But requires client to be smart.

Vanilla Redis sentinel

• Supports automatic failover
• An external monitoring service to monitor redis state
• But requires client to be smart.

Haproxy + Twemproxy + Smitty + Redis Sentinel

• Too many moving parts

Redis Sentinel with haproxy

• Monitor service with redis sentinel
• Switch master/slaves in haproxy by monitoring sentinel,
• No client code changes required

A closer look at Redis Sentinel Service

• Monitoring service, utilizes quorum
• Triggers election if quorum subjectively decides that masters are down

A closer look at Redis Sentinel Service

• In failover scenario, a sentinel leader is selected to promote a slave to master
• Failover can't happen in minority partition, because majority is required for leader selection

But how would clients detect master failover?

• Bring haproxy to the mix

Haproxy master external check script

• Check state from sentinel
• Sentinel is in quorum?
• Who is the current master?
• Is it tcp reachable?
• If all checks pass, mark the node up

What we achived with this setup

• Clients can be dumb. All HA logic resides with haproxy.
• No code changes required. Any project in any language can use this setup.

Do we have partition tolerance?

• In minority partition, haproxy will mark master down as sentinel won't be in quorum
• There is still a small window where two redis master can be up, during failover
• We can reduce that window by tuning configs of various services involved

Haproxy tunables

Parameter	Value	Description
external-check command	<check-script>	external script used to perform check
inter	1s	Frequency of performing checks
rise	7	Haproxy will mark a node up after 7 consecutive check pass
fall	3	Haproxy will mark a node down after 3 consecutive check fail

Redis tunables

Parameter	Value	Description
min_slaves_to_write	1	Redis will stop accepting writes if `min-slaves-to-write` number of
min-slaves-max-lag	10	slaves are lagging by `min-slaves-max-lag` seconds.

Redis sentinel tunables

Parameter	Value	Description
down-after-milliseconds	5000	Mark objectively down if server is seen down
min-quorum	2	Redis will mark a server subjectively down if min-quorum sentinel services notice it down
failover-timeout	180000	No new failover will trigger be triggered before failover timeout
parallel_syncs	1	Number of redis salves can parallelly sync

Key takeaways

• You can achieve reasonably consistent, partition tolerant redis cluster with Haproxy + Sentinel
• Haproxy is a versatile piece of software, it's a devops swiss army knife.

Questions?