A-8 — Redis Sentinel: Quorum, Failover, and Split-Brain Prevention

Who this module is for: You want Redis to automatically recover from a primary failure — promoting a replica and reconfiguring clients — without manual intervention. Redis Sentinel is the high-availability solution for single-node (non-Cluster) Redis deployments. This module covers the Sentinel model, the failover sequence, split-brain prevention, and Sentinel's limitations.

What Sentinel Is

Redis Sentinel is a separate process (not part of Redis itself) that monitors a Redis primary and its replicas. When the primary fails, Sentinel:

1. Detects the failure (agrees with other Sentinels that the primary is down)
2. Elects a leader Sentinel to run the failover
3. Selects the best replica to promote
4. Promotes the chosen replica to primary
5. Configures other replicas to replicate from the new primary
6. Notifies clients of the new primary address

Sentinel is not a proxy — it does not route traffic. It is a monitoring and orchestration layer that clients query to discover the current primary address.

Deployment Topology

A minimum Sentinel setup requires 3 Sentinel processes on separate machines. An odd number is required for quorum.

Clients connect to any Sentinel to discover the current primary's IP and port. They then connect directly to the primary for all operations.

Sentinel Configuration

The Failure Detection Sequence

Step 1: Subjective Down (SDOWN)

A single Sentinel marks the primary as "subjectively down" if it cannot reach the primary within down-after-milliseconds. This is one Sentinel's opinion — a network blip between just that Sentinel and the primary would cause an SDOWN that does not represent a real failure.

Step 2: Objective Down (ODOWN)

A Sentinel queries other Sentinels: "Do you also think the primary is down?" If at least quorum Sentinels agree, the primary is declared "objectively down" (ODOWN) — a real failure.

With quorum 2: at least 2 of 3 Sentinels must agree. This prevents a single Sentinel's network issue from triggering an unnecessary failover.

Step 3: Leader Election

One Sentinel must be elected to lead the failover. Sentinel uses a Raft-like election: each Sentinel requests votes from others. The first to receive a majority becomes the failover leader.

Step 4: Replica Selection

The leader Sentinel chooses which replica to promote. Selection criteria (in order of preference):

1. Replica with the lowest slave-priority (configured as replica-priority in replica's redis.conf)
2. Replica with the smallest replication lag (most up-to-date data)
3. Replica with the smallest Run ID (lexicographically) as tiebreaker

Step 5: Failover Execution

Step 6: Client Notification

Clients that use a Sentinel-aware client library (ioredis with sentinels config, Jedis with Sentinel support, etc.) subscribe to the __sentinel__:hello channel or periodically query Sentinel. When +switch-master fires, the client reconnects to the new primary address.

Failover Duration

A typical Sentinel failover takes:

• down-after-milliseconds (5 seconds default) to detect SDOWN
• ~1 second for ODOWN consensus
• ~1 second for leader election
• ~2–5 seconds for replica promotion and reconfiguration

Total: ~10 seconds of write downtime with default settings.

To reduce failover time: lower down-after-milliseconds (at the risk of false positives from brief network blips).

Split-Brain Prevention with min-replicas-to-write

Consider a network partition that isolates the primary from Sentinels but not from some clients:

Sentinels cannot reach the old primary → ODOWN → failover → Replica 1 promoted to new primary.

Meanwhile, the old primary is still accepting writes from the clients (they can still reach it). When the partition heals, the old primary reconnects as a replica of the new primary and loses all writes it accepted during the partition.

Prevention: min-replicas-to-write and min-replicas-max-lag on the primary:

During the partition, the old primary cannot reach any replica. After 10 seconds, it stops accepting writes. Clients receive errors instead of silently losing data.

ioredis Sentinel Client

For read replicas:

Sentinel CLI Commands

Sentinel Limitations

What Sentinel is not:

• It is not a proxy — clients connect directly to the primary, not through Sentinel
• It does not provide horizontal scaling — all writes go to one primary
• It does not protect against data loss during the replication lag window
• It cannot provide fencing tokens for distributed locking

Sentinel does not prevent data loss: If the primary fails before replication completes, writes in the lag window are lost when a replica is promoted. min-replicas-to-write reduces (but does not eliminate) this window.

Sentinel requires client support: Clients must be Sentinel-aware (know to query Sentinel for the primary address) or use a proxy (envoy, Twemproxy) that handles redirection. A client hardcoded to the primary's IP will not automatically reconnect after failover.

Sentinel vs Cluster

Use Sentinel when your dataset fits on a single Redis node and you need automatic failover without the operational complexity of Cluster. Use Cluster when you need horizontal scaling beyond what a single node can provide.

Summary

• Sentinel is a separate process that monitors Redis primary + replicas and orchestrates automatic failover
• SDOWN = one Sentinel's opinion; ODOWN = quorum agreement → triggers failover
• Failover sequence: detect ODOWN → elect leader Sentinel → select best replica → promote → reconfigure others → notify clients
• Configure quorum 2 with 3 Sentinels — majority agreement prevents false failovers from single-node network issues
• min-replicas-to-write 1 + min-replicas-max-lag 10 on the primary prevents split-brain data loss during network partition
• ioredis Sentinel client automatically discovers and reconnects to the new primary after failover
• Sentinel does not eliminate data loss — writes in the replication lag window are lost on primary failure
• Use Sentinel for single-node HA; use Cluster for horizontal scaling

Next: A-9 — Redis Cluster: Hash Slots and Data Distribution — the 16,384 hash slot model, key routing, MOVED vs ASK redirections, and the constraints multi-key commands impose in a Cluster.