How cluster nodes propagate topology changes via PING/PONG gossip, cluster-node-timeout in failure detection, split-brain handling with cluster-require-full-coverage, and when a Cluster sacrifices availability for consistency.
A-11 — Gossip Protocol and Network Partition Handling
Who this module is for: You run a Redis Cluster and want to understand how nodes discover each other, propagate topology changes, and handle scenarios where part of the cluster becomes unreachable. The gossip protocol is the nervous system of Redis Cluster — understanding it explains why certain failure modes occur and how to configure the cluster to balance availability against consistency.
The Gossip Protocol
In Redis Cluster, every node knows about every other node. There is no central registry. Instead, nodes maintain this knowledge through a gossip protocol: periodic exchange of cluster state information.
Every cluster-node-timeout / 2 milliseconds, each node sends a PING message to a random selection of other nodes. Each PING carries:
- The sender's view of the cluster topology (node IDs, IP addresses, slot assignments, node states)
- Information about nodes the sender considers potentially or definitely failed
Recipients respond with PONG messages carrying the same type of information. Through this continuous gossip, all nodes converge on a consistent view of the cluster topology.
Node States: PFAIL and FAIL
PFAIL (Probable Failure)
A node marks another node as PFAIL (Probable Failure) if it does not receive a PONG response within cluster-node-timeout:
PFAIL is a soft state — a single node's suspicion. It could be caused by:
- Node B genuinely crashed
- Network partition between Node A and Node B (but Node B is fine from other nodes' perspective)
- Node B is overloaded and slow to respond
FAIL (Definite Failure)
PFAIL becomes FAIL (Definite Failure) when a majority of primary nodes agree that a node is unreachable. This agreement propagates via gossip:
The FAIL transition is irreversible until the node comes back online and communicates with the cluster.
Failover Election
When a primary node is declared FAIL, its replica(s) detect this via gossip and initiate a failover election:
- The replica waits a delay proportional to its replication lag (less-lagged replicas go first)
- The replica broadcasts
FAILOVER_AUTH_REQUESTto all primary nodes - Primary nodes vote: they grant one vote per failover epoch; the first replica to ask in this epoch gets the vote
- If a replica receives votes from a majority of primaries: it promotes itself
- The new primary sends
PONGpackets with its updated slot ownership
Failover completes in approximately 1–2 × cluster-node-timeout.
Network Partitions: CAP Trade-offs
A network partition divides the cluster into isolated segments. Redis Cluster must choose between consistency (only one segment can accept writes) and availability (both segments continue accepting writes, potentially diverging).
Minority Partition
If a partition isolates a minority of nodes (fewer primary nodes than quorum), those nodes eventually stop accepting writes:
Node A entering fail state is the correct behaviour — it protects against writing data that would be lost when the partition heals and Node A rejoins as a replica.
Majority Partition
Sign in to keep reading
The rest of this module is free — sign in with Google to unlock it and track your progress.
Sign in & RegisterDiscussion
0Join the discussion