Module A-11·18 min read

How cluster nodes propagate topology changes via PING/PONG gossip, cluster-node-timeout in failure detection, split-brain handling with cluster-require-full-coverage, and when a Cluster sacrifices availability for consistency.

JJS
Written by Jatin Jain Saraf · Senior Software Engineer

A-11 — Gossip Protocol and Network Partition Handling

Who this module is for: You run a Redis Cluster and want to understand how nodes discover each other, propagate topology changes, and handle scenarios where part of the cluster becomes unreachable. The gossip protocol is the nervous system of Redis Cluster — understanding it explains why certain failure modes occur and how to configure the cluster to balance availability against consistency.


The Gossip Protocol

In Redis Cluster, every node knows about every other node. There is no central registry. Instead, nodes maintain this knowledge through a gossip protocol: periodic exchange of cluster state information.

Every cluster-node-timeout / 2 milliseconds, each node sends a PING message to a random selection of other nodes. Each PING carries:

  • The sender's view of the cluster topology (node IDs, IP addresses, slot assignments, node states)
  • Information about nodes the sender considers potentially or definitely failed

Recipients respond with PONG messages carrying the same type of information. Through this continuous gossip, all nodes converge on a consistent view of the cluster topology.

text

Node States: PFAIL and FAIL

PFAIL (Probable Failure)

A node marks another node as PFAIL (Probable Failure) if it does not receive a PONG response within cluster-node-timeout:

text

PFAIL is a soft state — a single node's suspicion. It could be caused by:

  • Node B genuinely crashed
  • Network partition between Node A and Node B (but Node B is fine from other nodes' perspective)
  • Node B is overloaded and slow to respond

FAIL (Definite Failure)

PFAIL becomes FAIL (Definite Failure) when a majority of primary nodes agree that a node is unreachable. This agreement propagates via gossip:

text

The FAIL transition is irreversible until the node comes back online and communicates with the cluster.


Failover Election

When a primary node is declared FAIL, its replica(s) detect this via gossip and initiate a failover election:

  1. The replica waits a delay proportional to its replication lag (less-lagged replicas go first)
  2. The replica broadcasts FAILOVER_AUTH_REQUEST to all primary nodes
  3. Primary nodes vote: they grant one vote per failover epoch; the first replica to ask in this epoch gets the vote
  4. If a replica receives votes from a majority of primaries: it promotes itself
  5. The new primary sends PONG packets with its updated slot ownership

Failover completes in approximately 1–2 × cluster-node-timeout.


Network Partitions: CAP Trade-offs

A network partition divides the cluster into isolated segments. Redis Cluster must choose between consistency (only one segment can accept writes) and availability (both segments continue accepting writes, potentially diverging).

Minority Partition

If a partition isolates a minority of nodes (fewer primary nodes than quorum), those nodes eventually stop accepting writes:

text

Node A entering fail state is the correct behaviour — it protects against writing data that would be lost when the partition heals and Node A rejoins as a replica.

Majority Partition

Sign in to keep reading

The rest of this module is free — sign in with Google to unlock it and track your progress.

Sign in & Register

Discussion

0

Join the discussion

Loading comments...

© 2026 Jatin Jain Saraf (JJS). All rights reserved.