A-11 — Gossip Protocol and Network Partition Handling

If the primary nodes are split with majority on one side and minority on the other:

The cluster-require-full-coverage Setting

cluster-require-full-coverage yes → default

With yes: if any slot has no reachable primary, the entire cluster stops accepting commands for all keys — not just the affected slots. This maximises consistency at the cost of availability.

With no: the cluster continues serving keys for slots that do have a reachable primary. Slots without a reachable primary return errors, but other slots continue working. This maximises availability at the cost of partial availability.

Trade-off decision:

• yes: prefer consistency — no reads/writes during partial failure
• no: prefer availability — partial reads/writes continue; clients must handle CLUSTERDOWN errors for unavailable slots

---

Gossip Topology Propagation

When a node joins or leaves the cluster, or when slot assignments change, this information propagates via gossip within a few round-trip times:

Slots reassigned during resharding propagate similarly — each CLUSTER SETSLOT NODE call is included in gossip messages until all nodes have updated their slot maps.

Convergence time: In a 6-node cluster with cluster-node-timeout 15s, topology changes propagate to all nodes within 1–2 seconds. In larger clusters (100+ nodes), convergence takes proportionally longer due to the O(N) gossip messages required.

---

CLUSTER NODES: Reading the Topology

CLUSTER NODES

Each line represents one node:

Flags:

• master — primary node
• slave — replica node
• fail? — PFAIL (suspected failure)
• fail — FAIL (confirmed failure)
• handshake — node recently added, still in handshake
• noaddr — no address known

Slot ranges: Listed for primary nodes. 10923-16383 means the node owns slots 10923 to 16383.

---

Manual Failover

Trigger a manual failover (useful for planned maintenance):

Without options (graceful failover):

1. Replica asks the primary to pause accepting writes
2. Primary waits for the replica to catch up in replication
3. Primary transfers its epoch to the replica
4. Replica promotes — no data loss

FORCE: Primary is unreachable but you want to force the replica to promote without waiting for the primary's cooperation.

TAKEOVER: Promote without even requesting votes from other primaries. Use only when the cluster has lost quorum and you need to recover a partition manually. Dangerous — can cause split-brain.

---

Tuning Cluster Timing

Lower cluster-node-timeout:

• Faster failure detection and failover
• Higher risk of false positives on brief network hiccups
• More gossip traffic (PINGs sent more frequently)
• Recommended minimum: 5000ms (5 seconds)

Higher cluster-node-timeout:

• Slower failure detection and failover (more downtime during primary failure)
• More resilient to brief network delays
• Less gossip traffic

---

Summary

• Gossip protocol: nodes send PING to random peers every timeout/2; PONG responses carry cluster topology
• PFAIL: one node's suspicion (no PONG within timeout)
• FAIL: majority agreement via gossip → triggers failover
• Failover: replica detects FAIL → broadcasts election request → receives majority votes → promotes → propagates new topology via gossip
• Network partitions: minority side eventually enters cluster_state:fail and stops accepting writes — correct behaviour prevents data divergence
• cluster-require-full-coverage yes (default): entire cluster stops if any slot has no primary — prefer consistency
• cluster-require-full-coverage no: cluster serves available slots — prefer partial availability
• CLUSTER NODES shows full topology with node states, slot assignments, and link state
• Graceful manual failover via CLUSTER FAILOVER on the target replica — zero data loss

Next: A-12 — Multi-Region Redis: Active-Active and Geo-Replication — distributing Redis across regions, CRDT-based conflict resolution, and the latency-vs-consistency trade-offs of global distribution.

---