Module P-5·26 min read

Leader–follower topology, sync vs async trade-offs, replication lag as a product bug, and read-your-own-writes.

JJS
Written by Jatin Jain Saraf · Senior Software Engineer

Module P-5 — Replication

What this module covers: How a leader ships its WAL or binlog to followers; the honest cost of asynchronous, semi-synchronous and synchronous commit, and what quorum commit buys; replication lag treated as a product bug rather than a dashboard number; why read replicas do nothing for a write bottleneck; failover with its split brain and its measurable data loss; the fight between long replica queries and WAL replay; and physical versus logical replication.


Three Promises, and the Fourth One People Assume

Replication keeps a copy of your data on another machine by shipping the changes. Teams buy it for durability (a disk, host or availability zone dies and the data survives), read capacity (spread SELECTs across followers), or availability (something can be promoted, so an outage lasts minutes rather than hours).

Write capacity is not on the list, and Module F-9 established why: scaling one tier relocates the bottleneck. Every follower applies 100% of the leader's writes, so it duplicates write work rather than dividing it — and Module F-3's forgotten multipliers apply directly, since replication factor three multiplies bandwidth, storage and IOPS instead of sharing them, with WAL egress landing on the bill (Module O-8). Worse than neutral, in fact: Postgres replays WAL in a single recovery process while the leader generated it from many concurrent backends, so under write-heavy load lag grows. Your read replicas are least trustworthy exactly during your traffic peak. A team at a write ceiling adds three read replicas and gets a larger bill and the same ceiling.

Replication is also where "the write succeeded" stops being a boolean. Module F-2 put durability on a spectrum — application buffer, page cache, device, then devices in separate failure domains. Every setting below decides which point on it you acknowledge the client at.


The Leader's Write-Ahead Log Is the Replication Protocol

A leader-based database doesn't replicate by re-running your SQL elsewhere. It replicates the log it was already writing for crash recovery.

Postgres writes every change to the WAL before touching data pages, because that's how it survives a power cut; MySQL/InnoDB has a redo log for recovery plus a binlog for replication. Either way the leader already produces an ordered, append-only stream of exactly what changed, and a follower applying it in order ends up identical. In Postgres a walsender streams WAL to a walreceiver on each follower over ordinary TCP; the follower writes it to its own WAL, then replays it into pages. Positions in that stream are LSNs — monotonically increasing byte offsets — which is what makes the lag fixes below correct rather than approximate.

Two consequences. The stream is ordered and gapless — a follower cannot apply write 500 before 499 — so slow apply produces a uniformly old replica rather than a partially-wrong one, which is what Module P-11's consistency models are built on. And the leader must retain WAL the follower hasn't consumed: in Postgres that's a replication slot, and a follower that goes away while its slot stays behind makes the leader accumulate WAL until the leader's disk fills and writes stop. A safety mechanism for the replica became an outage for the primary. max_slot_wal_keep_size bounds it, at the cost of a full replica rebuild if the follower falls past the bound.

Analogy: the WAL is a bound ledger with numbered lines, and a follower is a second clerk copying it line by line into an identical book. He never invents an entry and never skips ahead — he can't, because line 41,203 might say "amend the amount on line 40,900." If the original is on line 41,900 and the clerk is on 41,000, everything nine hundred lines' worth of customers just did does not exist in his book. He isn't broken. He's behind, and being behind looks exactly like the data was never written.

Why this matters in production: the metric to carry is not "is replication up" but how far behind, in bytes and in secondspg_stat_replication gives both. Bytes tell you whether you're about to blow the retention budget; seconds tell you what users are seeing, and how much data you lose if the leader dies now.


Synchronous Replication Turns a Replica Outage Into a Write Outage

Where in the commit path the leader waits has four positions, not two.

ModeLeader waits forLatency costLoss if leader diesFailure coupling
AsynchronousNothingNoneEverything unshipped (= current lag)None — a replica outage is invisible to writers
Semi-synchronousFollower received itOne datacentre RTT (~0.5 ms)Usually nothing; a received-not-applied write can still be lost in an unlucky promotionBounded by a timeout, then degrades to async
Synchronous (flush)Follower fsynced itRTT + the follower's fsyncNothing acknowledgedTotal — no eligible follower means no commits
Synchronous (apply)Follower replayed itRTT + fsync + applyNothing acknowledgedTotal, and a worse tail

Postgres expresses all four via synchronous_commit (off, local, remote_write, on, remote_apply) plus synchronous_standby_names.

Now the sentence people skip on the diagram and rediscover at 3 a.m. With synchronous replication, a replica outage is a write outage. If the leader must hear from a named follower before committing, a follower that is down, wedged, or across a flaky link stops every write. Two nodes in strict sync are less available for writes than one, because now two things must be up.

Quorum commit is the fix — require any k of n rather than one named node:

sql

The cost, named: an extra round trip and an extra fsync per commit, so a workload already at 1K–5K TPS with fsync-on-commit gets slower, and the tail gets much worse because commit latency is now the slowest of k acknowledgements rather than a local disk write. You need n ≥ k + 1 followers actually running to survive one loss. And synchronous replication guarantees durability, not that failover uses the synced copy — automation that promotes a lagging asynchronous follower loses data regardless.


Replication Lag Is a Product Bug, Not an Ops Metric

Lag lives on an ops dashboard, so it gets triaged as an ops concern: warn at 30 seconds, page at five minutes. That framing is why the bugs it causes get filed against the wrong team. Two hundred milliseconds of lag with reads spread across followers produces support tickets, not alerts.

"I saved it and it's gone." The user updates their profile, the write goes to the leader, the redirect's GET lands on a follower that hasn't replayed it, and the form renders the old value. That's a read-your-own-writes violation, and it's the most common lag symptom because write-then-immediately-read is how nearly every UI works. The user doesn't experience eventual consistency; they experience your product losing their data, and their next move — save again — creates a duplicate or clobbers something.

Time going backwards. Two reads in one session hit two followers with different lag. A thread shows five replies, the user refreshes, it shows three. An order goes shippedprocessing. That's a monotonic reads violation, and it's nastier: no write to correlate it with, nothing in the logs, and the reporter always says "it happens sometimes." Any balancer that spreads a session freely across followers makes this a permanent low-rate background bug. Background jobs get it too — a worker consuming "order created" reads a follower, finds nothing, and treats absent as cancelled.

Sign in to keep reading

The rest of this module is free — sign in with Google to unlock it and track your progress.

Sign in & Register

Discussion

0

Join the discussion

Loading comments...

© 2026 Jatin Jain Saraf (JJS). All rights reserved.