What this module covers: How a leader ships its WAL or binlog to followers; the honest cost of asynchronous, semi-synchronous and synchronous commit, and what quorum commit buys; replication lag treated as a product bug rather than a dashboard number; why read replicas do nothing for a write bottleneck; failover with its split brain and its measurable data loss; the fight between long replica queries and WAL replay; and physical versus logical replication.
Three Promises, and the Fourth One People Assume
Replication keeps a copy of your data on another machine by shipping the changes. Teams buy it for durability (a disk, host or availability zone dies and the data survives), read capacity (spread SELECTs across followers), or availability (something can be promoted, so an outage lasts minutes rather than hours).
Write capacity is not on the list, and Module F-9 established why: scaling one tier relocates the bottleneck. Every follower applies 100% of the leader's writes, so it duplicates write work rather than dividing it — and Module F-3's forgotten multipliers apply directly, since replication factor three multiplies bandwidth, storage and IOPS instead of sharing them, with WAL egress landing on the bill (Module O-8). Worse than neutral, in fact: Postgres replays WAL in a single recovery process while the leader generated it from many concurrent backends, so under write-heavy load lag grows. Your read replicas are least trustworthy exactly during your traffic peak. A team at a write ceiling adds three read replicas and gets a larger bill and the same ceiling.
Replication is also where "the write succeeded" stops being a boolean. Module F-2 put durability on a spectrum — application buffer, page cache, device, then devices in separate failure domains. Every setting below decides which point on it you acknowledge the client at.
The Leader's Write-Ahead Log Is the Replication Protocol
A leader-based database doesn't replicate by re-running your SQL elsewhere. It replicates the log it was already writing for crash recovery.
Postgres writes every change to the WAL before touching data pages, because that's how it survives a power cut; MySQL/InnoDB has a redo log for recovery plus a binlog for replication. Either way the leader already produces an ordered, append-only stream of exactly what changed, and a follower applying it in order ends up identical. In Postgres a walsender streams WAL to a walreceiver on each follower over ordinary TCP; the follower writes it to its own WAL, then replays it into pages. Positions in that stream are LSNs — monotonically increasing byte offsets — which is what makes the lag fixes below correct rather than approximate.
Two consequences. The stream is ordered and gapless — a follower cannot apply write 500 before 499 — so slow apply produces a uniformly old replica rather than a partially-wrong one, which is what Module P-11's consistency models are built on. And the leader must retain WAL the follower hasn't consumed: in Postgres that's a replication slot, and a follower that goes away while its slot stays behind makes the leader accumulate WAL until the leader's disk fills and writes stop. A safety mechanism for the replica became an outage for the primary. max_slot_wal_keep_size bounds it, at the cost of a full replica rebuild if the follower falls past the bound.
Analogy: the WAL is a bound ledger with numbered lines, and a follower is a second clerk copying it line by line into an identical book. He never invents an entry and never skips ahead — he can't, because line 41,203 might say "amend the amount on line 40,900." If the original is on line 41,900 and the clerk is on 41,000, everything nine hundred lines' worth of customers just did does not exist in his book. He isn't broken. He's behind, and being behind looks exactly like the data was never written.
Why this matters in production: the metric to carry is not "is replication up" but how far behind, in bytes and in seconds — pg_stat_replication gives both. Bytes tell you whether you're about to blow the retention budget; seconds tell you what users are seeing, and how much data you lose if the leader dies now.
Synchronous Replication Turns a Replica Outage Into a Write Outage
Where in the commit path the leader waits has four positions, not two.
Mode
Leader waits for
Latency cost
Loss if leader dies
Failure coupling
Asynchronous
Nothing
None
Everything unshipped (= current lag)
None — a replica outage is invisible to writers
Semi-synchronous
Follower received it
One datacentre RTT (~0.5 ms)
Usually nothing; a received-not-applied write can still be lost in an unlucky promotion
Bounded by a timeout, then degrades to async
Synchronous (flush)
Follower fsynced it
RTT + the follower's fsync
Nothing acknowledged
Total — no eligible follower means no commits
Synchronous (apply)
Follower replayed it
RTT + fsync + apply
Nothing acknowledged
Total, and a worse tail
Postgres expresses all four via synchronous_commit (off, local, remote_write, on, remote_apply) plus synchronous_standby_names.
Now the sentence people skip on the diagram and rediscover at 3 a.m. With synchronous replication, a replica outage is a write outage. If the leader must hear from a named follower before committing, a follower that is down, wedged, or across a flaky link stops every write. Two nodes in strict sync are less available for writes than one, because now two things must be up.
Quorum commit is the fix — require any k of n rather than one named node:
sql
The cost, named: an extra round trip and an extra fsync per commit, so a workload already at 1K–5K TPS with fsync-on-commit gets slower, and the tail gets much worse because commit latency is now the slowest of k acknowledgements rather than a local disk write. You need n ≥ k + 1 followers actually running to survive one loss. And synchronous replication guarantees durability, not that failover uses the synced copy — automation that promotes a lagging asynchronous follower loses data regardless.
Replication Lag Is a Product Bug, Not an Ops Metric
Lag lives on an ops dashboard, so it gets triaged as an ops concern: warn at 30 seconds, page at five minutes. That framing is why the bugs it causes get filed against the wrong team. Two hundred milliseconds of lag with reads spread across followers produces support tickets, not alerts.
"I saved it and it's gone." The user updates their profile, the write goes to the leader, the redirect's GET lands on a follower that hasn't replayed it, and the form renders the old value. That's a read-your-own-writes violation, and it's the most common lag symptom because write-then-immediately-read is how nearly every UI works. The user doesn't experience eventual consistency; they experience your product losing their data, and their next move — save again — creates a duplicate or clobbers something.
Time going backwards. Two reads in one session hit two followers with different lag. A thread shows five replies, the user refreshes, it shows three. An order goes shipped → processing. That's a monotonic reads violation, and it's nastier: no write to correlate it with, nothing in the logs, and the reporter always says "it happens sometimes." Any balancer that spreads a session freely across followers makes this a permanent low-rate background bug. Background jobs get it too — a worker consuming "order created" reads a follower, finds nothing, and treats absent as cancelled.
The discipline is classifying these as correctness bugs with a stated tolerance. Module F-1 asked for consistency tolerance as one of the four numbers precisely so this is answered before the routing is built: which reads may be stale, and by how long? Thirty seconds is fine for a public catalogue. For "did my payment go through," zero is the only acceptable answer.
Three Fixes for Read-Your-Own-Writes, All of Which Cost Something
Fix
Mechanism
Cost
Leader reads after a write
Mark the session for N seconds after any write; route its reads to the leader
The leader takes back a slice of read load; N is a guess, and a lag spike past N breaks it silently
Sticky routing
Pin a session to one follower
Fixes monotonic reads too, but inherits all four costs of stickiness from Module F-9 — and no help if that follower lags
LSN / token reads
Capture the commit position; require a follower at or past it
Correct rather than probabilistic, but the position must be plumbed through API and client, with a fallback when no follower qualifies
The third is the only one that is actually correct. The leader's commit position is an LSN, a follower's replay position is an LSN, and "has this follower seen my write" is a comparison of two integers.
typescript
MySQL offers the same guarantee through GTIDs: WAIT_FOR_EXECUTED_GTID_SET blocks on the replica until it has executed the transaction you name, making the check a bounded wait rather than a fallback. Bounded wait and fallback-to-leader are the same decision in different clothes — under lag you either add latency or add leader load. There is no third option, and pretending otherwise is how "eventually consistent reads" becomes an incident.
synchronous_commit = remote_apply delivers the same property with no application changes at all, since commit doesn't return until the follower has applied the change. It is also the most expensive row in the previous table and couples commit latency to a follower's replay speed — occasionally right for a small, high-value dataset, never right as a default.
Failover Either Saves You or Loses Your Data
Promotion — a follower stops replaying and starts accepting writes — is mechanically trivial. Everything hard is in the decisions around it.
Automatic versus manual is a real trade, not a maturity ladder. Automatic failover cuts recovery from tens of minutes to tens of seconds, and it fails the way all automated remediation fails: by acting on a wrong belief. A partition between the health-checker and a perfectly healthy leader looks identical to a dead leader. Manual failover has a human confirming the leader is really gone — slower, and far less likely to promote a second leader alongside the first. Choose by asking which you can afford more: minutes of downtime, or a chance of divergence.
Split brain is that divergence: two nodes both believe they lead, both accept writes, and the two write sets are both real and mutually unmergeable. That isn't an availability incident, it's a data reconciliation project — which is why the fix is never "detect it faster." Fencing prevents it, and it has to be mechanical: promotion gated on a majority of an odd-numbered set of observers holding a leader lease in a consensus store (what Patroni does with etcd or Consul), so a minority partition cannot promote; the old leader cut off by revoking its virtual IP, self-demoting when it loses its lease, power-fenced in the strict case; and target_session_attrs=read-write in Postgres connection strings, so a client reaching a demoted node fails to connect rather than writing into the void.
Then the part left off the diagram: on asynchronous failover you lose data, and the amount is knowable in advance. If the leader dies while the promoted follower is 1.8 seconds behind, every transaction acknowledged in those 1.8 seconds is gone — committed from the client's view, absent from the new leader's. Those are confirmed orders and recorded payments. And when the old node returns its WAL has diverged, so it can't resume as a follower; it must be rewound (pg_rewind) or rebuilt, which discards those transactions rather than merging them.
That quantity is your RPO — recovery point objective, the data you have accepted losing, measured in time. Asynchronous replication sets RPO to worst-case lag, not average, which is why bounding lag matters more than optimising it; quorum commit buys RPO ≈ 0 for the extra RTT and fsync above. Its sibling RTO is what automation buys and manual promotion spends. Module O-7, "Disaster Recovery: RTO and RPO", starts from both numbers and works back through backup tiers and point-in-time recovery. This module's job is to make you state your RPO out loud, because an unstated RPO is always discovered during the incident and always larger than anyone assumed.
The Reporting Replica Fights the Replay Process, and Somebody Pays
A cascade that surprises people, because both halves look like correct configuration. You put analytics on a follower to keep heavy queries off the leader. Sensible. But the follower is now doing two things at once: serving a 90-second query that needs a consistent snapshot, and replaying WAL that says "vacuum removed the row versions that snapshot depends on." These are irreconcilable, and Postgres resolves it by killing the query: ERROR: canceling statement due to conflict with recovery. Reports fail at random, only the long ones, only sometimes.
Two knobs, pushing the pain to different places:
max_standby_streaming_delay lets replay pause that long before cancelling the query. Queries survive; replication lag now has a floor equal to your setting, so every fix above gets weaker and your RPO gets worse.
hot_standby_feedback = on has the follower report its oldest snapshot to the leader, so the leader's vacuum won't remove versions the follower still needs. Queries survive and replay keeps up — and the cost lands on the leader as exactly the bloat mechanism from Module F-13. A long analytical query on the follower now holds back cleanup on the primary, the same pathology as an idle-in-transaction session, arriving from a machine nobody was watching. Tables grow past their live data, sequential scans read the dead rows, and the leader slows with no query on the leader to blame.
Protect the queries, protect the lag, or protect the leader from bloat — pick two. Which argues for the fourth option: don't co-locate an analytical workload on a replica of your OLTP database at all. Give reporting its own copy, fed by logical replication into a store shaped for it (Modules P-22 and P-23). That costs a pipeline and its own staleness, and it stops two workloads fighting over one snapshot horizon.
Physical Ships Bytes; Logical Ships Rows
Physical replication ships the WAL itself — literal 8 KB page changes — so the follower is a byte-for-byte clone. Cheap (the leader writes that stream anyway), complete (every table, index and DDL), and rigid: same major version, whole cluster or nothing, follower read-only. This is what you want for high availability and failover.
Logical replication decodes the WAL into row-level change events — insert, update, delete, with column values — and applies them as operations on the subscriber (wal_level = logical, plus CREATE PUBLICATION / CREATE SUBSCRIPTION). It buys selective replication of specific tables; cross-version and cross-system targets, which is what makes near-zero-downtime major upgrades possible (Module O-4); and a writable subscriber that can carry its own indexes and derived tables — the reporting escape hatch above.
The costs aren't small. Decoding is CPU work on the leader. Every replicated table needs a replica identity — a primary key, or REPLICA IDENTITY FULL, which makes updates and deletes far more expensive because the whole old row goes into the WAL. DDL isn't replicated, so schema changes must be coordinated by hand on both sides and the wrong order breaks the subscription. Sequence values need separate handling in the versions most teams run, initial sync copies whole tables, and a stuck logical slot fills the leader's disk exactly like a physical one.
That decoded stream is a general-purpose primitive, though. Once the database emits "here is what changed, in order, with values," you can feed a search index, a cache invalidator, a warehouse, or another service's read model — without Module F-14's dual-write problem, because the log is the single source of truth and everything downstream is derived from it. That's Change Data Capture, and Module P-23 is entirely about it. Logical replication is the mechanism; CDC is what you build with it.
Where This Shows Up in Your Stack
PostgreSQL:pg_stat_replication for per-follower LSNs and the three lag columns. Alert on slot retention size as aggressively as on lag — a slot is how a dead replica takes down a live primary. Patroni with etcd/Consul is the standard automatic-failover stack, and the part worth reviewing is its fencing, not its promotion.
Node.js: two pools, writer and reader, with routing decided per request rather than per call site. What survives production is a request-scoped flag ("this request has written, or carries a read token") that forces the leader; leaving the choice to call sites guarantees someone reads a follower right after a write. Both pools count against max_connections on their nodes (Module P-4).
Redis: replication is asynchronous, and WAIT is not a durability guarantee — it reports how many replicas acknowledged, after the fact, which is different from refusing the write. Sentinel and Cluster failover can lose acknowledged writes, so a Redis primary is the wrong home for data whose loss you'd have to explain (Module P-12).
MySQL: GTIDs make "has this replica executed my transaction" a first-class question, the cleanest read-your-own-writes primitive here. Semi-synchronous replication acknowledges receipt rather than apply and reverts to asynchronous on timeout, so your RPO quietly changes during exactly the network trouble that made you want it.
Summary
Replication buys durability, read capacity and failover — never write capacity. A follower replays every write the leader took, and serial replay means lag peaks when traffic does (Modules F-9, F-3).
The replication protocol is the recovery log — WAL/binlog shipping, ordered and gapless, with LSNs as positions.
A replication slot points a gun at the leader. An absent follower with a retained slot fills the leader's disk and stops writes; bounding retention costs a full replica rebuild.
Synchronous replication makes a replica outage a write outage. Quorum commit (ANY k (…)) restores availability at the cost of an extra RTT plus fsync per commit, a worse tail, and n ≥ k + 1 machines that must be running.
Lag is a product bug. Read-your-own-writes ("I saved it and it's gone") and monotonic reads (time running backwards) are correctness failures, not dashboard values. Fix with a post-write leader window, sticky routing (with F-9's four costs), or LSN/token reads — the only correct one, paid for in plumbing and leader fallback.
Asynchronous failover loses data, and the amount is your worst-case lag. That's your RPO; the diverged old leader must be rewound or rebuilt, discarding those acknowledged transactions permanently (Module O-7).
Split brain is a reconciliation project, not an outage. Prevent it with quorum-gated promotion and real fencing, not faster detection.
Long replica queries and WAL replay cannot both win.max_standby_streaming_delay protects queries and floors your lag; hot_standby_feedback protects both and moves MVCC bloat onto the leader (Module F-13).
Physical replication clones bytes for failover; logical replication emits rows for everything else — paid for in decode CPU, replica identity and manual DDL coordination.
The next module, Sharding and Partitioning, takes up the problem this one deliberately refused to solve. Replication makes one dataset redundant; when the write volume or the dataset itself no longer fits on one leader, the only move left is to split it — and every consistency question you just learned to handle across replicas returns harder across shards, where a single query may have no single machine that can answer it.
Knowledge Check
A user updates their delivery address, the app redirects to the profile page, and the old address renders. It corrects itself within a second. Reads are load-balanced across three asynchronous followers. Which framing and fix does the module endorse?
A leader's host fails. Automatic failover promotes an asynchronous follower in 25 seconds. Finance later finds 40 orders that returned 201 Created and do not exist. Fencing worked and there was no split brain. What happened?
Nightly reports on a read replica began failing with canceling statement due to conflict with recovery. A DBA set hot_standby_feedback = on and the reports now succeed. Two weeks later the *primary's* largest tables are three times their live-data size and unrelated queries have slowed. What is the connection?
Test your knowledge with more question sets
Sign in to access a wider variety of questions and get notified when new practice sets are added to this module.
-- Three followers; a commit needs an fsync acknowledgement from ANY one of them.-- Any single follower can die, or be patched, with zero write impact.ALTER SYSTEM SET synchronous_standby_names ='ANY 1 (repl_a, repl_b, repl_c)';ALTER SYSTEM SET synchronous_commit ='on';-- flushed on the standby, not merely received-- The trap: 'FIRST 1 (repl_a)' makes one *specific* standby mandatory, which-- reintroduces the exact single-replica write outage quorum exists to avoid.
// After the write, hand the leader's position back to the caller// (cookie, header, or response field — opaque to the client).const{ rows }=await leader.query('SELECT pg_current_wal_insert_lsn() AS lsn');res.setHeader('X-Read-Token', rows[0].lsn);asyncfunctionreaderFor(token?:string):Promise<Client>{if(!token)returnpickFollower();const f =pickFollower();const{ rows }=await f.query('SELECT pg_last_wal_replay_lsn() >= $1::pg_lsn AS caught_up',[token]);// Falling back to the leader is the honest failure mode: correctness over offload.// Count it — a rising rate means lag is eating the read capacity you paid for.return rows[0].caught_up ? f : leader;}