Module A-16·30 min read

Logical vs physical replication, replication slots and the slot bloat disaster, publication/subscription model, WAL sender process, Debezium + Kafka change data capture, zero-downtime major version upgrades using logical replication as a migration bridge, and replication identity modes.

JJS
Written by Jatin Jain Saraf · Senior Software Engineer

Module 17 — Logical Replication and CDC Pipelines

We needed to upgrade from PostgreSQL 12 to PostgreSQL 16. Physical streaming replication doesn't cross major versions. The naive approach: pg_dump, restore, brief downtime. Our database was 8TB. The dump alone took 4 hours. The restore took 6 hours. Then indexing. Total downtime: 14 hours. We told the team we could do it in 2 hours. We were wrong. Logical replication, done correctly, would have given us 0 seconds of downtime.


Physical Replication vs Logical Replication

Physical (Streaming)Logical
Unit of transferWAL bytes (block-level)Logical row changes (decoded)
Cross-version supportSame major version onlyWorks across major versions
Selective tablesEntire clusterPer-table granularity
DDL replicated automaticallyVia WAL blocks onlyNot automatically
Bi-directionalNoYes, carefully
Primary use caseHA, hot standby, read replicasCDC, version upgrades, ETL, Kafka

Physical replication copies WAL bytes verbatim — the standby replays the exact same block writes. It cannot cross major versions because the on-disk format changes between them. Logical replication decodes the WAL into row-level changes (INSERT/UPDATE/DELETE), which are version-agnostic. That version-agnosticism is the property that makes the zero-downtime upgrade possible.


Replication Slots — The Critical Operational Detail

Replication slots are PostgreSQL's mechanism for ensuring the WAL needed by a subscriber is not discarded before the subscriber reads it. This sounds helpful. It is helpful until the subscriber goes away.

The disaster scenario: You create a logical replication slot. The subscriber is a Debezium connector that reads changes and publishes to Kafka. The Debezium Kafka Connect worker goes down for a maintenance window. Three days pass. PostgreSQL has been faithfully retaining WAL since the slot was last consumed. WAL accumulates on disk. Your 500GB disk fills up. PostgreSQL cannot write new WAL. PostgreSQL stops accepting writes. Your entire application is down. Every application team is paging the on-call. This scenario is not hypothetical — it has killed production systems.

Monitoring slot lag — make this a production alert, not an afterthought:

sql

Alert when any slot's lag_size exceeds 5GB. Page someone when it exceeds 20GB.

Cap WAL retention per slot using max_slot_wal_keep_size (PostgreSQL 13+):

ini

When a slot is dropped due to the size limit, the slot's active column goes to false and pg_replication_slots shows the slot as invalidated. The subscriber gets an error on its next attempt to read, knows it needs to resync, and can do so. Your disk stays healthy. Set the alert lower than the limit so you have time to act before the slot is dropped.


The WAL Sender Process

Every replication connection — physical or logical — is served on the publisher/primary side by a WAL sender process (walsender), visible in pg_stat_replication and as a distinct backend in pg_stat_activity. For logical replication specifically, the WAL sender does more than stream raw WAL bytes the way it does for a physical standby: it runs the logical decoding process, invoking the output plugin (pgoutput, or a third-party plugin like the one Debezium uses) to translate each WAL record back into a logical change — an INSERT/UPDATE/DELETE with actual column values — before sending it to the subscriber. This is why logical replication needs wal_level = logical: the WAL sender's decoding step requires extra information (including catalog history) that wal_level = replica doesn't retain. One WAL sender process serves one replication slot; a publisher with many subscriptions has one walsender per active subscriber connection, each independently tracking its own decoding position via that subscriber's slot.


Publication and Subscription

On the source database:

sql

On the destination database:

sql

PostgreSQL handles the initial data sync (copying existing rows) and then streams ongoing changes. No application changes required during initial sync.

Monitoring subscription progress:

sql
sql

Replication Identity — The Row Identification Problem

Logical replication needs to identify which row to UPDATE or DELETE on the destination. Without a way to find the target row, UPDATE and DELETE cannot be applied.

sql

Tables without a primary key and without REPLICA IDENTITY FULL will fail to replicate UPDATEs and DELETEs:

text

Before setting up logical replication, audit all tables in the publication for missing primary keys:

sql

Zero-Downtime Major Version Upgrade

The playbook for upgrading PostgreSQL 12 to PostgreSQL 16 with under 5 minutes of write downtime.

Step 1: Prepare the source (PG12)

sql

Sign in to keep reading

The rest of this module is free — sign in with Google to unlock it and track your progress.

Sign in & Register

Discussion

0

Join the discussion

Loading comments...

© 2026 Jatin Jain Saraf (JJS). All rights reserved.