The semantic lock is the one to reach for first, and its cost is worth stating plainly: introducing a pending state means every consumer of that data must now handle it. A report that counts pending orders as real, or a UI that shows a pending charge as complete, is the isolation failure appearing as a product bug rather than a data bug.

---

Compensation Is a Forward Transaction, Not a Rollback

A database rollback erases history: after it, nothing happened. A compensation is a new transaction that attempts to reach a state resembling the original one, and the difference is not pedantic.

So compensation is approximate, observable, and sometimes impossible. Each of those three properties has a design consequence, and the third is the one that reshapes the whole saga.

---

The Part Usually Left Out: Compensation Fails Too

Sagas are almost always presented as if compensation succeeds. It does not, and the resulting states are where real systems get stuck.

Non-compensable steps, and the pivot transaction

Some effects cannot be undone at all: an email or SMS delivered, a webhook fired to a customer's system (Module P-17), a message posted publicly, a physical parcel handed to a courier, a report filed with a regulator.

This gives the single most useful structural rule in saga design. Divide the saga into three zones:

• Compensatable steps come first. Each has a working compensation, so a failure anywhere in this zone unwinds cleanly.
• The pivot is the point of no return — the step after which the saga must complete. Committing it is the decision that the operation will happen.
• Retriable steps come last. They have no compensation and therefore must be retried until they succeed, with no bounded attempt limit that gives up silently. They must also be designed so that success is eventually achievable — which means they cannot have preconditions that can become permanently false.

Ordering a saga this way is a design act, and it frequently conflicts with the instinct to "do the cheap things first." Sending the receipt email before capturing payment feels harmless and makes the saga uncompensatable from step one. Every non-compensable step belongs after the pivot.

Unrecoverable saga states

Even with correct ordering, compensation can fail:

• Transiently — the payment provider is down. Retry with backoff; this is the easy case and most failures are this.
• Permanently — the refund is rejected because the customer's card is closed or expired. The inventory you released was already sold to somebody else. The shipment cannot be cancelled because it is on a van. The loyalty points were spent.

Now the saga can neither go forward (the failing step will not succeed) nor backward (the compensation will not succeed). This state is real, it happens at a low but non-zero rate, and there is no algorithm that resolves it. Somebody has to decide: refund by bank transfer instead, write off the stock, issue a credit note, contact the customer.

The actual safety net is operator tooling

Which produces the honest conclusion of this module: the mechanism that makes sagas safe in production is not the saga pattern. It is the human-in-the-loop dead-letter queue and the tooling around it, and that tooling is a first-class deliverable rather than a follow-up ticket.

What it must provide:

• A queryable list of stuck sagas, with an alert on count and on age of the oldest — the same metric that matters for jobs (Module P-19), for the same reason.
• The full saga record: every step attempted, its outcome, its error, its timestamps, the accumulated context, and the compensations attempted. An operator who has to reconstruct this from logs will not do it correctly at 3 a.m.
• Safe operator actions: retry a single step, force-compensate a step, skip a step with a recorded reason, mark the saga resolved manually. Each action recorded, attributable, and — critically — each idempotent, because an operator will click twice.
• A financial escape hatch: the ability to write a reversing ledger entry with a reason (Module P-24), because when compensation is impossible the resolution is usually an accounting one.
• Reconciliation comparing your saga outcomes against the downstream systems' own records, because a saga that believes it completed and a payment provider that disagrees is a break you will only find by looking.

Why this matters in production: a saga implementation without this tooling appears to work. The happy path is fine, transient failures retry successfully, and the compensation path was tested. What happens instead is that a handful of sagas per week enter unrecoverable states, nobody notices because nothing alerts on them, and the discovery comes weeks later from a customer or from finance — at which point resolution is a support engineer writing UPDATE statements against production, which is both the least safe and most expensive possible remediation. Build the operator queue with the saga, not after it.

Durable execution engines are worth knowing about

A category of tooling — Temporal, Restate, AWS Step Functions, and similar — exists specifically to remove the boilerplate here. What they give you is durable execution: the orchestrator's control flow is itself persisted, so you write what looks like ordinary sequential code with awaits, and the engine records each step's completion, replays after a crash, and handles retries, timeouts and compensation hooks.

The costs are real: an additional stateful system to operate, a programming model with constraints (your workflow code must be deterministic and side effects must live in activities), and a learning curve. They do not remove the need for the pivot ordering or the operator queue — but they do remove the hand-rolled state machine, the retry bookkeeping, and roughly the top half of the bugs in this module.

---

The Outbox: Not a Distributed Transaction, But It Removes the Need for One

A large share of "we need a distributed transaction" requirements are actually the shape update my own state and tell the world. That does not need atomicity across services — it needs one local write and a guaranteed publish.

The dual write is broken, for the fourth time in this course (Modules P-16, P-19, P-20, P-23):

The outbox makes it one transaction, because the event is written to a table in the same database:

A relay then publishes outbox rows — either a poller (simple, adds latency and load) or, better, log-based CDC over the outbox table (Module P-23), which gives you no polling, no relay process to operate, and an insert-only source with no before-image or TOAST complications.

Delivery is at-least-once, so the receiving side needs an inbox: a table with UNIQUE (event_id) where consumers deduplicate before processing (Module P-16, Module P-17). Outbox on one side, inbox on the other, and the pair gives you exactly-once effect over at-least-once delivery.

Two details that make or break it. Ordering is per aggregate, so partition on aggregate_id and accept that events for different orders may interleave. And the outbox table is high-churn — insert, publish, delete — so either partition it by day and drop partitions, or let CDC read it and delete in batches, because otherwise it becomes the bloat pattern from Module P-9.

The outbox does not give you atomicity across services. What it gives you is the guarantee that no step is ever lost, which — per the opening of this module — is what the requirement usually was.

---

Choosing

And the question to ask before any of them: does this actually need to be atomic, or does it need to not lose anything? The second is far cheaper, and it is usually the real requirement.

---

Where This Shows Up in Your Stack

• PostgreSQL: max_prepared_transactions defaults to 0, and that default is a deliberate warning. A prepared transaction that is never resolved holds locks and pins its transaction ID, so vacuum cannot advance anywhere in the cluster (Module F-13) — this is one of the few ways to degrade an entire Postgres instance with a single forgotten statement, ending in wraparound protection refusing writes. If you enable it, alert on pg_prepared_xacts with any row older than minutes. The outbox is the pattern to use instead, and SELECT … FOR UPDATE SKIP LOCKED is how the relay claims rows (Module P-19).
• Node.js: a saga hand-rolled as promise chains with try/catch compensation loses all its state on process exit, which is the failure the whole module is about — the state machine must be in the database, not the call stack. Durable execution engines exist precisely to fix this; if you hand-roll, make the step table the source of truth and the code a stateless driver over it.
• Kafka: transactions give exactly-once for Kafka-to-Kafka pipelines, and nothing beyond that boundary — a card charge inside a consumer is outside the transaction and can repeat. Partition the outbox topic on aggregate_id for per-entity ordering, and remember that consumer rebalances mean a saga step handler can run twice (Module P-15).
• Redis: not a participant in any of this. MULTI/EXEC is not 2PC and offers no rollback on logical failure, and Redis's failover semantics (Module A-1) make it unsuitable as the durable store for saga state.
• Payment providers and third-party APIs: cannot participate in 2PC and often cannot be compensated exactly (fees, FX, closed cards). This is what forces the pivot-transaction discipline, and it is why Module P-24's reconciliation exists.

---

Summary

• Before choosing a mechanism, ask whether atomicity is actually required. Usually the requirement is that no step is lost, and that is a much cheaper guarantee than "all or nothing."
• 2PC is correct and its failure mode is fatal. A participant that votes YES surrenders the right to decide, so a coordinator crash between phases leaves it in-doubt indefinitely: locks held, connection pool filling with waiters, and in Postgres a pinned transaction ID that stops vacuum cluster-wide and ends in wraparound protection. Recovery needs a human, who may commit one participant and abort another — violating the very property 2PC provides.
• Even when nothing crashes, 2PC costs two round trips plus fsyncs, and its availability is the product of all participants' — five services at 99.9% gives 99.5%. It also cannot include an HTTP API, and under partition it blocks outright. It is right inside one database, for Kafka-to-Kafka transactions, and in a single-vendor cluster with a replicated coordinator and short transactions.
• A saga trades atomicity for progress: local transactions that each commit, with compensations for those already done. Orchestrate rather than choreograph beyond two or three steps, because "which sagas are stuck and where?" must have a cheap answer. The saga's state must be durable, every step idempotent, and every step retried with backoff before being treated as failed.
• Sagas also abandon isolation, and that is where the bugs are. Intermediate states are visible system-wide, reintroducing dirty reads and lost updates at the architecture level. Counter with a semantic lock (an explicit pending state — whose cost is that every consumer must now handle it, or the isolation failure shows up as a product bug), commutative updates, re-validation at the pivotal step, reordering, or version checks.
• Compensation is a forward transaction that approximates undo, not a rollback: a refund is not an un-charge (fees, FX, and both entries on the statement), releasing stock may find it already sold, and cancelling a shipment that has left is a return with a cost.
• Order the saga into compensatable steps, a pivot, then retriable steps. Non-compensable effects — email sent, SMS delivered, webhook fired, parcel dispatched — have no compensation and must therefore come after the point of no return, where they are retried until they succeed. Doing the cheap non-compensable thing first, like emailing before capturing payment, makes the saga uncompensatable from step one.
• Compensation itself fails, transiently (retry) or permanently (closed card, sold stock, shipment on a van, points already spent). The resulting state can go neither forward nor backward, and no algorithm resolves it.
• The real safety net is a human-in-the-loop dead-letter queue with first-class tooling: a queryable list of stuck sagas alerting on count and oldest age, the full step-by-step record with errors and context, idempotent operator actions (retry, force-compensate, skip with a reason, resolve), a path to write a reversing ledger entry, and reconciliation against downstream records. Without it a saga implementation appears to work while accumulating stuck cases that are discovered weeks later by a customer, and remediated by hand-written UPDATEs in production.
• Durable execution engines (Temporal, Restate, Step Functions) persist the orchestrator's control flow so a crash replays completed steps rather than re-running them, removing the hand-rolled state machine and most of the retry bookkeeping — at the cost of another stateful system and a constrained programming model. They do not remove the pivot ordering or the operator queue.
• The outbox is not a distributed transaction; it removes the need for one in the common "update my state and tell the world" case. One local transaction writes both the state change and the event, a relay publishes — best via log-based CDC over the outbox table (Module P-23) — and an inbox with UNIQUE (event_id) deduplicates on the consumer, giving exactly-once effect over at-least-once delivery. Partition on the aggregate ID for per-entity ordering, and partition the table by day so it does not bloat.

The next module, Microservices vs Monolith, asks the question that determines how much of this you need at all. Every saga, every outbox, every fenced lock and every eventual-consistency countermeasure in this phase is a cost imposed by a service boundary — so the boundary had better be somewhere that earns it. A-4 is about where those boundaries belong, what a distributed monolith looks like when you guess wrong, and the honest list of reasons not to split.

---