Duplicates do not arrive politely spaced out. The ambiguous-failure retry often arrives while the original is still running — that is precisely the case where the response was slow enough to time out. A user's double-click arrives 80 ms apart. A queue redelivery after a worker's visibility timeout arrives while the original worker is still alive and working. So the two requests interleave, both SELECT nothing, and both charge.

The fix is to make the insert itself the claim, and let the database's uniqueness constraint be the arbiter. This is Module F-13's argument about pushing concurrency control into the constraint rather than into application logic, and it is the load-bearing idea of the whole module:

Read the three branches, because each is a real production case with a different correct answer:

The 409 deserves a word, because "just wait for the first request and return its answer" is tempting and expensive. Holding request two open while request one finishes means two connections and two request slots consumed for one unit of work, and under a duplicate storm that is how you exhaust the connection pool (Module P-4) and then the web tier. A short, explicit 409 with Retry-After: 1 costs the caller a round trip and costs you almost nothing.

The expired lock is the failure mode people forget

A worker claims a key, starts the work, and its process is killed — an OOM kill, a deploy, a spot instance reclaimed (Module O-8). The row sits at in_flight forever. Without a lease, every future retry of that key gets 409 in perpetuity: the key is poisoned, the work never completes, and the customer's order is stuck in a state no code can leave.

Hence locked_until. When a retry finds an in_flight row whose lease has lapsed, it may steal the claim — take the row, extend the lease, and re-run. That is safe only if the underlying work is itself re-drivable, which is exactly what the next section is about. Set the lease longer than the operation's p99 and shorter than a human's patience; if the operation can legitimately run for ten minutes, it belongs in a background job (Module P-19) with the key recording the job, not the work.

Should a failed attempt be retryable under the same key?

Split by cause, because conflating them causes real incidents:

• Transient failure (timeout to a downstream, deadlock, 503): the key should be released — delete the row or mark it retryable — so the caller's retry can genuinely try again. Storing a permanent failure here means the caller's legitimate retry gets a replayed error forever, and the payment that would have succeeded on attempt two never does.
• Deterministic rejection (validation error, card declined, insufficient funds): store it and replay it. Repeating it wastes a call to the payment network, and each declined authorisation attempt is a signal that fraud systems count against the cardholder.

The distinction is the same one Module A-6 draws about what is worth retrying, applied to the record rather than the request.

---

The Hard Part Is Atomicity: the Key and the Effect Must Land Together

Everything above assumes the key record and the work commit as one unit. When both live in your database, they do — put them in one transaction and the problem disappears:

Either both exist or neither does. This is the easy case, and if your operation is entirely local, stop here.

The hard case is the one that matters: the effect is outside your database. You charge a card, you call a shipping API, you publish to Kafka, you send an email. There is now a gap you cannot close, in either order:

There is no ordering of two non-transactional writes that survives a crash between them. Anyone who tells you otherwise has moved the gap, not removed it. What you can do is make one of the two sides recoverable, and there are exactly three honest strategies.

1. Propagate your key to the downstream, so their idempotency covers your gap. This is the best answer whenever it is available, and it is available more often than teams check. Stripe, Adyen, most payment processors, and many modern APIs accept an idempotency key. Derive it deterministically from yours and pass it along:

Now Order A above is safe: the retry re-issues the same charge with the same downstream key, the processor recognises it, no second capture occurs, and it returns the original result — which you then record. Exactly-once effect achieved across a boundary you do not control, using two at-least-once systems and one shared name. The cost is that the downstream's dedupe window now bounds your correctness, so read their documentation for the number and check it exceeds your retry window.

2. Record intent first, then drive the effect from a durable record. This is the transactional outbox (Module A-3) and the shape is worth internalising: in one local transaction write the business row and a pending_effect row. A separate worker reads pending effects and performs them, marking them done. A crash anywhere leaves a pending row that gets retried — so the effect is at-least-once, which is why it must still be idempotent by strategy 1 or 4. What this buys is that nothing is ever lost, and the ambiguity is confined to one place with one retry policy instead of scattered across request handlers.

3. Make it detectable and reconcile. For effects where neither of the above is possible, accept the gap and build the sweeper: a periodic job that lists the downstream's recent operations, compares against yours, and repairs the difference. Payment reconciliation exists for exactly this reason and is not optional in a payments system (Module P-24). The cost is a scheduled job, a watermark, and an alert on non-zero unreconciled counts — which is far cheaper than the alternative, since the alternative is finding out from customers.

4. Choose an operation with a natural unique key downstream. Sometimes you can convert the problem away: instead of "increment the ledger balance," insert a ledger entry with UNIQUE (source_event_id). The duplicate insert fails on the constraint and the balance, computed as a sum, is correct. This is the absolute-versus-relative lever from earlier, and in ledger design it is the whole game — which is why double-entry ledgers are append-only tables with uniqueness constraints rather than mutable balance columns.

Why this matters in production: the incident is not "we sent a duplicate charge." It is that duplicates concentrate during incidents. Latency rises, timeouts fire everywhere at once, every retry mechanism in the system engages simultaneously, and the duplicate rate goes from negligible to a large fraction of traffic in the same ten minutes your team is busy with the original outage. Idempotency is not a refinement you add when you have time; it is the thing that decides whether an availability incident also becomes a financial and data-integrity incident.

---

Exactly-Once Delivery Does Not Exist; Exactly-Once Effect Does

This distinction is worth being pedantic about, because vendor marketing is not.

Exactly-once delivery is impossible over an unreliable network. The two-generals problem is the short proof: any protocol that requires both parties to know the message arrived needs an acknowledgement, and the acknowledgement is itself a message that can be lost, so it needs acknowledging. There is no finite bottom. Formally, atomic delivery between two parties over a lossy channel cannot be guaranteed, and no amount of protocol design changes it — this is a result about the world, not a limitation of current libraries.

So every real system picks one:

The third row is the honest formulation: exactly-once effect built from at-least-once delivery plus an idempotent receiver. The delivery layer keeps duplicating; the receiver keeps collapsing them. From the outside the composition behaves as if each message took effect once, which is what anyone asking for exactly-once actually wants.

Read vendor claims through that lens. Kafka's exactly-once semantics (Module P-15) are genuine and narrow: transactions atomically tie consumed offsets to records produced back into Kafka, so a read-process-write pipeline that stays inside Kafka can be exactly-once. The moment your processing charges a card or writes to Postgres outside that transaction, the guarantee stops at the boundary — the external effect is not in the transaction, so it can repeat. Kafka's own broker-side idempotent producer is likewise real and narrow: it deduplicates retries of the same producer's sequence numbers, which prevents duplicates from producer-side retry, not duplicates from your application publishing the same logical event twice.

SQS FIFO deduplicates on a MessageDeduplicationId — within a five-minute window. That number is the entire caveat, and it is smaller than most retry schedules, which brings us to the last section.

---

A Dedupe Window Shorter Than Your Retry Schedule Is Decoration

Deduplication needs memory, memory costs money, so the window is finite. Choosing its width is the part that gets done by intuition and should be done by arithmetic.

The floor is the maximum age of a duplicate you could receive. Walk your own retry paths and take the longest:

If you accept webhooks with a nine-attempt, two-day schedule and keep a one-hour dedupe window, you have built something that works in testing and fails in exactly the case it exists for: the receiver was down for three hours, the sender's later attempts land outside your window, and you process the same event twice. The window must exceed the longest retry schedule that can reach it, with margin for the operator replay that nobody put in a design document.

The practical answer for most systems is 24 hours to 7 days, with two refinements:

• Compute the storage cost rather than fearing it. A dedupe row of a key, a hash, a small response and timestamps is on the order of 300–500 bytes with index overhead. At 1,000 writes per second for seven days that is roughly 600 million rows and a few hundred gigabytes — genuinely expensive. At 50 writes per second, the same seven days is around 30 million rows and 15 GB, which is nothing. Do this multiplication before choosing; the answer differs by two orders of magnitude between services and intuition tracks neither.
• Expire by dropping a partition, not by deleting rows. A DELETE FROM idempotency_keys WHERE created_at < now() - interval '7 days' on a high-write table generates dead tuples faster than autovacuum reclaims them, and Module P-9 walked that exact bloat cascade. Range-partition by day and DROP yesterday's partition — a metadata operation that unlinks files and costs nothing.

Never let Redis be the only dedupe layer for a financially significant effect. SET key 1 NX EX 86400 is a beautiful two-line deduplicator and it is a cache, which means an eviction under memory pressure, a failover to a replica missing recent writes, or a restart without persistence all silently reopen the window — and Module P-12 showed exactly how quietly a Redis instance reaches its memory ceiling. Redis in front of a database check is an excellent optimisation: it absorbs the duplicate storm cheaply and the database constraint remains the authority. Redis instead of the database check trades a duplicate charge against a memory graph you were not watching.

Finally, decide what happens after the window. An event arriving 8 days late with a key you have forgotten will be processed as new. Sometimes that is fine. Sometimes the correct behaviour is to reject anything older than the window outright — the timestamp check in Module P-17's signature verification is this rule applied at the edge, and it converts an unbounded correctness problem into a bounded one you can reason about.

---

Where This Shows Up in Your Stack

• PostgreSQL: the whole mechanism is PRIMARY KEY (tenant_id, endpoint, key) plus ON CONFLICT DO NOTHING, and the insert-as-claim pattern is why. INSERT ... ON CONFLICT takes the row lock the loser waits on, so two concurrent duplicates serialise without you writing a lock. Range-partition the table by day for cheap expiry (Module P-9). For a non-row claim — "only one import job per tenant may run" — pg_advisory_xact_lock(hashtext($1)) gives you a lock keyed on an arbitrary string, released automatically at transaction end. Note the interaction with isolation (Module P-1): under SERIALIZABLE a duplicate can surface as a 40001 serialisation failure rather than a conflict, so the retry loop and the idempotency check are the same code path.
• Node.js: generate the key with crypto.randomUUID() at the point of intent, and put it outside the retry loop — the commonest client bug is a key minted per attempt. Beware the in-memory Set or Map used as a deduplicator: it is per-process, so it does nothing once you run two instances, and it is unbounded, so it is a slow memory leak. AbortController timeouts are what create your ambiguous failures, so every place you set one is a place that needs a key.
• Redis: SET NX EX is the right first-line dedupe filter for high-volume, low-value events, and INCR is the canonical non-idempotent operation — a redelivered event double-counts, which is acceptable for a view counter and not for a quota that gates billing. Use it as a fast path in front of the database, never as the sole record for money.
• Kafka / SQS: Kafka's enable.idempotence prevents duplicates from producer retries only; transactions give exactly-once for Kafka-to-Kafka pipelines only. SQS FIFO's MessageDeduplicationId window is five minutes, which is shorter than most retry schedules — treat it as an optimisation, not a guarantee. In both, the consumer commits after processing (at-least-once) and therefore must be idempotent, which is the contract Module P-15 stated as a fork with only two branches.
• Stripe and payment APIs: they accept an Idempotency-Key header precisely because the ambiguous failure is unavoidable, and their window (24 hours for Stripe) is the bound on your correctness when you rely on it. Derive the downstream key deterministically from your own so a retry reproduces it — a randomly generated downstream key inside the retry path is the same bug as the client-side one, one layer down.

---

Summary

• Duplicates are not an anomaly, they are the cost of retries you already chose. The cause is the ambiguous failure: a caller that receives no response has learned nothing about whether the work happened, and no observation from its side distinguishes "never arrived" from "succeeded."
• Idempotent means the effect converges, not that the response matches. Absolute operations (SET, PUT, upsert, insert-with-constraint) are naturally idempotent; relative ones (amount + 40, INCR, POST /create) are not. Reaching for the absolute form is one line of SQL instead of a subsystem.
• Idempotency and deduplication are different tools. Idempotency makes repetition harmless and needs no memory; deduplication remembers what it has seen and therefore has a window that can expire. Prefer the first, and reserve the second for creations, relative adjustments, and external effects like email that have no absolute form.
• The idempotency key must come from the caller, be minted once outside the retry loop, and be scoped per tenant. A server-side body hash cannot distinguish two legitimate identical purchases from one retry. Store a request fingerprint with the key and return 422 on reuse with different content, because both silent options are worse.
• Claim the key with the insert, not with a read-then-write. INSERT ... ON CONFLICT DO NOTHING makes the uniqueness constraint the arbiter, so concurrent duplicates — which is how they actually arrive — serialise. Handle four distinct outcomes: fingerprint mismatch (422), in-flight with a live lease (409 + Retry-After), completed (replay the stored response), and in-flight with an expired lease, which without a locked_until poisons the key forever after a worker is killed mid-flight.
• Release the key on transient failures and store it on deterministic ones. Persisting a timeout as a permanent failure means the caller's legitimate retry replays an error forever and the charge that would have succeeded never does.
• Atomicity across a non-transactional boundary is the real problem, and no ordering of two writes survives a crash between them. The four honest strategies: propagate your key to the downstream so their dedupe covers your gap; record intent locally and drive the effect from a durable record (the outbox, Module A-3); reconcile against the downstream on a schedule; or restructure the effect so it has a natural unique key, which is why ledgers are append-only tables with UNIQUE (source_event_id) rather than mutable balances.
• Exactly-once delivery is provably unavailable; exactly-once effect is achievable as at-least-once delivery plus an idempotent receiver. Kafka's exactly-once is real but bounded to Kafka-to-Kafka transactions, and its idempotent producer deduplicates producer retries only — neither covers a card charge inside your consumer.
• A dedupe window shorter than the longest retry schedule that can reach it is decoration. Webhook senders retry for two days and operators replay batches by hand, so 24 hours to 7 days is the usual answer — chosen by multiplying row size by write rate, not by intuition, since the storage answer varies a hundredfold between services.
• Expire dedupe records by dropping a daily partition, not by DELETE, or the table bloats faster than autovacuum reclaims it. And never make Redis the only dedupe layer for money: an eviction or a failover silently reopens the window. In front of a database constraint it is an excellent optimisation; instead of one, it is a duplicate charge waiting on a memory graph nobody is watching.
• Duplicates cluster during incidents, when latency rises and every retry mechanism engages at once. That is why this is foundational rather than a refinement — it decides whether an availability incident stays one, or also becomes a financial one.

The next module, Webhooks and Callback Architectures, is this module applied across a boundary you do not control. As a sender you will retry for two days and therefore create duplicates by design; as a receiver you will get them from a sender whose behaviour you cannot change. Everything here — the stable event ID, the insert-as-claim, the window wide enough to cover the schedule — becomes the contract between two companies rather than two functions.

---