Module P-14·34 min read

Decoupling, at-least-once delivery, visibility timeouts, dead-letter queues, and exactly how much ordering a queue can actually promise you.

JJS
Written by Jatin Jain Saraf · Senior Software Engineer

Module P-14 — Message Queues

What this module covers: A queue is the cheapest way to stop your request path from doing work it has no business doing synchronously — and the fastest way to acquire a whole new category of production bugs. This module covers what a queue actually buys you (decoupling, load levelling, retries), the honest truth about delivery semantics, visibility timeouts and the redelivery bug they cause, dead-letter queues as an operational obligation rather than a checkbox, why ordering and parallelism are the same dial pulling in opposite directions, poison messages and retry policy, queue depth as your earliest saturation signal, and when SELECT ... FOR UPDATE SKIP LOCKED in Postgres is genuinely the right queue.


A Queue Is How You Keep a Promise You Cannot Keep Yet

The endpoint that got you here looks like this. A user submits an order, and before responding you charge a card, send a confirmation email, notify the warehouse, update a search index, and push an event to analytics. Six dependencies, six timeouts, six ways to fail, and a p99 that is the sum of all of them — Module F-4's tail amplification, self-inflicted.

A queue changes the shape of that endpoint to: write the order, record the fact that five things need to happen, respond. The five things happen on someone else's clock.

That buys you three distinct things, and it is worth separating them because teams often want one and end up paying for all three:

1. Decoupling. The producer stops knowing who consumes, how many consumers there are, or whether they are up. You can add a sixth consumer without touching the checkout path. The cost is that you also stop knowing whether the work succeeded — the producer gets an "accepted", not a "done", and every "accepted" is a promise you now have to keep asynchronously. Status endpoints, job records, and user-visible "processing" states are all part of the bill.

2. Load levelling. This is the one people underrate. Your consumer fleet is sized for the average rate; the queue absorbs the peak. A Node instance doing one database query per job handles roughly 200–1,000 jobs/s (Module F-3's anchors), and a marketing send that dumps 2 million notifications in ten minutes would obliterate a synchronous path. Behind a queue it just makes the queue deep for a while. The cost is latency: work that used to complete in 50ms now completes in 50ms or forty minutes, depending on backlog, and the difference is invisible to the producer.

3. Retries with somewhere to retry from. A synchronous call that fails has nowhere to put the work except a log line and an apology. A queued message survives the failure of the consumer, the process, and the machine, because the message is durable and the consumer's acknowledgement is what removes it.

The Module F-10 payoff: durable enqueue is the only safe way to do post-response work

Module F-10 established the rule bluntly — no work after the response returns. On serverless the runtime may freeze your process the instant the response is flushed; on a container, a deploy or a scale-in kills it mid-flight; on any platform, an unhandled rejection in a fire-and-forget promise takes the work with it silently.

This module is where that rule gets its mechanism. The only reliable way to do work after responding is to make a durable record of the intent before responding, and to have something else pick it up.

ts

Note what that snippet quietly avoids. If you INSERT the order in Postgres and then call sqs.sendMessage(), you have two systems and no transaction across them: the order can commit and the enqueue can fail, or the enqueue can succeed and the transaction can roll back. That is exactly Module F-14's dual-write problem, and the general solution — a transactional outbox with a relay that publishes committed rows — is Module A-3's subject. The version above is the degenerate, and often sufficient, case where the queue is the database table.

Why this matters in production: the failure mode of fire-and-forget is not an error, it is an absence. No exception, no alert, no row — just a customer who never got their receipt, discovered three weeks later when they complain. A durable enqueue converts "silently didn't happen" into "is sitting in a queue, visibly, with a retry count".


Delivery Semantics: Pick Which Failure You Prefer

Every broker gives you a choice about when the message is considered delivered, and the choice is really about which side of a network partition you would rather be wrong on.

SemanticMechanismWhat you loseHonest use
At-most-onceAck (or delete) before processingMessages, whenever a consumer dies mid-workMetrics, telemetry samples, cache warming — anything where a gap is cheaper than a duplicate
At-least-onceAck after the work is durableNothing, but you get duplicatesAlmost everything real: payments, emails, indexing, provisioning
"Exactly-once delivery"Does not exist

The third row is not a rhetorical flourish. Exactly-once delivery is impossible over an unreliable network, and the reason is small enough to state in one paragraph: the consumer must tell the broker "I have it", that acknowledgement is itself a message, and it can be lost. If the broker never hears the ack, it has two options — redeliver (risking a duplicate) or not redeliver (risking a loss) — and no amount of protocol removes that fork, because the broker cannot distinguish "consumer died before processing" from "consumer processed and the ack was dropped".

What is achievable is exactly-once processing: at-least-once delivery plus a consumer whose effects are idempotent, so a second delivery of the same message produces no additional change. That is the whole game, and Module P-16 makes the full argument — dedupe keys, idempotency tables, natural idempotency versus enforced idempotency, and where the dedupe window has to live. For now, take the design rule: assume every message will be delivered more than once, and build the handler so that this is boring. Brokers advertising exactly-once are describing deduplication within a bounded window on their side, which is useful and is not the same claim.


Visibility Timeout Is a Lease, and Overrunning It Runs Your Job Twice

Here is the mechanism that makes at-least-once work, and it is a lease, not a lock.

When a consumer receives a message, the broker does not delete it. It hides it from other consumers for a fixed period — the visibility timeout (SQS), ack deadline (Pub/Sub), or consumer timeout (RabbitMQ), depending on the vendor. Three things can then happen:

  1. The consumer acks within the window. The message is deleted. Done.
  2. The consumer nacks, or crashes, or the connection drops. The message becomes visible again immediately or when the lease expires, and someone retries it.
  3. The consumer is still working when the lease expires. The message becomes visible again, another consumer picks it up, and now the same job is running twice, concurrently.

Case 3 is the classic bug and it is worth dwelling on, because it is not a crash — it is a slow success.

Analogy: a kitchen order rail. The chef pulls a ticket off the rail and spikes it at their station, which stops anyone else picking it up. But the rule is "if a spiked ticket isn't marked done in ten minutes, put it back on the rail" — because the usual reason a ticket goes quiet is that the chef walked out. On a night when the fryer is slow, the ticket goes back up while the first chef is still cooking, a second chef starts the same dish, and two identical plates arrive at one table. Nobody was negligent. The timeout was calibrated for a normal night.

The production shape of this is unmistakable once you have seen it: duplicate charges, duplicate emails, two workers writing conflicting rows for the same entity, UPDATE ... SET status = 'processing' racing itself. And it appears precisely when things are already going badly, because that is when processing is slow — which means it correlates with your incidents rather than distributing itself politely across quiet Tuesdays.

Four mitigations, each with its cost:

  • Set the timeout from measured processing time, not from a guess. p99 of actual handler duration with headroom, and re-derive it when the handler changes. Cost: a longer timeout also means a longer delay before a genuinely crashed consumer's message is retried, so you are trading duplicate risk against recovery latency.
  • Heartbeat the lease while working. Extend the visibility timeout periodically from inside the handler. Cost: complexity, and a handler that hangs forever now holds its message forever instead of being retried — you need a hard ceiling.
  • Make the handler idempotent (Module P-16). Cost: a dedupe store, which is state, with its own TTL and its own failure modes.
  • Take a work lock keyed on the entity, not the message. Cost: this is a distributed lock, and Module A-2 explains why distributed locks are harder than they look — a lock whose lease also expires gets you back to the same problem one layer up.
ts

Sign in to keep reading

The rest of this module is free — sign in with Google to unlock it and track your progress.

Sign in & Register

Discussion

0

Join the discussion

Loading comments...

© 2026 Jatin Jain Saraf (JJS). All rights reserved.