What this module covers: A queue is the cheapest way to stop your request path from doing work it has no business doing synchronously — and the fastest way to acquire a whole new category of production bugs. This module covers what a queue actually buys you (decoupling, load levelling, retries), the honest truth about delivery semantics, visibility timeouts and the redelivery bug they cause, dead-letter queues as an operational obligation rather than a checkbox, why ordering and parallelism are the same dial pulling in opposite directions, poison messages and retry policy, queue depth as your earliest saturation signal, and when SELECT ... FOR UPDATE SKIP LOCKED in Postgres is genuinely the right queue.
A Queue Is How You Keep a Promise You Cannot Keep Yet
The endpoint that got you here looks like this. A user submits an order, and before responding you charge a card, send a confirmation email, notify the warehouse, update a search index, and push an event to analytics. Six dependencies, six timeouts, six ways to fail, and a p99 that is the sum of all of them — Module F-4's tail amplification, self-inflicted.
A queue changes the shape of that endpoint to: write the order, record the fact that five things need to happen, respond. The five things happen on someone else's clock.
That buys you three distinct things, and it is worth separating them because teams often want one and end up paying for all three:
1. Decoupling. The producer stops knowing who consumes, how many consumers there are, or whether they are up. You can add a sixth consumer without touching the checkout path. The cost is that you also stop knowing whether the work succeeded — the producer gets an "accepted", not a "done", and every "accepted" is a promise you now have to keep asynchronously. Status endpoints, job records, and user-visible "processing" states are all part of the bill.
2. Load levelling. This is the one people underrate. Your consumer fleet is sized for the average rate; the queue absorbs the peak. A Node instance doing one database query per job handles roughly 200–1,000 jobs/s (Module F-3's anchors), and a marketing send that dumps 2 million notifications in ten minutes would obliterate a synchronous path. Behind a queue it just makes the queue deep for a while. The cost is latency: work that used to complete in 50ms now completes in 50ms or forty minutes, depending on backlog, and the difference is invisible to the producer.
3. Retries with somewhere to retry from. A synchronous call that fails has nowhere to put the work except a log line and an apology. A queued message survives the failure of the consumer, the process, and the machine, because the message is durable and the consumer's acknowledgement is what removes it.
The Module F-10 payoff: durable enqueue is the only safe way to do post-response work
Module F-10 established the rule bluntly — no work after the response returns. On serverless the runtime may freeze your process the instant the response is flushed; on a container, a deploy or a scale-in kills it mid-flight; on any platform, an unhandled rejection in a fire-and-forget promise takes the work with it silently.
This module is where that rule gets its mechanism. The only reliable way to do work after responding is to make a durable record of the intent before responding, and to have something else pick it up.
ts
Note what that snippet quietly avoids. If you INSERT the order in Postgres and then call sqs.sendMessage(), you have two systems and no transaction across them: the order can commit and the enqueue can fail, or the enqueue can succeed and the transaction can roll back. That is exactly Module F-14's dual-write problem, and the general solution — a transactional outbox with a relay that publishes committed rows — is Module A-3's subject. The version above is the degenerate, and often sufficient, case where the queue is the database table.
Why this matters in production: the failure mode of fire-and-forget is not an error, it is an absence. No exception, no alert, no row — just a customer who never got their receipt, discovered three weeks later when they complain. A durable enqueue converts "silently didn't happen" into "is sitting in a queue, visibly, with a retry count".
Delivery Semantics: Pick Which Failure You Prefer
Every broker gives you a choice about when the message is considered delivered, and the choice is really about which side of a network partition you would rather be wrong on.
Semantic
Mechanism
What you lose
Honest use
At-most-once
Ack (or delete) before processing
Messages, whenever a consumer dies mid-work
Metrics, telemetry samples, cache warming — anything where a gap is cheaper than a duplicate
At-least-once
Ack after the work is durable
Nothing, but you get duplicates
Almost everything real: payments, emails, indexing, provisioning
"Exactly-once delivery"
Does not exist
—
—
The third row is not a rhetorical flourish. Exactly-once delivery is impossible over an unreliable network, and the reason is small enough to state in one paragraph: the consumer must tell the broker "I have it", that acknowledgement is itself a message, and it can be lost. If the broker never hears the ack, it has two options — redeliver (risking a duplicate) or not redeliver (risking a loss) — and no amount of protocol removes that fork, because the broker cannot distinguish "consumer died before processing" from "consumer processed and the ack was dropped".
What is achievable is exactly-once processing: at-least-once delivery plus a consumer whose effects are idempotent, so a second delivery of the same message produces no additional change. That is the whole game, and Module P-16 makes the full argument — dedupe keys, idempotency tables, natural idempotency versus enforced idempotency, and where the dedupe window has to live. For now, take the design rule: assume every message will be delivered more than once, and build the handler so that this is boring. Brokers advertising exactly-once are describing deduplication within a bounded window on their side, which is useful and is not the same claim.
Visibility Timeout Is a Lease, and Overrunning It Runs Your Job Twice
Here is the mechanism that makes at-least-once work, and it is a lease, not a lock.
When a consumer receives a message, the broker does not delete it. It hides it from other consumers for a fixed period — the visibility timeout (SQS), ack deadline (Pub/Sub), or consumer timeout (RabbitMQ), depending on the vendor. Three things can then happen:
The consumer acks within the window. The message is deleted. Done.
The consumer nacks, or crashes, or the connection drops. The message becomes visible again immediately or when the lease expires, and someone retries it.
The consumer is still working when the lease expires. The message becomes visible again, another consumer picks it up, and now the same job is running twice, concurrently.
Case 3 is the classic bug and it is worth dwelling on, because it is not a crash — it is a slow success.
Analogy: a kitchen order rail. The chef pulls a ticket off the rail and spikes it at their station, which stops anyone else picking it up. But the rule is "if a spiked ticket isn't marked done in ten minutes, put it back on the rail" — because the usual reason a ticket goes quiet is that the chef walked out. On a night when the fryer is slow, the ticket goes back up while the first chef is still cooking, a second chef starts the same dish, and two identical plates arrive at one table. Nobody was negligent. The timeout was calibrated for a normal night.
The production shape of this is unmistakable once you have seen it: duplicate charges, duplicate emails, two workers writing conflicting rows for the same entity, UPDATE ... SET status = 'processing' racing itself. And it appears precisely when things are already going badly, because that is when processing is slow — which means it correlates with your incidents rather than distributing itself politely across quiet Tuesdays.
Four mitigations, each with its cost:
Set the timeout from measured processing time, not from a guess. p99 of actual handler duration with headroom, and re-derive it when the handler changes. Cost: a longer timeout also means a longer delay before a genuinely crashed consumer's message is retried, so you are trading duplicate risk against recovery latency.
Heartbeat the lease while working. Extend the visibility timeout periodically from inside the handler. Cost: complexity, and a handler that hangs forever now holds its message forever instead of being retried — you need a hard ceiling.
Make the handler idempotent (Module P-16). Cost: a dedupe store, which is state, with its own TTL and its own failure modes.
Take a work lock keyed on the entity, not the message. Cost: this is a distributed lock, and Module A-2 explains why distributed locks are harder than they look — a lock whose lease also expires gets you back to the same problem one layer up.
ts
The comment on the ack line is the whole delivery-semantics table compressed into one statement: the position of your ack relative to your side effect is the choice between at-most-once and at-least-once. Nothing else about the broker configuration matters as much as that ordering.
A Dead-Letter Queue Nobody Monitors Is a Silent Data-Loss Queue
A message that fails repeatedly cannot be retried forever. After N attempts, the broker moves it to a dead-letter queue — a separate queue holding messages that could not be processed, with their failure metadata.
The DLQ is genuinely valuable. It stops one bad message from consuming your consumer capacity indefinitely, it preserves the payload so you can inspect and replay it, and it converts "this job keeps failing" from a log-grep exercise into a queryable set.
And it is the single most common place I see engineered data loss, for a reason that has nothing to do with the technology: the DLQ is a queue with no consumer. Messages arrive. Nothing reads them. Retention expires — often 4 to 14 days depending on the broker's configured maximum — and they are deleted. The system reported no errors, because moving a message to the DLQ is the successful completion of the retry policy.
So the operational rule, and treat it as non-negotiable:
Alert on DLQ depth greater than zero, not on a threshold. A threshold implies some number of permanently failed business operations is acceptable, and if that is genuinely true for this queue you should be discarding rather than dead-lettering.
Own the redrive. Someone must be able to inspect a dead-lettered message, decide whether the cause was transient or a bug, and replay it back to the source queue after the fix. If replay requires writing a script during an incident, it will not happen.
Never point a DLQ at another DLQ, and never wire a consumer that automatically redrives on a timer. That is an infinite retry loop with extra steps and a bigger bill.
Record why. The exception, the attempt count, and the consumer version. A DLQ full of payloads with no failure reason is an archive, not a tool.
Why this matters in production: with a DLQ that has an alert and an owner, a poison message is a ticket. Without one, it is a quiet subtraction from your data that surfaces months later as an unreconcilable total — and by then the payload has been deleted, so you cannot even determine what was lost.
Ordering and Parallelism Are the Same Dial
The uncomfortable structural fact about queues: you can have ordered processing, or you can have parallel consumers, and the amount you get of one is the amount you give up of the other.
The reason is mechanical. Ordering means message n must complete before message n+1 starts. That is a serial constraint. Parallelism means multiple messages in flight simultaneously. A standard queue with ten consumers gives you a tenfold throughput increase and no ordering guarantee whatsoever — messages can and will be processed out of sequence, and a retry reorders things further, since a failed message returns to the queue behind messages that arrived after it.
FIFO queues resolve this by scoping the ordering guarantee. You attach a message group ID to each message, and the broker guarantees:
Messages within the same group are delivered in order, one in flight at a time.
Different groups proceed independently and in parallel.
So the group ID is the dial. Choose it and you have chosen your ordering/parallelism trade-off:
Group ID choice
Ordering
Max useful consumers
Failure mode
One constant for everything
Total order
1
Throughput ceiling of a single consumer; one slow message blocks all traffic
Per entity (account:42, doc:99)
Per entity, which is usually what you needed
Number of distinct active entities
A hot entity becomes a serial bottleneck while other groups idle
Per message (unique)
None
Unbounded
You are paying FIFO's cost for standard-queue semantics
Per-entity is almost always the right answer, because the business requirement is almost never "global order". It is "do not apply this account's debit before its credit" or "do not process document v2 before v1". Those are per-key constraints, and scoping the group ID to the key gives you exactly the ordering you need and parallelism everywhere else.
Two costs to name explicitly:
Head-of-line blocking within a group. One message that keeps failing blocks every subsequent message in its group until it is retried out or dead-lettered. With per-account grouping, one broken account stalls that account only — which is the argument for fine-grained groups. With a single group, it stalls the business.
Ordering does not survive your handler. The broker delivers in order; if your handler acks and then hands work to a thread pool, an await-free background task, or another queue, you have discarded the guarantee downstream. Ordering is end-to-end or it is decorative.
If your requirement is durable, replayable, partition-ordered streams rather than a work queue — many independent consumer groups reading the same ordered log at their own pace — you want a log, not a queue. That is Module P-15.
Poison Messages, and Where Backoff Belongs
A poison message is one that will never succeed no matter how many times you retry: a malformed payload that throws on parse, a reference to a row that was deleted, a schema version the consumer does not understand, a job whose target account was closed. The defining property is that the failure is deterministic.
The reason poison messages deserve their own name is that the default retry configuration treats them as transient, and a message retried every few seconds forever is a self-inflicted denial of service on your own consumers. Ten poison messages in a queue with a 1-second retry delay is ten requests per second of pure waste, competing for the same consumer slots as real work, for as long as retention allows.
The fix is to classify the failure before deciding the retry policy — which is Module F-8's retryable-versus-terminal error shape, applied to consumers:
Terminal failures (validation error, unknown message type, referenced entity permanently gone) should go straight to the DLQ on the first attempt. Retrying a JSON.parse failure is superstition.
Transient failures (dependency timeout, 503, connection reset, lock contention) should retry with exponential backoff and jitter — Module A-6's mechanism, and the queue is the ideal place for it, because the delay costs you nothing: the message waits in durable storage rather than a sleeping process holding a connection.
Rate-limit responses (429 with Retry-After) should respect the header rather than the backoff schedule, and ideally slow the whole consumer rather than just that message, or you will spend your entire quota on retries.
Backoff interacts with queues in one way worth stating because it surprises people: backoff moves the retry into the future, which means your queue depth stops being a measure of pending work and becomes a measure of pending work plus scheduled work. A broker with per-message delay or a run_after column keeps these distinguishable. A broker without it, where you implement backoff by re-enqueueing with a delay, makes your depth metric noisier — which matters for the next section.
The other interaction is with maxReceiveCount. Exponential backoff with a base of 2 reaches long delays quickly, so a 5-attempt policy with a 30-second base spans roughly eight minutes, while a 10-attempt policy spans hours. Pick the attempt count against how long you are willing to leave the work undone, not by whether the number looks generous.
Queue Depth Is Your Earliest Honest Saturation Signal
This is the operational payoff of the whole module, and it is why queues are worth instrumenting well.
Latency and error rate are lagging indicators. By the time p99 has risen, the system is already degraded and users are already affected. CPU is misleading, because a consumer waiting on I/O is idle while being completely saturated in the sense that matters (Module F-4's utilisation curve is about the busy fraction of the bottleneck resource, which is rarely the CPU).
Queue depth leads. Specifically, the derivative does:
Depth flat and low. Consumers are keeping up. Fine.
Depth flat and high. Consumers are keeping up now, but you are permanently one incident behind — there is a persistent backlog absorbing your headroom.
Depth rising monotonically. Arrival rate exceeds service rate. This is not a spike, it is a deficit, and it will not resolve on its own. Nothing about the request path has broken yet, and it will.
Oldest-message age rising while depth is flat. Head-of-line blocking, or a starved message group. Depth alone hides this, which is why you alert on age of the oldest unacked message, not only on count.
That last metric is the honest one, because it is expressed in the unit the business cares about: how stale is the most delayed piece of work. "12,000 messages queued" means nothing without a service rate. "Oldest message is 40 minutes old" means the customer who ordered 40 minutes ago has no confirmation.
Little's Law from Module F-4 turns depth into a forecast directly. With arrival rate λ and total service capacity μ, a backlog L drains in roughly L / (μ − λ) — and if λ ≥ μ the answer is "never", which is the number you want on a dashboard during an incident.
ts
What you do when depth rises is Module A-8's subject — backpressure and load shedding, including the genuinely hard question of whether to reject new work at the producer to protect the consumers. The queue-specific version of that decision: a queue with unbounded depth has converted a fast failure into an unbounded latency, and unbounded latency is often worse for the user than a clear rejection. Bounded queues that reject on full are an honest design, not a limitation.
When Postgres Is Genuinely the Right Queue
Module F-14 made the one-system argument: every additional datastore is another thing to operate, monitor, back up, secure, upgrade, and page someone about at 3am, and that cost is paid continuously while the benefit is often speculative. Queues are where this argument has the most teeth, because Postgres has a genuinely good queue primitive.
sql
Three things to notice. SKIP LOCKED is what makes this scale past one worker — without it, ten workers all take a lock on the same oldest row and nine of them wait, serialising your entire fleet. leased_until is you implementing visibility timeout manually, which means a sweeper query that returns expired leases to ready is now your responsibility. And run_after is where backoff and scheduled retries live, which keeps them queryable in a way a broker's opaque delay does not.
Postgres is the right queue when:
Throughput is within its envelope. Each claim-and-complete cycle is a couple of committed writes, and Postgres does roughly 1K–5K TPS with fsync-on-commit, so a job queue in the low hundreds per second is comfortable and a few thousand per second is a tuning project.
You need the enqueue to be transactional with a business write. This is the strongest reason. INSERT INTO orders and INSERT INTO jobs in one transaction has no dual-write problem, no outbox relay, and no window where one committed and the other did not. Getting this property with an external broker requires Module A-3's machinery.
You want to query your queue. "How many jobs for tenant 42 are pending", "show me every failed payout job from Tuesday", "cancel all jobs for this deleted account" are one SELECT each. Most brokers cannot answer any of them.
The team is small and already operates Postgres well. One system you understand deeply beats two you understand adequately.
Postgres is the wrong queue when:
You need high fan-out or pub/sub. Many independent consumer groups reading the same messages at their own offsets is a log's job (Module P-15), and simulating it with per-consumer job rows multiplies your write volume by the number of consumers.
Throughput is genuinely high. Tens of thousands of messages per second means tens of thousands of row updates per second on one hot table, plus the MVCC consequence from Module F-13: every claim is an UPDATE, every UPDATE is a dead tuple, and a high-churn queue table bloats fast enough that autovacuum tuning becomes a standing chore.
Retention or replay matters. Keeping seven days of processed messages for replay means a large, ever-growing table competing for cache with your live data.
You would be reimplementing the broker. Visibility timeouts, DLQs, backoff scheduling, priority, delayed delivery, metrics, and a sweeper are each 20 lines and collectively a product. If you find yourself building the sixth feature, the one-system argument has stopped applying — you now operate two queue systems, and one of them is yours.
Consumers hold connections while polling. Naive polling with a 100ms interval across 50 workers is 500 queries/s of pure overhead; LISTEN/NOTIFY fixes the latency but each listening connection is a forked backend process (Module F-13), which interacts badly with Module P-4's pool arithmetic.
The honest summary: for most applications' background work, Postgres with SKIP LOCKED is the correct first choice, and the migration to a dedicated broker is a decision you make when a named constraint above starts binding — not in advance.
Where This Shows Up in Your Stack
PostgreSQL:SELECT ... FOR UPDATE SKIP LOCKED for the claim, a leased_until sweeper for expired leases, run_after for backoff and scheduled work, and a partial index on (run_after) WHERE state = 'ready' so the claim query never scans completed rows. Watch dead tuples on the jobs table — it is the most write-churned table you will own.
Node.js: the consumer is a long-lived process, so it needs an explicit shutdown path: stop fetching on SIGTERM, finish in-flight messages, then exit — otherwise every deploy converts in-flight work into redeliveries. Concurrency per consumer must be bounded against the database pool (Module P-4), because 200 concurrent handlers and 20 connections is pool exhaustion with extra steps.
Redis: Streams with consumer groups give you group semantics, per-consumer pending-entry lists, and XAUTOCLAIM for lease recovery at 100K+ ops/s. The cost is that durability is your configuration decision — an unreplicated instance with default persistence can lose acknowledged messages on failure, which is fine for cache warming and not fine for payments.
SQS / Cloud Pub/Sub: managed durability and effectively unlimited depth, so the operational failure mode shifts from "the broker fell over" to "nobody noticed the DLQ" and "egress and request costs scaled with our retry policy" (Module O-8). Long polling instead of tight polling loops; it is both faster and cheaper.
Summary
A queue buys decoupling, load levelling, and durable retries — and costs you end-to-end visibility, unbounded worst-case latency, and a second execution environment to operate and observe.
Durable enqueue before responding is the only safe way to do post-response work, which is the mechanism Module F-10's "no work after the response returns" rule was waiting for. Enqueue in the same transaction as the business write, or you have recreated Module F-14's dual-write problem.
At-most-once means ack first; at-least-once means ack after the effect is durable. The position of the ack is the entire choice.
Exactly-once delivery is impossible because the acknowledgement is itself a losable message. Exactly-once processing is achievable through idempotency, and Module P-16 makes the full argument.
Visibility timeout is a lease, not a lock. A handler slower than its lease gets redelivered and runs twice concurrently — the classic bug, and it correlates with incidents because that is when handlers are slow. Calibrate from measured p99, heartbeat the lease, and make the handler idempotent anyway.
A DLQ nobody monitors is a silent data-loss queue. Alert on depth greater than zero, give it an owner and a rehearsed redrive path, and record the failure reason — dead-lettering is the successful completion of your retry policy, so it produces no errors.
Ordering and parallelism are the same dial. Message group IDs scope the guarantee; per-entity grouping gives the ordering you actually needed with parallelism everywhere else, at the cost of head-of-line blocking within a hot group. Ordering is end-to-end or it is decorative.
Classify failures before retrying. Terminal failures go to the DLQ on attempt one; transient failures get Module A-6's exponential backoff with jitter, which is nearly free in a queue because the message waits in storage rather than in a sleeping process.
Queue depth, and especially oldest-message age, is the earliest honest saturation signal — leading where latency and error rate lag. Little's Law turns backlog into a drain-time estimate; λ ≥ μ means "never". What to do about it is Module A-8.
Postgres with SKIP LOCKED is genuinely the right queue for transactional enqueue, queryable state, and moderate throughput — and the wrong one for fan-out, replay, very high throughput, or when you notice you are building the sixth broker feature yourself.
The next module, Event Streaming with Kafka, takes the one guarantee this module could not give you — that a message survives being consumed, so many independent consumers can read the same ordered history at their own pace — and shows what it costs: partitions as the unit of both ordering and parallelism, consumer group rebalancing, and offsets as the consumer's own state rather than the broker's.
Knowledge Check
A payment-capture consumer normally finishes in 8 seconds against a 30-second visibility timeout. During a vendor slowdown, captures take 45 seconds. Support reports customers charged twice, and the logs show two workers processing the same message ID within the same minute. What happened, and what is the correct fix?
A team migrates to a FIFO queue after an out-of-order bug corrupted account balances. They assign every message the group ID "default". Correctness is fixed, but throughput drops from 800 to about 40 messages per second and adding consumers changes nothing. What does the module say?
A four-engineer team runs one Postgres instance and needs background jobs for emails, PDF generation, and webhook delivery — a few hundred jobs per second at peak, with a hard requirement that a job must never exist for an order that rolled back. They are debating a managed broker. What does the module recommend, and what is the cost?
Test your knowledge with more question sets
Sign in to access a wider variety of questions and get notified when new practice sets are added to this module.
// Wrong. The response is on the wire; the process may be frozen or killed// before the email call completes, and nothing anywhere records that it didn't.app.post('/orders',async(req, res)=>{const order =await db.orders.insert(req.body); res.status(201).json(order);awaitsendConfirmationEmail(order);// Module F-10: this work is not guaranteed});// Right. The intent is durable before the response, and it commits atomically// with the business write.app.post('/orders',async(req, res)=>{const order =await db.tx(async(tx)=>{const o =await tx.orders.insert(req.body);// Same transaction as the order. Either both rows exist or neither does — which is// how you avoid Module F-14's dual-write problem between your DB and the broker.await tx.jobs.insert({ kind:'send_confirmation', dedupe_key:`confirm:${o.id}`,// Module P-16 uses this to make retries safe payload:{ orderId: o.id }, run_after:newDate(),});return o;}); res.status(201).json(order);// now the promise is backed by durable state});
constLEASE_SECONDS=60;asyncfunctionhandleMessage(msg: Message){// Extend the lease every 20s while we work. Without this, any handler slower than// LEASE_SECONDS gets redelivered and runs concurrently with itself.const heartbeat =setInterval(()=>{ broker.changeVisibility(msg.receiptHandle,LEASE_SECONDS).catch(()=>{/* log and continue: a failed extension means we may be about to be redelivered */});},20_000);try{// Idempotency is what makes a redelivery harmless rather than a duplicate charge.awaitwithDedupe(msg.dedupeKey,()=>doWork(msg.body));// Ack only after the side effect is durable. Acking first is at-most-once.await broker.ack(msg.receiptHandle);}finally{clearInterval(heartbeat);}}
// The two numbers worth alerting on, and why they differ.const backlog =await broker.approximateDepth(queue);// countconst oldestAgeSec =await broker.approximateOldestAge(queue);// secondsconst drainSeconds = serviceRate > arrivalRate
? backlog /(serviceRate - arrivalRate)// Little's Law: finite means recoverable:Infinity;// λ >= μ: adding consumers is the only fix
-- Claim a batch of jobs. SKIP LOCKED is the whole trick: concurrent workers step over-- rows already locked by a peer instead of blocking behind them.WITH claimed AS(SELECT id
FROM jobs
WHERE state ='ready'AND run_after <=now()-- backoff and scheduling live in this columnORDERBY priority DESC, run_after
FORUPDATE SKIP LOCKED -- without SKIP LOCKED every worker queues on row 1LIMIT20)UPDATE jobs j
SET state ='running', attempts = attempts +1, leased_until =now()+interval'5 minutes'-- your visibility timeout, by handFROM claimed c
WHERE j.id = c.id
RETURNING j.id, j.kind, j.payload, j.attempts;