---

Priority Is Separate Queues, Not a Column

The instinct is a priority integer and ORDER BY priority DESC, run_after. It fails in two ways that a single queue cannot fix.

Starvation. A steady supply of high-priority work means low-priority jobs are never claimed. Not delayed — never. The nightly export sits behind a stream of password-reset emails indefinitely, and the symptom is a job that "seems stuck" with a perfectly healthy queue.

Head-of-line blocking. Priority orders the claim, not the capacity. Twenty workers on a shared queue, one tenant enqueues 50,000 low-priority image resizes at 8 seconds each, and every worker is busy for hours. A high-priority password reset arriving now is first in line for a worker that will not be free for two hours. Ordering has bought you nothing, because the constraint was never ordering.

The design is separate queues with dedicated worker pools — Module A-7's bulkhead, applied to worker capacity:

Now bulk saturating is a bulk problem. The cost is real and should be named: partitioning capacity means each pool is smaller, so an idle nightly pool cannot help a busy interactive one, and you have four sets of scaling parameters instead of one. Weighted work-stealing — a worker that prefers its own queue but may take from another when idle — recovers some of that utilisation at the cost of reintroducing partial coupling, and it is worth doing only once you can measure that the isolated pools are meaningfully idle.

One more axis that matters more than priority: separate by expected duration. Mixing 50 ms jobs with 8-second jobs in one queue means the fast ones inherit the slow ones' waiting time. Duration classes are usually a better partition than importance classes, because duration is what actually causes the blocking.

---

Worker Concurrency, and Why Prefetch Makes One Slow Worker Everyone's Problem

How many jobs should one worker process run at once? The answer depends entirely on what the jobs do, and getting it wrong produces opposite failures.

I/O-bound jobs — HTTP calls, database queries, S3 uploads — spend nearly all their time waiting, so concurrency should be high. Twenty or fifty concurrent jobs per process is reasonable, bounded by the thing they contend for. That bound is usually the database connection pool: 20 workers × 20 concurrent jobs × 1 connection each is 400 connections, and Module P-4 explained exactly what that does to one Postgres instance. The pool size, not the job count, is your real concurrency limit.

CPU-bound jobs — image resizing, PDF generation, big JSON transforms, compression — should be concurrency 1 per process, one process per core. In Node this is not a tuning preference but a correctness issue: a synchronous 400 ms image operation blocks the event loop, so every other job in that process stalls, and any HTTP server sharing the process stops answering health checks and gets removed from the load balancer (Module F-11). Put CPU-bound jobs in their own deployment, or in worker_threads, and never in the process serving traffic.

Then there is prefetch, which almost every broker enables by default and which quietly creates head-of-line blocking inside a single worker:

High prefetch is right when jobs are fast and uniform, because it amortises the fetch round trip. It is wrong when jobs are slow or variable, because reserved-but-unstarted work is invisible in queue depth while being very visible in latency. For jobs measured in seconds, prefetch 1 is usually correct; the round trip you save is a rounding error against the work.

---

Long-Running Jobs Need Checkpoints, Heartbeats, and a Way to Stop

A job that processes 40,000 CSV rows over twenty minutes breaks every assumption above. Three mechanisms make it operable.

Checkpoint, so a restart resumes rather than restarts. Record progress durably as you go, and make the resume idempotent:

Both the work and the cursor commit together, so a crash resumes at the last completed chunk. Without this, a failure at row 39,000 re-does 39,000 rows on every attempt, and a job that takes twenty minutes and fails at ninety percent may never complete within its attempt budget.

Heartbeat, so the lease does not expire under a healthy worker. A five-minute lease on a twenty-minute job means the lease expires at minute five, another worker claims the same job, and now two workers process the same CSV concurrently — usually producing duplicates rather than an error. The fix is for the worker to extend its lease periodically while working, and the rule is: the lease must be short relative to failure detection and long relative to a heartbeat interval. A 60-second lease refreshed every 20 seconds detects a dead worker in a minute while never expiring under a live one. Do not solve it by setting a two-hour lease, because then a worker that dies at minute one blocks that job for two hours.

Cancellation, which has to be cooperative. There is no safe way to kill a worker mid-transaction from outside. The pattern is a flag the job checks at chunk boundaries — a cancel_requested column, or a Redis key — plus honouring SIGTERM the same way. Which brings up the deploy path: a rolling deploy sends SIGTERM, and a worker that exits immediately abandons in-flight jobs to their leases (correct, but a 60-second delay for the user). A worker that finishes the current job and then exits, within the orchestrator's grace period, is the right behaviour — and if your jobs can exceed the grace period, that is another argument for chunking.

Why this matters in production: the failure that gets people is not a job crashing. It is the silent double execution — lease expired under a live worker, two workers, one CSV, 80,000 rows imported. There is no error to alert on, the job reports success, and the corruption is found by a user weeks later. Every long-running job needs the heartbeat, and every handler needs to be idempotent as the backstop for when the heartbeat itself was too slow.

---

Scheduling: Cron Fires Twice, or Not at All

Recurring work looks like the simplest part of this module and contains the most bugs, because every one of them involves time.

It fires N times, once per instance. A node-cron schedule inside your application process runs on all 12 instances. Your "nightly reconciliation" runs twelve times concurrently. This is the single most common scheduling bug, and it is usually discovered via duplicate emails to customers. The fixes, cheapest first:

or better, make the schedule a row and the tick a claim, so the lock and the record of what ran are the same thing:

The second form is better because it is also the answer to the next two problems. Best of all, give the enqueued job a dedupe_key of name:scheduled_slot — nightly-reconcile:2026-07-30T02:00Z — so the unique index makes duplicate enqueues impossible regardless of how many schedulers fire. That is Module P-16 applied to time, and it is the belt to the lock's braces.

It does not fire at all. The instance holding the schedule was mid-deploy at 02:00, or the machine was down, and in-process cron has no memory: the run is simply skipped, silently, and you find out when a report is missing. A schedule table fixes this because next_run_at <= now() is still true at 02:04 — the tick is late, not lost. Which forces a policy decision you should make explicitly per job: after downtime, does a missed run execute late, get skipped, or catch up all missed occurrences? An hourly aggregation missed six times probably wants one catch-up run covering six hours; a "send the daily digest" job almost certainly wants to skip rather than send six digests at once.

The previous run has not finished. A job scheduled every five minutes that occasionally takes eight creates overlapping executions, and overlapping executions of an aggregation produce double-counted output. Decide the overlap policy explicitly — usually "skip this tick if the previous is still running," implemented as a lock held for the job's duration rather than for the claim.

Timezones and DST. "Every day at 02:30" in a timezone with DST runs twice on one day of the year and zero times on another. Schedule in UTC and convert for display; if a job genuinely must run at a local wall-clock time — a market open, a business-day cutoff — store the timezone explicitly and accept that the anomalous days need a documented behaviour rather than a surprise. Month-end is the sibling bug: "the 31st" does not exist in seven months.

Everything scheduled on the hour runs on the hour. A thousand tenants' hourly syncs all set to :00 create a load spike sixty times larger than the average, plus a synchronised stampede against every downstream. Jitter the schedule per tenant — derive an offset from a hash of the tenant ID, so it is stable rather than random — and spread the work across the interval. Module P-12's thundering herd, produced by your own scheduler.

Delayed jobs are a different mechanism at different scales

"Run this in 30 seconds," "in 3 days," "in 6 months" are not the same problem:

A setTimeout for anything beyond the current request is not a delay mechanism; it dies with the process, silently, and Node's timer maximum is about 24.8 days anyway, after which it fires immediately. Durability is the whole requirement.

---

Measure Queue Age, Not Throughput

Job systems are where dashboards most often measure the wrong thing. Throughput — jobs completed per minute — looks like health and is not: a system completing 5,000 jobs a minute while 400,000 pile up is failing, and a system completing 40 a minute with an empty queue is fine.

The two metrics that matter:

• Age of the oldest unstarted job, per queue. This is your actual latency SLI and the only number that catches every failure — workers dead, workers saturated, a poison job cycling, a queue nobody deployed a consumer for. Alert on it.
• Queue depth, per queue. Its derivative is the useful part: depth rising steadily means arrival rate exceeds processing rate, and the time until catastrophe is computable rather than a surprise.

Both must be per queue, for the same reason Module P-17 insisted on per-endpoint webhook metrics: a global average hides a healthy default queue alongside a bulk queue eight hours behind. Add to those a job duration histogram per type (a p99 that doubles is a change in your data, not your code), a dead-job count per queue, and a retry rate per type.

And the pairing that makes autoscaling work: scale worker count on queue age, not on CPU. Workers waiting on I/O have low CPU while the queue backs up, so CPU-based autoscaling on a job fleet routinely scales down during a backlog. This also connects to Module A-8 — an unbounded queue is not backpressure, it is a delay in noticing. If depth exceeds what you could drain within your latency target, the honest responses are to shed enqueues (reject the request with 429, Module P-18), to degrade the feature, or to accept a stated longer SLA. Silently accumulating work you cannot do is the option that looks like none of the above and is the worst of them.

---

Where This Shows Up in Your Stack

• PostgreSQL: SELECT … FOR UPDATE SKIP LOCKED is what makes a database-backed queue viable, and a partial index (WHERE state='queued') keeps the claim query fast while the table grows to millions (Module F-13). Two hazards: the jobs table is high-churn, so every claim, retry and completion writes a new row version — keep completed jobs out of the hot table by deleting or partitioning them by day (Module P-9), or the bloat and index growth will show up as a slow claim query. And LISTEN/NOTIFY is a good latency optimisation over polling, but notifications are not durable and are lost if no session is listening, so it must be an accelerator on top of polling rather than the trigger.
• Node.js: never void-call async work after res.json() — a killed process loses it and an unhandled rejection can terminate the process. CPU-bound handlers block the event loop and must live in worker_threads or a separate deployment, not alongside the HTTP server whose health checks they will stall. Handle SIGTERM to finish the current job and exit inside the orchestrator's grace period, and remember Node's ~24.8-day setTimeout ceiling silently becomes "immediately."
• Redis / BullMQ: excellent latency and a good developer experience, with the caveat that Redis durability is a configuration decision (Module P-14) — a queue on an allkeys-lru instance can have jobs evicted, which is silent job loss, so job queues belong on their own instance with appendonly yes and no eviction policy that can touch them. Sorted sets scored by timestamp are the natural delayed-job structure.
• SQS / broker queues: the visibility timeout is the lease, with the same failure if it is shorter than your job runtime — extend it via ChangeMessageVisibility as a heartbeat rather than setting it to the maximum. DelaySeconds caps at 15 minutes, redrive policy plus DLQ is the dead-letter machinery, and there is no priority, so priority means separate queues, which is the design this module recommends anyway.
• Kafka: not a job queue, and using it as one is a recurring mistake (Module P-15). There is no per-message acknowledgement, no per-message retry, no delay primitive, and no way to skip one bad record without advancing past it — a poison record blocks its whole partition. Kafka is right for the event that triggers work; the work itself belongs in a queue with per-message semantics.
• Serverless / edge: Module F-10's constraint decides the architecture — no work after the response, hard execution ceilings, ephemeral disk. The enqueue must complete before you respond, long jobs must be chunked into invocations that each fit the ceiling with checkpoints in durable storage, and a scheduled function (EventBridge, Vercel Cron) fires once per schedule rather than once per instance, which removes the duplicate-cron problem and hands you the timeout problem instead.

---

Summary

• void doWork() after a response is not a background job. It has no durable record, no retry, no visibility, dies with the process, can crash it via unhandled rejection, and on serverless may never run at all (Module F-10). A job is a durable record, a lease, a bounded retry policy, and a terminal destination for failure.
• Enqueue in the same transaction as the write that justified it, or use an outbox (Module A-3). Publishing to an external broker before the transaction commits produces the "worker fetched a row that does not exist yet" bug that then succeeds on retry and reads as flakiness.
• Claim with FOR UPDATE SKIP LOCKED, or concurrent workers block on the same row and your twenty-process fleet runs jobs serially. Increment attempts at claim time, not on failure — otherwise a job that crashes the worker never records an attempt and retries forever, killing a worker each time.
• Every ownership must expire on its own. A worker killed mid-job cannot run a finally block, so without a lease the row sits at running permanently — the same failure as an idempotency claim without a lease (Module P-16) and a concurrency permit without a TTL (Module P-18).
• Split retry by error class: back off with jitter on transient failures, honour Retry-After on 429, and fail immediately to dead on permanent errors rather than spending eleven hours proving a validation error is still a validation error. max_attempts must be finite, because a poison job without a cap is a worker-killing loop.
• The dead-letter state is a product feature: alert on non-zero count per queue, retain payload and last error, and build the operator re-drive path on day one — the alternative during an incident is hand-written UPDATEs against production.
• Priority is separate queues with dedicated pools, not a priority column. A single queue starves low-priority work indefinitely and head-of-line blocks regardless of ordering, because ordering allocates the claim and the constraint is capacity. Partitioning by expected duration is usually more valuable than partitioning by importance.
• Concurrency follows the work: high for I/O-bound jobs, bounded by the database connection pool rather than by the job count (Module P-4); one per process for CPU-bound jobs, in a separate deployment, because a synchronous 400 ms operation blocks the event loop and stalls health checks. Keep prefetch at 1 for slow or variable jobs — reserved-but-unstarted work is invisible in queue depth and very visible in latency.
• Long jobs need checkpoints, heartbeats, and cooperative cancellation. Commit the cursor in the same transaction as the chunk. Keep the lease short and refresh it while working, because a lease that expires under a live worker gives you two workers on one CSV and duplicated data with no error anywhere — the failure mode that gets found weeks later by a user.
• Cron in-process fires once per instance. Use an advisory lock, or better a schedule table where claiming the tick and recording it are one UPDATE, plus a dedupe_key of name:scheduled_slot so duplicate enqueues are impossible. A schedule table also makes a missed run late rather than lost — then decide explicitly whether a missed run executes late, is skipped, or catches up.
• Decide the overlap policy, schedule in UTC, and jitter per tenant. Overlapping runs of an aggregation double-count; DST makes a 02:30 job run twice a year and never on another day; and a thousand tenants all syncing at :00 is a sixtyfold spike your own scheduler created.
• Delays have three mechanisms by scale — broker delay (SQS caps at 15 minutes), an indexed run_after column or Redis sorted set, and for weeks-to-months a durable row swept periodically. setTimeout is not durable, and beyond ~24.8 days it fires immediately.
• Alert on age of the oldest unstarted job, per queue — it is the one metric that catches dead workers, saturated workers, poison jobs, and a queue with no consumer deployed. Throughput looks like health and is not. Autoscale on queue age, not CPU, because I/O-bound workers have low CPU precisely while the backlog grows.

The next module, Search Systems, is where one of these queues earns its keep permanently: keeping a search index consistent with a database is a background-job problem before it is a search problem. It also starts the second half of Practitioner's data story — everything so far has been about getting data in and keeping it correct, and search, vector retrieval, OLAP and CDC are about getting it back out in shapes the write path was never designed to serve.

---