Running workers

Workers can run in the same process as your API (simple) or a separate process (recommended for production — independent scaling and crash isolation):

---

Stalled Jobs: Lock Renewal and Detection

When a worker picks up a job, it holds a Redis-backed lock on it for lockDuration (default 30s), renewing that lock automatically while the job keeps running. If the worker process dies, freezes on a blocking call, or gets killed mid-job by a deploy, the lock stops being renewed. Once it expires, BullMQ's stalledInterval check (default: every 30s) notices the job is locked but hasn't been renewed, and puts it back on the queue for another worker to pick up:

Two consequences worth knowing before they surprise you in production:

• A job that legitimately runs longer than lockDuration gets marked stalled and reassigned to another worker even though the original worker is still processing it fine. This is the classic BullMQ footgun for slow jobs — if a job routinely takes longer than the default 30s, raise lockDuration to comfortably exceed its expected runtime, not just nudge it up slightly.
• maxStalledCount caps how many times a job can stall before BullMQ gives up on it — a job that stalls repeatedly (e.g. the worker keeps crashing on it specifically) is moved to failed after this many stalls rather than bouncing between workers forever.

Because a recovered job can end up processed by two workers in an overlapping window, stalled-job recovery is another reason job processors need to be idempotent — not just retries after an outright failure.

---

Retries and Exponential Backoff

BullMQ, like virtually every job queue, guarantees at-least-once delivery — not exactly-once. A block-indexing worker once crashed mid-batch: it had written half its rows to Postgres but hadn't yet reached the point where the job gets acknowledged as complete. BullMQ, correctly, retried it. The processor wasn't idempotent, so the retry re-ran the whole batch — thousands of already-indexed transactions got inserted a second time, and every downstream balance calculation that summed those rows quietly went wrong. Nothing threw an error; the bug surfaced only when a reconciliation audit found balances that didn't match on-chain totals, until a unique constraint on (tx_hash, log_index) was added and started rejecting the duplicate inserts, which is what finally pointed at the retry as the cause.

When a job fails (throws an error), BullMQ retries it according to the attempts and backoff settings:

Override per-job for different retry strategies:

Dead letter queue pattern

After all retry attempts are exhausted, jobs land in BullMQ's failed set. For critical jobs, move them to a dedicated dead letter queue for manual review:

Idempotent Job Design

Retries mean "at least once," not "exactly once" — any job can run twice (a crash after the DB write but before the ack, a network blip during the completion handshake, a stalled-job recovery reassigning a job that's still finishing elsewhere). If a job's side effect isn't safe to repeat, a retry turns into a bug instead of a fix. Two patterns make a processor safe to re-run:

Pattern A: a unique constraint as the dedup gate. Let the database reject the duplicate instead of trying to detect it yourself:

The UNIQUE (tx_hash, log_index) constraint does the idempotency work — the second insert fails fast and cheaply, and the worker treats that specific failure as success instead of retrying again.

Pattern B: check-then-write with an idempotency key. Useful when the operation isn't a plain insert — e.g. calling a third-party payment API:

Derive the idempotency key from job.id (or a value stored in job.data) so that every retry of the same job reuses the same key. Most payment APIs accept an idempotency key argument for exactly this reason — use it instead of inventing your own dedup logic against their API.

---

Repeatable Jobs (Cron)

BullMQ supports repeatable jobs using cron syntax:

The worker for scheduled jobs is identical to any other worker. BullMQ handles the timing.

---

Multiple Queues by Priority

Separate queues for different workload types:

---

BullMQ Dashboard

Inspect queues in a web UI with Bull Board:

Visit /admin/queues to see job counts, retry failed jobs, clear queues, and inspect job data.

---

Real-World Patterns

Pattern 1: Always-on image processing

Pattern 2: Chaining jobs

Job instances aren't event emitters — there's no per-job .on('completed', ...). Listening for one specific job's completion means using a QueueEvents instance (or job.waitUntilFinished(), which uses QueueEvents under the hood):

This works for a simple two-step sequence, but it's a hand-rolled version of something BullMQ already ships. FlowProducer models job dependencies natively — the direction runs opposite to the waitUntilFinished chain above: children execute first, and the parent only starts once every child has completed, with each child's return value available to the parent via job.getChildrenValues(). That maps directly onto dependencies your own code can't express cleanly with sequential chaining — for example, an order can only ship once payment has cleared and an inventory hold has been confirmed, two independent precursor jobs rather than one:

Children can have their own children in turn, so FlowProducer also covers multi-level trees. For a plain "run A, then run B" sequence, waitUntilFinished chaining is still the simplest option; FlowProducer earns its keep once a step depends on more than one precursor completing, or the dependency graph goes deeper than one level — situations where nested waitUntilFinished calls get unwieldy fast.

Pattern 3: Bulk operations off the request path

---

Summary

• Job queues decouple slow/unreliable work from the request path. The response returns in milliseconds; the worker does the heavy lifting asynchronously.
• BullMQ uses Redis for persistence — jobs survive process restarts. Workers and producers are just Node.js processes connected to the same Redis instance.
• Exponential backoff retries transient failures automatically. Failed jobs after all attempts go to the failed set — move them to a dead letter queue for critical workflows.
• Concurrency and rate limiting prevent workers from overwhelming downstream services. Set concurrency per worker instance, limiter per queue.
• Repeatable jobs replace setInterval for cron-style work — they persist across restarts and won't double-schedule.
• Separate queues by workload type so a spike in image processing doesn't delay emails.
• Bull Board provides a web UI for inspecting, retrying, and clearing jobs without writing code.

Next: JSON internals — what JSON.parse and JSON.stringify actually do, the edge cases that bite at scale, streaming JSON for large payloads, and when MessagePack is worth the complexity.

---