P-6 — BullMQ Internals: The Redis Data Structures Behind the Job Queue

Who this module is for: You use BullMQ (or Bull) for job queues and have run into issues — jobs that get stuck, queues that slow down under load, stalled job detection that is too aggressive or not aggressive enough. This module explains the Redis data structures BullMQ uses for every queue state, so you can reason about its behaviour, tune it correctly, and debug it at the Redis level.

Why Understanding BullMQ Internals Matters

BullMQ is a job queue built on Redis. Most engineers treat it as a black box — they add jobs with queue.add() and process them in a worker.process() function. But when queues misbehave (jobs stay in "active" forever, delayed jobs fire late, rate limits fail), you cannot diagnose or fix the problem without understanding the Redis layer.

Every BullMQ behaviour maps to specific Redis operations. Knowing this lets you:

• Query queue state directly with redis-cli without going through BullMQ's API
• Understand why a job is "stuck" and fix it
• Tune TTL, stall checks, and rate limiter settings appropriately
• Identify Redis memory usage caused by large queues

The Key Schema

BullMQ uses a namespaced key prefix. For a queue named emails:

Job Lifecycle in Redis

Adding a Job (queue.add)

What happens in Redis:

1. INCR bull:emails:id → generates job ID, e.g., 42
2. HSET bull:emails:42 with all job fields:
   • id: "42"
   • name: "send-welcome"
   • data: '{"userId":"1001","email":"j@example.com"}'
   • opts: '{"attempts":1,"delay":0,...}'
   • timestamp: "1717000000000"
   • delay: "0"
   • priority: "0"
3. RPUSH bull:emails:wait 42 → add job ID to the wait list
4. XADD bull:emails:events * event added jobId 42 → emit event to the events stream

The job data (step 2) is stored in a Hash for O(1) field access. The queue lists and sorted sets store only the job ID — the actual data is always in the Hash.

Adding a Delayed Job

Instead of RPUSH bull:emails:wait, BullMQ uses:

A scheduler process (the QueueScheduler in Bull v3, built into BullMQ workers) polls the delayed sorted set with:

When jobs become ready (their score ≤ current timestamp), the scheduler moves them to bull:emails:wait via LPUSH and ZREM.

Adding a Priority Job

Workers preferentially consume from prioritized before wait.

Processing a Job (worker picks up)

The worker calls:

This atomically moves the job ID from the tail of wait to the head of active. If no jobs are waiting, the worker calls:

Blocking for up to 5 seconds. When a job arrives, the BLMOVE completes and the job ID is in active.

The worker then reads the job data:

And acquires a "lock" on the job:

This lock prevents another worker from claiming the same job. The lock expires in 30 seconds (configurable with lockDuration).

Job Completion

BullMQ executes a Lua script that atomically:

1. Verifies the worker still holds the lock (GET bull:emails:{jobId}:lock)
2. LREM bull:emails:active 0 {jobId} — removes from active list
3. ZADD bull:emails:completed {timestamp} {jobId} — adds to completed set
4. Optionally trims completed set if removeOnComplete is configured
5. DEL bull:emails:{jobId}:lock — releases the lock
6. XADD bull:emails:events * event completed jobId {jobId} — emits event

Job Failure

Similar Lua script:

1. Verify lock
2. LREM bull:emails:active 0 {jobId}
3. If retries remain: RPUSH bull:emails:wait {jobId} (or with backoff delay: ZADD bull:emails:delayed ...)
4. If no retries remain: ZADD bull:emails:failed {timestamp} {jobId}
5. Update job Hash with failedReason, stacktrace, attemptsMade
6. Release lock, emit event

Stalled Job Detection

A job becomes "stalled" when the worker crashes (SIGKILL, OOM) after moving the job to active but before completing or failing it. The lock expires but no worker claims the job — it is stuck in active indefinitely.

The stall check runs periodically (configurable with stalledInterval, default 30 seconds):

The Lua-based stall check:

1. Scans bull:emails:active for job IDs
2. For each: checks if bull:emails:{jobId}:lock exists
3. If the lock does not exist (expired): the job is stalled
4. If attemptsMade < maxAttempts: moves back to wait (retry)
5. If exhausted retries: moves to failed

Tuning stall detection:

• lockDuration should be longer than the maximum expected job processing time
• lockRenewTime is automatically set to lockDuration / 2 — the worker renews its lock halfway through the duration
• If a job legitimately takes 5 minutes: set lockDuration: 360000 (6 minutes)
• maxStalledCount: 0 means stalled jobs are retried indefinitely (dangerous for infinite loops)

Rate Limiter Internals

BullMQ's rate limiter uses a sliding window implemented with a Sorted Set:

Before processing each job, the worker:

1. Removes entries older than duration ms from the rate limiter key
2. Counts remaining entries
3. If count ≥ max: delays the current job by inserting it back into delayed for the next window
4. Otherwise: increments the window counter and proceeds

Querying Queue State Directly

With this knowledge, you can inspect BullMQ queues using raw Redis commands:

Memory Considerations

For high-throughput queues, BullMQ keys accumulate:

• Completed jobs: bull:emails:{jobId} Hashes persist after completion unless removeOnComplete is set
• Failed jobs: Same — persist forever unless removeOnFail

Without this, a queue processing 1,000 jobs/hour generates 24,000 job Hashes per day. Each Hash is ~300–500 bytes. At 1M jobs total: ~300–500MB just for the job Hashes.

The completed and failed Sorted Sets also grow unboundedly. removeOnComplete.count limits the Sorted Set size by trimming (ZREMRANGEBYRANK) after each completion.

Summary

• BullMQ uses wait (List) for FIFO queuing, active (List) for in-progress jobs, completed/failed (Sorted Sets) for history, delayed (Sorted Set with timestamp score) for scheduling
• Job data lives in a Hash bull:{queue}:{jobId}; queues store only the ID
• Workers use LMOVE wait active (atomic) to claim jobs; a Lua-based lock prevents double-processing
• Stalled jobs (lock expired, still in active) are detected and retried or failed by the stall checker
• Tune lockDuration to exceed max job processing time; lockRenewTime defaults to half lockDuration
• Rate limiting uses a sliding window Sorted Set — delayed jobs are re-queued when the window is full
• Enable removeOnComplete and removeOnFail to prevent unbounded memory growth
• Query queue state directly with Redis commands for debugging without the BullMQ API overhead

Next: P-7 — Cache Stampede, Avalanche, and Penetration — three cache failure modes that look similar in monitoring but require different solutions.