BullMQ Internals: The Redis Data Structures Behind the Job Queue22 min read
Module P-6·22 min read
How BullMQ maps job lifecycle to Sorted Sets, Lists, and Hashes. Worker polling, delayed job scheduling, stalled job detection via heartbeat, the rate limiter internals, and choosing BullMQ vs raw Streams.
P-6 — BullMQ Internals: The Redis Data Structures Behind the Job Queue
Who this module is for: You use BullMQ (or Bull) for job queues and have run into issues — jobs that get stuck, queues that slow down under load, stalled job detection that is too aggressive or not aggressive enough. This module explains the Redis data structures BullMQ uses for every queue state, so you can reason about its behaviour, tune it correctly, and debug it at the Redis level.
Why Understanding BullMQ Internals Matters
BullMQ is a job queue built on Redis. Most engineers treat it as a black box — they add jobs with queue.add() and process them with a processor function passed straight into new Worker(queueName, processorFn). But when queues misbehave (jobs stay in "active" forever, delayed jobs fire late, rate limits fail), you cannot diagnose or fix the problem without understanding the Redis layer.
Every BullMQ behaviour maps to specific Redis operations. Knowing this lets you:
Query queue state directly with redis-cli without going through BullMQ's API
Understand why a job is "stuck" and fix it
Tune TTL, stall checks, and rate limiter settings appropriately
Identify Redis memory usage caused by large queues
The Key Schema
BullMQ uses a namespaced key prefix. For a queue named emails:
text
Job Lifecycle in Redis
Adding a Job (queue.add)
javascript
What happens in Redis:
INCR bull:emails:id → generates job ID, e.g., 42
HSET bull:emails:42 with all job fields:
id: "42"
name: "send-welcome"
data: '{"userId":"1001","email":"j@example.com"}'
opts: '{"attempts":1,"delay":0,...}'
timestamp: "1717000000000"
delay: "0"
priority: "0"
RPUSH bull:emails:wait 42 → add job ID to the wait list
XADD bull:emails:events * event added jobId 42 → emit event to the events stream
The job data (step 2) is stored in a Hash for O(1) field access. The queue lists and sorted sets store only the job ID — the actual data is always in the Hash.
A scheduler process (the QueueScheduler class — a BullMQ-specific concept used before BullMQ 2.0, since merged directly into the Worker itself) polls the delayed sorted set with:
If retries remain: RPUSH bull:emails:wait {jobId} (or with backoff delay: ZADD bull:emails:delayed ...)
If no retries remain: ZADD bull:emails:failed {timestamp} {jobId}
Update job Hash with failedReason, stacktrace, attemptsMade
Release lock, emit event
Stalled Job Detection
A job becomes "stalled" when the worker crashes (SIGKILL, OOM) after moving the job to active but before completing or failing it. The lock expires but no worker claims the job — it is stuck in active indefinitely.
The stall check runs periodically (configurable with stalledInterval, default 30 seconds):
javascript
The Lua-based stall check:
Scans bull:emails:active for job IDs
For each: checks if bull:emails:{jobId}:lock exists
If the lock does not exist (expired): the job is stalled
If attemptsMade < maxAttempts: moves back to wait (retry)
If exhausted retries: moves to failed
javascript
Tuning stall detection:
lockDuration should be longer than the maximum expected job processing time
lockRenewTime is automatically set to lockDuration / 2 — the worker renews its lock halfway through the duration
If a job legitimately takes 5 minutes: set lockDuration: 360000 (6 minutes)
maxStalledCount: 0 means stalled jobs are retried indefinitely (dangerous for infinite loops)
Rate Limiter Internals
javascript
BullMQ's rate limiter is a global limiter shared across every worker consuming a queue — it isn't scoped per job. It's backed by a simple key holding a counter and a TTL, closer to a fixed-window than a per-job sliding window:
bull:emails:limiter → String: counter, with a TTL of `duration` ms
Before processing each job, the worker:
Checks whether the shared limiter key exists and how many jobs it's already counted within the current window
If the count has reached max for the current window: the worker pauses pulling new jobs from this queue until the window's TTL expires, rather than delaying that one job individually
Otherwise: it increments the counter (setting the TTL on first increment) and proceeds
The practical effect is the same as what most readers actually want — no more than max jobs processed per duration across all workers combined — but the mechanism is a shared counter with a TTL, not a per-job Sorted Set keyed by job ID.
Querying Queue State Directly
With this knowledge, you can inspect BullMQ queues using raw Redis commands:
bash
Memory Considerations
For high-throughput queues, BullMQ keys accumulate:
Completed jobs:bull:emails:{jobId} Hashes persist after completion unless removeOnComplete is set
Failed jobs: Same — persist forever unless removeOnFail
javascript
Without this, a queue processing 1,000 jobs/hour generates 24,000 job Hashes per day. Each Hash is ~300–500 bytes. At 1M jobs total: ~300–500MB just for the job Hashes.
The completed and failed Sorted Sets also grow unboundedly. removeOnComplete.count limits the Sorted Set size by trimming (ZREMRANGEBYRANK) after each completion.
Summary
BullMQ uses wait (List) for FIFO queuing, active (List) for in-progress jobs, completed/failed (Sorted Sets) for history, delayed (Sorted Set with timestamp score) for scheduling
Job data lives in a Hash bull:{queue}:{jobId}; queues store only the ID
Workers use LMOVE wait active (atomic) to claim jobs; a Lua-based lock prevents double-processing
Stalled jobs (lock expired, still in active) are detected and retried or failed by the stall checker
Tune lockDuration to exceed max job processing time; lockRenewTime defaults to half lockDuration
Rate limiting uses a shared counter key with a TTL — once the whole queue hits max jobs in the current duration window, workers pause pulling new jobs until the window resets
Enable removeOnComplete and removeOnFail to prevent unbounded memory growth
Query queue state directly with Redis commands for debugging without the BullMQ API overhead
Next: P-7 — Cache Stampede, Avalanche, and Penetration — three cache failure modes that look similar in monitoring but require different solutions.
Knowledge Check
A BullMQ worker is processing a video transcoding job that takes exactly 45 seconds to complete. The worker is configured with a lockDuration of 30,000 milliseconds (30 seconds) and maxStalledCount of 1. What will happen during the execution of this job, assuming the worker does not crash?
An operations engineer wants to know exactly how many jobs are currently waiting to be processed in the emails queue without writing a Node.js script. Which raw Redis command provides this exact number in O(1) time?
A team deploys a high-throughput BullMQ queue processing 10,000 jobs per minute. After three days, they receive an alert that Redis memory usage has spiked by several gigabytes, eventually triggering an OOM kill. The queue is fully processed (wait and active lists are empty). What is the most likely architectural misconfiguration?
Test your knowledge with more question sets
Sign in to access a wider variety of questions and get notified when new practice sets are added to this module.
// Worker signals failure (after all retries exhausted)await job.moveToFailed(error, workerToken);
// Worker's internal stall check (runs in QueueEvents or Worker itself)// Checks all jobs in 'active' that have an expired lock
// Configure stall detectionconst worker =newWorker('emails', processor,{stalledInterval:30000,// check every 30 secondsmaxStalledCount:1,// mark as failed after 1 stalllockDuration:30000,// lock expires in 30 secondslockRenewTime:15000,// renew lock every 15 seconds});
const worker =newWorker('emails', processor,{limiter:{max:100,duration:1000,// 100 jobs per second},});
# How many jobs are waiting?redis-cli LLEN bull:emails:wait
# How many jobs are active?redis-cli LLEN bull:emails:active
# What jobs are active? (get their IDs)redis-cli LRANGE bull:emails:active 0-1# Get details of a specific jobredis-cli HGETALL bull:emails:42
# What delayed jobs are coming up in the next 60 seconds?redis-cli ZRANGEBYSCORE bull:emails:delayed 0$(($(date +%s%3N)+60000)) WITHSCORES
# How many failed jobs?redis-cli ZCARD bull:emails:failed
# View the events streamredis-cli XREVRANGE bull:emails:events + - COUNT 10
// Recommended: auto-remove jobs after a count or ageconst worker =newWorker('emails', processor,{removeOnComplete:{count:1000},// keep last 1000 completedremoveOnFail:{count:500},// keep last 500 failed});