Module F-9·24 min read

XADD/XREAD/XRANGE, the XREADGROUP consumer group model, XACK and the Pending Entry List, XAUTOCLAIM for crash recovery, dead-letter handling, and when Redis Streams beats Kafka or BullMQ.

JJS

Written by Jatin Jain Saraf · Senior Software Engineer

F-9 — Streams: Append-Only Logs and Consumer Groups

Q: A consumer reads a message from a Redis Stream using `XREADGROUP`, processes the job successfully, but the application crashes right before it can send the `XACK` command back to Redis. What is the state of that specific message in Redis, and how should a resilient system handle this scenario?

The message remains in the consumer group's Pending Entries List (PEL). A resilient system should periodically run `XAUTOCLAIM` (or `XPENDING` + `XCLAIM`) from active consumers to reassign and reprocess idle messages that belonged to crashed consumers. — When a message is read via `XREADGROUP`, Redis marks it as delivered to that specific consumer and adds it to the group's Pending Entries List (PEL). If the consumer crashes before acknowledging it (`XACK`), the message simply stays in the PEL indefinitely. Redis does not auto-reassign messages or track consumer health automatically. Robust implementations must actively monitor the PEL using `XPENDING` and use `XAUTOCLAIM` to take ownership of messages that have been idle for too long, ensuring they are ultimately processed.

Q: To prevent a Redis Stream from consuming all available server memory over time, a developer adds a trimming argument to their insertion command: `XADD events:orders MAXLEN = 1000 * order_id ORD-123`. A senior engineer suggests changing the `=` to a `~` (tilde). Why is this recommendation important for production performance?

The `=` operator performs an exact trim, forcing Redis to delete individual entries exactly at the boundary, which is computationally expensive. The `~` operator allows Redis to trim approximately (leaving slightly more than 1000 entries) by deleting entire internal listpack nodes at once, vastly improving performance. — Redis Streams are internally structured as a radix tree containing macro-nodes called listpacks. Trimming to an exact count (`=`) forces Redis to split these listpacks, precisely counting and deleting individual entries, which incurs a performance penalty on every `XADD`. Approximate trimming (`~`) allows Redis to drop entire listpack nodes at once. The stream length might sit slightly above the requested limit (by a few dozen entries), but the operation becomes incredibly fast and constant-time, making it the standard practice for high-throughput production systems.

Q: Two separate microservices—the "Inventory Service" and the "Email Service"—need to react to every new order placed in the `events:orders` stream. If both services use `XREADGROUP` to consume from the stream, how should their consumer groups be configured?

Each service must create its own distinct consumer group (e.g., `group-inventory` and `group-emails`). — A consumer group operates on a competing-consumers model: every message delivered to a group is sent to *only one* consumer within that group. If both microservices joined the same group, half the orders would go to the Inventory Service and the other half to the Email Service. To achieve a fan-out architecture where multiple independent systems each process *every* message, each system must register its own dedicated consumer group against the stream. Redis will then maintain a separate delivery offset for each group.

Who this module is for: You have outgrown Pub/Sub — you need messages to survive subscriber disconnections, multiple consumers to share the work of processing a stream, and the ability to replay past events. Redis Streams is the answer. This module covers the full Stream data model, the XADD/XREAD/XRANGE command family, consumer groups, acknowledgement semantics, and how Streams compare to Kafka and traditional queues.

What Redis Streams Are

A Redis Stream is an append-only log — a sequence of entries, each identified by a unique ID and containing a set of field-value pairs. Unlike Pub/Sub, entries are stored durably in Redis (subject to your persistence configuration) and remain available for consumers to read at any time, including after reconnection.

The model is inspired by Apache Kafka's partitioned log, but implemented as a single-node (or clustered) Redis data structure with simpler semantics and lower throughput at extreme scale.

text

Stream: events:orders
─────────────────────────────────────────────────────────
ID                    Fields
─────────────────────────────────────────────────────────
1716000000001-0       order_id:ORD-001 status:placed amount:2500
1716000000150-0       order_id:ORD-002 status:placed amount:750
1716000001234-0       order_id:ORD-001 status:shipped tracking:TRK-99
1716000005678-0       order_id:ORD-003 status:placed amount:1200
─────────────────────────────────────────────────────────

Stream Entry IDs

Every stream entry has an ID in the format {milliseconds}-{sequence}:

1716000000001-0 — millisecond timestamp 1716000000001, sequence 0
If two entries arrive in the same millisecond, the sequence increments: 1716000000001-1, 1716000000001-2

Redis generates IDs automatically using the current time when you pass *:

text

XADD mystream * field1 value1 field2 value2
→ "1716000000001-0"

You can also provide explicit IDs for deterministic streams or replaying historical data:

XADD mystream 1716000000001-0 field1 value1

IDs must be monotonically increasing — you cannot add an entry with an ID earlier than the stream's last entry.

Core Commands

Writing to a Stream

XADD key [NOMKSTREAM] [MAXLEN|MINID [=|~] threshold [LIMIT count]] id field value [field value ...]

text

# Add an entry with auto-generated ID
XADD events:orders * order_id "ORD-001" status "placed" amount "2500"
→ "1716000001234-0"

# Cap stream at 1000 entries (approximate trim, more efficient than exact)
XADD events:orders MAXLEN ~ 1000 * order_id "ORD-002" status "placed"

# Cap stream at 1000 entries (exact trim)
XADD events:orders MAXLEN = 1000 * order_id "ORD-003" status "placed"

MAXLEN ~ 1000 (tilde = approximate) trims the stream to approximately 1000 entries. It is more efficient than exact trimming because it trims at listpack node boundaries rather than individual entries. Use approximate trimming in production.

Reading a Range

text

XRANGE key start end [COUNT count]
XREVRANGE key end start [COUNT count]

Special IDs:

- = minimum possible ID (start of stream)
+ = maximum possible ID (end of stream)

text

# All entries
XRANGE events:orders - +

# Entries from ID 1716000000000-0 onwards
XRANGE events:orders 1716000000000-0 +

# Last 10 entries (most recent first)
XREVRANGE events:orders + - COUNT 10

Stream Metadata

text

XLEN key             → number of entries
XINFO STREAM key     → detailed stream info (length, first/last IDs, groups, etc.)
XINFO GROUPS key     → list consumer groups
XINFO CONSUMERS key group → list consumers in a group

Trimming

text

XTRIM key MAXLEN [=|~] count          → trim to N most recent entries
XTRIM key MINID [=|~] id             → trim entries older than a given ID

Use XTRIM MAXLEN ~ 10000 periodically (or via XADD MAXLEN ~) to prevent unbounded stream growth.

Reading Without Consumer Groups (XREAD)

XREAD lets you read new entries from one or more streams, starting from a given ID:

XREAD [COUNT count] [BLOCK milliseconds] STREAMS key [key ...] id [id ...]

text

# Read up to 10 entries from events:orders after ID 0-0 (from the very beginning)
XREAD COUNT 10 STREAMS events:orders 0-0

# Read entries after the last one you saw
XREAD COUNT 10 STREAMS events:orders 1716000001234-0

# Block for 5 seconds waiting for new entries (0 = block forever)
XREAD COUNT 10 BLOCK 5000 STREAMS events:orders $

The special ID $ means "only entries added after this XREAD command was issued" — useful for a simple consumer that only cares about new events.

XREAD without consumer groups is fan-out, not competing consumers. Two clients both running XREAD BLOCK 0 STREAMS mystream $ will both receive every new entry. This is like Pub/Sub but with persistence — missed-while-blocked messages are available when you reconnect.

Consumer Groups

Consumer groups are the killer feature of Redis Streams. A consumer group allows multiple consumer instances to share the work of processing a stream — each entry is delivered to exactly one consumer in the group, not all of them.

This is the correct model for job queues and event processing pipelines.

text

Stream: events:orders
         ↓
Consumer Group: order-processors
         ↓
  ┌──────┬──────┬──────┐
  Consumer-1  Consumer-2  Consumer-3
  (processes  (processes  (processes
  ORD-001)    ORD-002)    ORD-003)

Creating a Consumer Group

XGROUP CREATE key groupname id [MKSTREAM]

The id is the starting position:

0 — process all existing entries from the beginning
$ — only process entries added after group creation
A specific ID — start from that ID

text

# Create group starting from the beginning of the stream
XGROUP CREATE events:orders order-processors 0 MKSTREAM
# MKSTREAM creates the stream key if it doesn't exist

Reading as a Consumer

XREADGROUP GROUP groupname consumername [COUNT count] [BLOCK milliseconds] [NOACK] STREAMS key [key ...] id [id ...]

The special ID > means "give me entries not yet delivered to any consumer in this group":

text

# Consumer "worker-1" reads up to 10 undelivered entries
XREADGROUP GROUP order-processors worker-1 COUNT 10 BLOCK 5000 STREAMS events:orders >

The response includes the entry ID and fields. Redis internally records that these entries are "pending" — delivered to worker-1 but not yet acknowledged.

Acknowledging Messages

After successfully processing an entry, the consumer acknowledges it:

XACK key groupname id [id ...]

text

# Acknowledge that worker-1 finished processing entry 1716000001234-0
XACK events:orders order-processors 1716000001234-0

Once acknowledged, the entry is removed from the group's pending entries list (PEL — Pending Entries List). It is not deleted from the stream itself — other groups or XRANGE queries can still read it.

The Pending Entries List

The PEL tracks entries that have been delivered to a consumer but not yet acknowledged:

XPENDING key groupname [[IDLE min-idle-time] start end count [consumername]]

text

# Summary of pending entries for the group
XPENDING events:orders order-processors - + 10

# Detailed view: entries idle for more than 30 seconds
XPENDING events:orders order-processors IDLE 30000 - + 10

The output shows: entry ID, consumer name, time since delivery (milliseconds), delivery count.

Delivery count > 1 means the entry was redelivered — usually because a consumer died before acknowledging. This is your signal to inspect the entry for poison pills (entries that cannot be processed successfully).

Claiming Stale Entries

When a consumer dies, its pending entries remain in the PEL forever (unless manually claimed). To recover them:

XAUTOCLAIM key group consumer min-idle-time start [COUNT count]

XAUTOCLAIM (Redis 7.0+) automatically transfers entries idle for more than min-idle-time milliseconds from their original consumer to the claiming consumer:

text

# Claim all entries idle for more than 60 seconds (consumer crashed)
XAUTOCLAIM events:orders order-processors worker-2 60000 0-0

The older XCLAIM command does the same but requires you to specify each entry ID manually. Prefer XAUTOCLAIM for automated recovery.

Dead Letter Handling

An entry with a high delivery count (e.g., > 5) is likely a poison pill — it consistently fails to process. Move it to a dead-letter stream:

javascript

const DEAD_LETTER_THRESHOLD = 5;

// In your consumer's processing loop:
const pending = await redis.xpending('events:orders', 'order-processors', '-', '+', 100);
for (const entry of pending) {
  const [id, consumer, idleMs, deliveryCount] = entry;
  if (deliveryCount >= DEAD_LETTER_THRESHOLD) {
    // Move to dead letter stream
    const data = await redis.xrange('events:orders', id, id);
    await redis.xadd('events:orders:dead', '*', ...Object.entries(data[0][1]).flat(),
      'original_id', id, 'delivery_count', String(deliveryCount));
    await redis.xack('events:orders', 'order-processors', id);
  }
}

Consumer Groups in Node.js

typescript

import Redis from 'ioredis';
const redis = new Redis();

const STREAM = 'events:orders';
const GROUP = 'order-processors';
const CONSUMER = `worker-${process.pid}`;

async function setupGroup() {
  try {
    await redis.xgroup('CREATE', STREAM, GROUP, '0', 'MKSTREAM');
  } catch (err: any) {
    // Group already exists — fine
    if (!err.message.includes('BUSYGROUP')) throw err;
  }
}

async function processOrders() {
  await setupGroup();

  while (true) {
    // First check for pending entries from crashed consumers
    const pending = await redis.xreadgroup(
      'GROUP', GROUP, CONSUMER,
      'COUNT', '10',
      'STREAMS', STREAM, '0'  // '0' reads our own pending entries
    );

    // Then read new entries
    const fresh = await redis.xreadgroup(
      'GROUP', GROUP, CONSUMER,
      'COUNT', '10',
      'BLOCK', '5000',
      'STREAMS', STREAM, '>'
    );

    const entries = fresh?.[0]?.[1] ?? [];
    for (const [id, fields] of entries) {
      try {
        await handleOrder(id, parseFields(fields));
        await redis.xack(STREAM, GROUP, id);
      } catch (err) {
        console.error(`Failed to process ${id}:`, err);
        // Do not ACK — entry stays in PEL for retry
      }
    }
  }
}

function parseFields(fields: string[]): Record<string, string> {
  const obj: Record<string, string> = {};
  for (let i = 0; i < fields.length; i += 2) {
    obj[fields[i]] = fields[i + 1];
  }
  return obj;
}

Redis Streams vs Kafka vs BullMQ

Feature	Redis Streams	Kafka	BullMQ (Redis-backed)
Persistence	In-memory + optional AOF/RDB	Disk-first, configurable retention	In-memory + optional
Throughput	~100K msg/sec per node	Millions/sec (partitioned)	~10K jobs/sec
Horizontal scaling	Redis Cluster (manual partitioning)	Native partitions	Multiple queues
Consumer groups	Yes	Yes (consumer groups)	Yes
Message replay	Yes (by ID)	Yes (by offset)	No
Complex scheduling	No	No	Yes (delay, retry, cron)
Ops complexity	Low	High	Low
Best for	Medium-volume event pipelines	High-volume event streaming	Job queues with retry logic

Redis Streams is the right choice when:

You need durable messaging without Kafka's operational complexity
Your throughput is < 500K messages/second on a single stream
You want everything in Redis — no additional infrastructure
You need consumer groups with ACK semantics

Summary

A Redis Stream is an append-only log of field-value entries, each with a monotonic ID
XADD appends entries; XRANGE/XREVRANGE reads by ID range; XREAD reads new entries (blocking or non-blocking)
Use MAXLEN ~ on XADD or XTRIM to prevent unbounded stream growth
Consumer groups (XGROUP CREATE, XREADGROUP, XACK) allow multiple consumers to share a stream — each entry delivered to exactly one consumer
The Pending Entries List (PEL) tracks delivered-but-not-ACKed entries — use XPENDING and XAUTOCLAIM to recover from crashed consumers
Streams provide at-least-once delivery (retry on crash) — your processing logic must be idempotent
Use Streams over Pub/Sub when you need persistence, replay, or competing consumers
Use Kafka over Streams when you need millions of messages/second or cross-DC replication

Next: F-10 — Memory Layout and Object Encoding Internals — how Redis actually stores data in memory, the jemalloc allocator, memory fragmentation, and the OBJECT ENCODING / DEBUG OBJECT commands for inspecting real-world memory usage.

Knowledge Check

A consumer reads a message from a Redis Stream using XREADGROUP, processes the job successfully, but the application crashes right before it can send the XACK command back to Redis. What is the state of that specific message in Redis, and how should a resilient system handle this scenario?

To prevent a Redis Stream from consuming all available server memory over time, a developer adds a trimming argument to their insertion command: XADD events:orders MAXLEN = 1000 * order_id ORD-123. A senior engineer suggests changing the = to a ~ (tilde). Why is this recommendation important for production performance?

Two separate microservices—the "Inventory Service" and the "Email Service"—need to react to every new order placed in the events:orders stream. If both services use XREADGROUP to consume from the stream, how should their consumer groups be configured?

Test your knowledge with more question sets

Sign in to keep reading

The rest of this module is free — sign in with Google to unlock it and track your progress.

PreviousModule F-8: Pub/Sub and the Message Fanout Model Next Module F-10: Memory Layout and Object Encoding Internals

Discussion

Join the discussion

Loading comments...