F-8 — Streams: Append-Only Logs and Consumer Groups

What Redis Streams Are

A Redis Stream is an append-only log — a sequence of entries, each identified by a unique ID and containing a set of field-value pairs. Unlike Pub/Sub, entries are stored durably in Redis (subject to your persistence configuration) and remain available for consumers to read at any time, including after reconnection.

The model is inspired by Apache Kafka's partitioned log, but implemented as a single-node (or clustered) Redis data structure with simpler semantics and lower throughput at extreme scale.

Stream: events:orders
─────────────────────────────────────────────────────────
ID                    Fields
─────────────────────────────────────────────────────────
1716000000001-0       order_id:ORD-001 status:placed amount:2500
1716000000150-0       order_id:ORD-002 status:placed amount:750
1716000001234-0       order_id:ORD-001 status:shipped tracking:TRK-99
1716000005678-0       order_id:ORD-003 status:placed amount:1200
─────────────────────────────────────────────────────────

Stream Entry IDs

Every stream entry has an ID in the format {milliseconds}-{sequence}:

• 1716000000001-0 — millisecond timestamp 1716000000001, sequence 0
• If two entries arrive in the same millisecond, the sequence increments: 1716000000001-1, 1716000000001-2

Redis generates IDs automatically using the current time when you pass *:

XADD mystream * field1 value1 field2 value2
→ "1716000000001-0"

You can also provide explicit IDs for deterministic streams or replaying historical data:

XADD mystream 1716000000001-0 field1 value1

IDs must be monotonically increasing — you cannot add an entry with an ID earlier than the stream's last entry.

Core Commands

Writing to a Stream

XADD key [NOMKSTREAM] [MAXLEN|MINID [=|~] threshold [LIMIT count]] id field value [field value ...]

# Add an entry with auto-generated ID
XADD events:orders * order_id "ORD-001" status "placed" amount "2500"
→ "1716000001234-0"

# Cap stream at 1000 entries (approximate trim, more efficient than exact)
XADD events:orders MAXLEN ~ 1000 * order_id "ORD-002" status "placed"

# Cap stream at 1000 entries (exact trim)
XADD events:orders MAXLEN = 1000 * order_id "ORD-003" status "placed"

MAXLEN ~ 1000 (tilde = approximate) trims the stream to approximately 1000 entries. It is more efficient than exact trimming because it trims at listpack node boundaries rather than individual entries. Use approximate trimming in production.

Reading a Range

XRANGE key start end [COUNT count]
XREVRANGE key end start [COUNT count]

Special IDs:

• - = minimum possible ID (start of stream)
• + = maximum possible ID (end of stream)

# All entries
XRANGE events:orders - +

# Entries from ID 1716000000000-0 onwards
XRANGE events:orders 1716000000000-0 +

# Last 10 entries (most recent first)
XREVRANGE events:orders + - COUNT 10

Stream Metadata

XLEN key             → number of entries
XINFO STREAM key     → detailed stream info (length, first/last IDs, groups, etc.)
XINFO GROUPS key     → list consumer groups
XINFO CONSUMERS key group → list consumers in a group

Trimming

XTRIM key MAXLEN [=|~] count          → trim to N most recent entries
XTRIM key MINID [=|~] id             → trim entries older than a given ID

Use XTRIM MAXLEN ~ 10000 periodically (or via XADD MAXLEN ~) to prevent unbounded stream growth.

Reading Without Consumer Groups (XREAD)

XREAD lets you read new entries from one or more streams, starting from a given ID:

XREAD [COUNT count] [BLOCK milliseconds] STREAMS key [key ...] id [id ...]

# Read up to 10 entries from events:orders after ID 0-0 (from the very beginning)
XREAD COUNT 10 STREAMS events:orders 0-0

# Read entries after the last one you saw
XREAD COUNT 10 STREAMS events:orders 1716000001234-0

# Block for 5 seconds waiting for new entries (0 = block forever)
XREAD COUNT 10 BLOCK 5000 STREAMS events:orders $

The special ID $ means "only entries added after this XREAD command was issued" — useful for a simple consumer that only cares about new events.

XREAD without consumer groups is fan-out, not competing consumers. Two clients both running XREAD BLOCK 0 STREAMS mystream $ will both receive every new entry. This is like Pub/Sub but with persistence — missed-while-blocked messages are available when you reconnect.

Consumer Groups

Consumer groups are the killer feature of Redis Streams. A consumer group allows multiple consumer instances to share the work of processing a stream — each entry is delivered to exactly one consumer in the group, not all of them.

This is the correct model for job queues and event processing pipelines.

Stream: events:orders
         ↓
Consumer Group: order-processors
         ↓
  ┌──────┬──────┬──────┐
  Consumer-1  Consumer-2  Consumer-3
  (processes  (processes  (processes
  ORD-001)    ORD-002)    ORD-003)

Creating a Consumer Group

XGROUP CREATE key groupname id [MKSTREAM]

The id is the starting position:

• 0 — process all existing entries from the beginning
• $ — only process entries added after group creation
• A specific ID — start from that ID

# Create group starting from the beginning of the stream
XGROUP CREATE events:orders order-processors 0 MKSTREAM
# MKSTREAM creates the stream key if it doesn't exist

Reading as a Consumer

XREADGROUP GROUP groupname consumername [COUNT count] [BLOCK milliseconds] [NOACK] STREAMS key [key ...] id [id ...]

The special ID > means "give me entries not yet delivered to any consumer in this group":

# Consumer "worker-1" reads up to 10 undelivered entries
XREADGROUP GROUP order-processors worker-1 COUNT 10 BLOCK 5000 STREAMS events:orders >

The response includes the entry ID and fields. Redis internally records that these entries are "pending" — delivered to worker-1 but not yet acknowledged.

Acknowledging Messages

After successfully processing an entry, the consumer acknowledges it:

XACK key groupname id [id ...]

# Acknowledge that worker-1 finished processing entry 1716000001234-0
XACK events:orders order-processors 1716000001234-0

Once acknowledged, the entry is removed from the group's pending entries list (PEL — Pending Entries List). It is not deleted from the stream itself — other groups or XRANGE queries can still read it.

The Pending Entries List

The PEL tracks entries that have been delivered to a consumer but not yet acknowledged:

XPENDING key groupname [[IDLE min-idle-time] start end count [consumername]]

# Summary of pending entries for the group
XPENDING events:orders order-processors - + 10

# Detailed view: entries idle for more than 30 seconds
XPENDING events:orders order-processors IDLE 30000 - + 10

The output shows: entry ID, consumer name, time since delivery (milliseconds), delivery count.

Delivery count > 1 means the entry was redelivered — usually because a consumer died before acknowledging. This is your signal to inspect the entry for poison pills (entries that cannot be processed successfully).

Claiming Stale Entries

When a consumer dies, its pending entries remain in the PEL forever (unless manually claimed). To recover them:

XAUTOCLAIM key group consumer min-idle-time start [COUNT count]

XAUTOCLAIM (Redis 7.0+) automatically transfers entries idle for more than min-idle-time milliseconds from their original consumer to the claiming consumer:

# Claim all entries idle for more than 60 seconds (consumer crashed)
XAUTOCLAIM events:orders order-processors worker-2 60000 0-0

The older XCLAIM command does the same but requires you to specify each entry ID manually. Prefer XAUTOCLAIM for automated recovery.

Dead Letter Handling

An entry with a high delivery count (e.g., > 5) is likely a poison pill — it consistently fails to process. Move it to a dead-letter stream:

javascript
const DEAD_LETTER_THRESHOLD = 5;

// In your consumer's processing loop:
const pending = await redis.xpending('events:orders', 'order-processors', '-', '+', 100);
for (const entry of pending) {
  const [id, consumer, idleMs, deliveryCount] = entry;
  if (deliveryCount >= DEAD_LETTER_THRESHOLD) {
    // Move to dead letter stream
    const data = await redis.xrange('events:orders', id, id);
    await redis.xadd('events:orders:dead', '*', ...Object.entries(data[0][1]).flat(),
      'original_id', id, 'delivery_count', String(deliveryCount));
    await redis.xack('events:orders', 'order-processors', id);
  }
}

Consumer Groups in Node.js

typescript
import Redis from 'ioredis';
const redis = new Redis();

const STREAM = 'events:orders';
const GROUP = 'order-processors';
const CONSUMER = `worker-${process.pid}`;

async function setupGroup() {
  try {
    await redis.xgroup('CREATE', STREAM, GROUP, '0', 'MKSTREAM');
  } catch (err: any) {
    // Group already exists — fine
    if (!err.message.includes('BUSYGROUP')) throw err;
  }
}

async function processOrders() {
  await setupGroup();

  while (true) {
    // First check for pending entries from crashed consumers
    const pending = await redis.xreadgroup(
      'GROUP', GROUP, CONSUMER,
      'COUNT', '10',
      'STREAMS', STREAM, '0'  // '0' reads our own pending entries
    );

    // Then read new entries
    const fresh = await redis.xreadgroup(
      'GROUP', GROUP, CONSUMER,
      'COUNT', '10',
      'BLOCK', '5000',
      'STREAMS', STREAM, '>'
    );

    const entries = fresh?.[0]?.[1] ?? [];
    for (const [id, fields] of entries) {
      try {
        await handleOrder(id, parseFields(fields));
        await redis.xack(STREAM, GROUP, id);
      } catch (err) {
        console.error(`Failed to process ${id}:`, err);
        // Do not ACK — entry stays in PEL for retry
      }
    }
  }
}

function parseFields(fields: string[]): Record<string, string> {
  const obj: Record<string, string> = {};
  for (let i = 0; i < fields.length; i += 2) {
    obj[fields[i]] = fields[i + 1];
  }
  return obj;
}

Redis Streams vs Kafka vs BullMQ

Redis Streams is the right choice when:

• You need durable messaging without Kafka's operational complexity
• Your throughput is < 500K messages/second on a single stream
• You want everything in Redis — no additional infrastructure
• You need consumer groups with ACK semantics

Summary

• A Redis Stream is an append-only log of field-value entries, each with a monotonic ID
• XADD appends entries; XRANGE/XREVRANGE reads by ID range; XREAD reads new entries (blocking or non-blocking)
• Use MAXLEN ~ on XADD or XTRIM to prevent unbounded stream growth
• Consumer groups (XGROUP CREATE, XREADGROUP, XACK) allow multiple consumers to share a stream — each entry delivered to exactly one consumer
• The Pending Entries List (PEL) tracks delivered-but-not-ACKed entries — use XPENDING and XAUTOCLAIM to recover from crashed consumers
• Streams provide at-least-once delivery (retry on crash) — your processing logic must be idempotent
• Use Streams over Pub/Sub when you need persistence, replay, or competing consumers
• Use Kafka over Streams when you need millions of messages/second or cross-DC replication

Next: F-9 — Memory Layout and Object Encoding Internals — how Redis actually stores data in memory, the jemalloc allocator, memory fragmentation, and the OBJECT ENCODING / DEBUG OBJECT commands for inspecting real-world memory usage.