F-8 — Streams: Append-Only Logs and Consumer Groups

Who this module is for: You have outgrown Pub/Sub — you need messages to survive subscriber disconnections, multiple consumers to share the work of processing a stream, and the ability to replay past events. Redis Streams is the answer. This module covers the full Stream data model, the XADD/XREAD/XRANGE command family, consumer groups, acknowledgement semantics, and how Streams compare to Kafka and traditional queues.

What Redis Streams Are

A Redis Stream is an append-only log — a sequence of entries, each identified by a unique ID and containing a set of field-value pairs. Unlike Pub/Sub, entries are stored durably in Redis (subject to your persistence configuration) and remain available for consumers to read at any time, including after reconnection.

The model is inspired by Apache Kafka's partitioned log, but implemented as a single-node (or clustered) Redis data structure with simpler semantics and lower throughput at extreme scale.

Stream Entry IDs

Every stream entry has an ID in the format {milliseconds}-{sequence}:

• 1716000000001-0 — millisecond timestamp 1716000000001, sequence 0
• If two entries arrive in the same millisecond, the sequence increments: 1716000000001-1, 1716000000001-2

Redis generates IDs automatically using the current time when you pass *:

You can also provide explicit IDs for deterministic streams or replaying historical data:

IDs must be monotonically increasing — you cannot add an entry with an ID earlier than the stream's last entry.

Core Commands

Writing to a Stream

MAXLEN ~ 1000 (tilde = approximate) trims the stream to approximately 1000 entries. It is more efficient than exact trimming because it trims at listpack node boundaries rather than individual entries. Use approximate trimming in production.

Reading a Range

Special IDs:

• - = minimum possible ID (start of stream)
• + = maximum possible ID (end of stream)

Stream Metadata

Trimming

Use XTRIM MAXLEN ~ 10000 periodically (or via XADD MAXLEN ~) to prevent unbounded stream growth.

Reading Without Consumer Groups (XREAD)

XREAD lets you read new entries from one or more streams, starting from a given ID:

The special ID $ means "only entries added after this XREAD command was issued" — useful for a simple consumer that only cares about new events.

XREAD without consumer groups is fan-out, not competing consumers. Two clients both running XREAD BLOCK 0 STREAMS mystream $ will both receive every new entry. This is like Pub/Sub but with persistence — missed-while-blocked messages are available when you reconnect.

Consumer Groups

Consumer groups are the killer feature of Redis Streams. A consumer group allows multiple consumer instances to share the work of processing a stream — each entry is delivered to exactly one consumer in the group, not all of them.

This is the correct model for job queues and event processing pipelines.

Creating a Consumer Group

The id is the starting position:

• 0 — process all existing entries from the beginning
• $ — only process entries added after group creation
• A specific ID — start from that ID

Reading as a Consumer

The special ID > means "give me entries not yet delivered to any consumer in this group":

The response includes the entry ID and fields. Redis internally records that these entries are "pending" — delivered to worker-1 but not yet acknowledged.

Acknowledging Messages

After successfully processing an entry, the consumer acknowledges it:

Once acknowledged, the entry is removed from the group's pending entries list (PEL — Pending Entries List). It is not deleted from the stream itself — other groups or XRANGE queries can still read it.

The Pending Entries List

The PEL tracks entries that have been delivered to a consumer but not yet acknowledged:

The output shows: entry ID, consumer name, time since delivery (milliseconds), delivery count.

Delivery count > 1 means the entry was redelivered — usually because a consumer died before acknowledging. This is your signal to inspect the entry for poison pills (entries that cannot be processed successfully).

Claiming Stale Entries

When a consumer dies, its pending entries remain in the PEL forever (unless manually claimed). To recover them:

XAUTOCLAIM (Redis 7.0+) automatically transfers entries idle for more than min-idle-time milliseconds from their original consumer to the claiming consumer:

The older XCLAIM command does the same but requires you to specify each entry ID manually. Prefer XAUTOCLAIM for automated recovery.

Dead Letter Handling

An entry with a high delivery count (e.g., > 5) is likely a poison pill — it consistently fails to process. Move it to a dead-letter stream:

Consumer Groups in Node.js

Redis Streams vs Kafka vs BullMQ

Redis Streams is the right choice when:

• You need durable messaging without Kafka's operational complexity
• Your throughput is < 500K messages/second on a single stream
• You want everything in Redis — no additional infrastructure
• You need consumer groups with ACK semantics

Summary

• A Redis Stream is an append-only log of field-value entries, each with a monotonic ID
• XADD appends entries; XRANGE/XREVRANGE reads by ID range; XREAD reads new entries (blocking or non-blocking)
• Use MAXLEN ~ on XADD or XTRIM to prevent unbounded stream growth
• Consumer groups (XGROUP CREATE, XREADGROUP, XACK) allow multiple consumers to share a stream — each entry delivered to exactly one consumer
• The Pending Entries List (PEL) tracks delivered-but-not-ACKed entries — use XPENDING and XAUTOCLAIM to recover from crashed consumers
• Streams provide at-least-once delivery (retry on crash) — your processing logic must be idempotent
• Use Streams over Pub/Sub when you need persistence, replay, or competing consumers
• Use Kafka over Streams when you need millions of messages/second or cross-DC replication

Next: F-9 — Memory Layout and Object Encoding Internals — how Redis actually stores data in memory, the jemalloc allocator, memory fragmentation, and the OBJECT ENCODING / DEBUG OBJECT commands for inspecting real-world memory usage.