Module P-15·36 min read

The log as source of truth, partitions and consumer groups, offset management, and the rebalance storm that stops all consumption at once.

JJS

Written by Jatin Jain Saraf · Senior Software Engineer

Module P-15 — Event Streaming with Kafka

Q: A consumer group with 12 partitions and 12 instances falls behind during a traffic spike. Lag climbs steadily, and the group's logs show repeated rebalances. An engineer scales the deployment from 12 to 30 instances to catch up. Lag gets worse. Why?

Only 12 instances can hold partitions — a partition goes to at most one consumer per group, so 18 sit idle contributing nothing. Worse, the deployment itself triggers rebalances, and under an eager protocol every rebalance stops consumption for the entire group while the backlog keeps growing, which is the feedback loop that produces a storm. The repeated rebalances were the actual problem, and scaling out fed them. — The module states both halves of this: parallelism is capped by partition count, so instances beyond it idle, and scaling consumers during a lag incident is usually wrong because each change triggers another rebalance in a group that is already behind. The storm's feedback loop is the mechanism — pause, larger backlog, fuller batches, next missed `max.poll.interval.ms`. The module's prescription is to raise throughput per consumer (smaller `max.poll.records`, timeouts on slow dependencies, cheaper handlers) plus static membership and cooperative rebalancing. Option C is the respectable near-miss: broker disk saturation is a genuine Kafka bottleneck and more fetchers do add load, but caught-up consumers read from the page cache rather than disk, and it does not explain the rebalances already in the logs. Option B misdescribes `auto.offset.reset`, which applies only when a group has no committed offset. Option A invents a coordinator limit that does not exist.

Q: An inventory service consumes `stock-events` and occasionally applies a `stock_decremented` event before the `stock_replenished` event that preceded it, producing negative stock. The topic has 24 partitions; the producer sends events without a key. The team's design doc says "Kafka guarantees ordering." What is wrong and what is the fix, with its cost?

Kafka guarantees ordering only within a partition, and unkeyed records are distributed across all 24 partitions, so two events for the same SKU can land in different partitions and be processed concurrently in any order. The fix is to key by SKU so all of that SKU's events hash to one partition and are read in write order. The cost is skew: a high-traffic SKU makes one partition hot, and one consumer then sets the lag for the whole topic. — The module's exact formulation is that Kafka gives you same-key ordering, never global ordering: partitions are separate files read by separate consumers at separate speeds, and `hash(key) % numPartitions` is what routes related records together. It also names the cost of keying — the hot partition, Module P-8's skew problem arriving through the message bus — and the mitigations of a composite key or a custom partitioner. Option C is the near-miss worth respecting: the idempotent producer does prevent one real reordering failure, where an in-flight retry lands after a later batch *within a partition*. But it cannot create ordering between partitions, which is what is happening here, and unkeyed records are spread across all 24. Option B confuses rebalancing with ordering; a rebalance causes duplicates and pauses, not cross-partition reordering. Option D is a real technique for strict global ordering, but it discards all parallelism when per-SKU ordering is what is actually needed.

Q: A payments consumer commits offsets after each handler completes. A pod is killed mid-batch during a deploy, and on restart three customers see duplicate charges. A senior engineer says the commit placement is correct and should not change. What is the reasoning, and what should actually change?

Committing after processing is deliberately at-least-once: the alternative — committing first — is at-most-once and would silently drop payments instead, which is far worse. Duplicates are normal operation under this design, produced by every rebalance and every retry, so the fix is an idempotent handler (Module P-16) that recognises an already-applied charge, not a change to the commit point. — The module frames commit placement as a fork with only two branches — commit before is at-most-once and loses records, commit after is at-least-once and duplicates them — and puts payments firmly in the second camp, because a quietly missing charge is worse than a detectable duplicate one. It then states the obligation that comes with the choice: the handler must tolerate being run twice. Option A is the near-miss, and the reason it fails is the module's caveat about exactly-once — transactions are atomic for Kafka-to-Kafka work, and a card charge is an external side effect outside the transaction, so `read_committed` cannot help. Option B fixes a real and related problem: cooperative rebalancing shortens pauses and is worth adopting, but any redelivery of uncommitted records still duplicates work, so it reduces the frequency rather than the possibility. Option D makes things worse — auto-commit on a timer can commit records still in flight, converting this into silent loss.

What this module covers: Kafka is Module F-2's central finding sold as a product: an append-only log, written sequentially, served out of the page cache. This module covers the log as a primary abstraction and how it differs from the destructive queues of Module P-14, why sequential append plus zero-copy makes a broker fast, partitions as the unit of both parallelism and ordering, keys and the per-partition-only ordering guarantee, consumer groups and offset commits and the at-most-once/at-least-once fork they create, rebalance storms and their three mitigations, consumer lag, retention and log compaction, why partition count is nearly irreversible, and the operational bill you pick up by adopting it.

Kafka Is Module F-2 Turned Into a Product

Module F-2 ended on a number that surprises most people: a sequential write to an NVMe SSD sustains 3–7 GB/s, while a random access pattern against DRAM — nominally 20–50 GB/s — delivers something closer to 80 MB/s of useful bytes, because every access pulls a 64-byte cache line to use eight bytes of it and defeats the prefetcher. Sequential beats random by so much that the storage medium stops being the interesting variable. The access pattern is.

Kafka is what happens when someone takes that seriously and designs an entire product around one access pattern. There is no B-tree. There is no in-place update. There is no random write path to optimise, because there is no random write. A Kafka topic partition is a file that only ever grows at the end, and a Kafka consumer is a cursor that only ever moves forward through it. Everything else — the replication, the consumer groups, the retention policies — is scaffolding around that single decision.

That is why a Kafka broker sustains hundreds of MB/s of sequential throughput on ordinary hardware while a Postgres instance with fsync on commit tops out at roughly 1K–5K transactions per second (Module F-3's capacity anchors). It is not that Kafka is better engineered than Postgres. It is that Postgres has to support random reads, in-place updates, secondary indexes, and MVCC visibility rules, and Kafka has decided in advance that you do not get any of those. Module F-14's rule — NoSQL scales by removing features — applies here in its purest form. Kafka scales by removing almost all of them.

The primary abstraction is the log. Not a queue, not a topic, not a message. A log: an ordered, append-only, immutable sequence of records, each with a monotonically increasing position called an offset. Records are retained after they are read. Reading does not consume. Ten independent consumers can read the same partition at the same time, at different offsets, and none of them affects the others, because reading is just a file read at a byte position.

Module P-14's queues work the other way. A message is delivered, acknowledged, and deleted — consumption is destructive, and the broker's job is to track the state of every individual message (invisible, in-flight, acked, dead-lettered). That per-message state is what makes a queue broker's storage random-access, and it is also what makes a queue impossible to replay.

	Queue (Module P-14)	Log (Kafka)
Consumption	Destructive — ack deletes the message	Non-destructive — offset advances, record stays
Broker state per message	Visibility, in-flight, retry count, DLQ	None; only the record's byte position
Multiple independent readers	Needs fan-out to N queues	Native — each group has its own offsets
Replay	Impossible once acked	Reset the offset and read again
Ordering	Best-effort, breaks under redelivery	Strict within a partition
Storage access pattern	Random-ish	Purely sequential
Per-message retry/delay	Built in	You build it (Module P-19)
Natural unit of scale	Consumers	Partitions

Read that last-but-one row carefully, because it is the honest cost of the log model and it is the one teams discover late. A queue gives you per-message retry, per-message delay, and a dead-letter queue as first-class features. Kafka gives you none of them, because a log has no per-message state to hang them off. If record 4,001 fails and you want to skip it and continue, you have to build that yourself — commit past it and write it to a separate retry topic. Kafka's ordering guarantee and its lack of per-message handling are the same property viewed from two sides.

Analogy: a queue is the spike on a diner counter — the cook takes the docket off, fills the order, and bins it; the docket is gone, and only one cook can ever hold it. Kafka is the till roll. The printer only advances, nothing is ever erased, and any number of people can walk up and read from any line number they like — the kitchen reading line 400, the accountant reading line 12, the auditor starting again from line 1 next March. The roll is the record; where each reader is up to is just a line number they keep in their own pocket.

Why It's Fast: The Page Cache Does the Work

Three mechanisms, all of them Module F-2 in application.

Sequential append. A produce request appends to the active segment file of a partition. No seek, no index maintenance, no page split, no read-modify-write. On spinning disks — where Kafka's design was born — this was the difference between ~150–250 MB/s sequential and 100–200 IOPS random. On NVMe the ratio narrows but the direction holds.

The page cache, not a heap cache. Kafka does not maintain its own record cache. It writes to the OS page cache and lets the kernel flush; it reads from the page cache too. So a consumer keeping up with the tail of the log is reading pages that were written seconds ago and are still resident in memory — the read never touches disk at all. This is why Kafka brokers are given modest JVM heaps and enormous amounts of RAM: the RAM is for the kernel, not for Kafka. It also means the working-set arithmetic from Module F-12 applies directly. Consumers reading the tail are a memory workload. A consumer that has fallen an hour behind is a disk workload, and it evicts pages that the caught-up consumers wanted.

Zero-copy on the read path. Because a stored record is byte-for-byte what goes out on the wire, the broker can hand a file region straight to a socket via sendfile — the data goes page cache → NIC without being copied into the JVM and back out. That eliminates two copies and a great deal of garbage collection pressure.

Why this matters in production: the zero-copy path is conditional, and the conditions are easy to break. Enable TLS and the broker must encrypt in user space, so the copies come back. Let producers and consumers disagree on message format or compression such that the broker has to re-compress or convert, and the copies come back along with substantial CPU cost. A cluster that was network-bound becomes CPU-bound, and the symptom is rising produce latency with disk and network both apparently idle. The fix is usually to make producers, brokers, and consumers agree on compression codec and format so the broker can pass bytes through untouched — but TLS is not optional in most environments, and the honest position is that you pay for it. Budget CPU accordingly rather than being surprised by it.

Partitions Are Both Your Parallelism and Your Only Ordering Guarantee

A topic is a named stream. It is a logical thing; it does not exist on disk. A topic is divided into partitions, and a partition is the real object: one directory of segment files on one broker (plus its replicas), with its own offset sequence starting at 0.

Everything important about Kafka's data model follows from partitions being the physical unit:

Parallelism is capped by partition count. Within a consumer group, a partition is assigned to at most one consumer. Twelve partitions means at most twelve useful consumer instances; the thirteenth sits idle holding nothing. Adding consumers beyond the partition count does not add throughput, which is the single most common wasted deployment in Kafka operations.
Ordering exists only inside a partition. Kafka guarantees that records in a partition are read in the order they were written. It guarantees nothing whatsoever about the relative order of records in different partitions of the same topic — they are separate files, read by separate consumers, at separate speeds.
Throughput and durability are per-partition. Each partition has one leader replica handling all reads and writes for it, with followers replicating. A partition is therefore also the unit of load imbalance.

The bridge between "I need ordering" and "ordering is per-partition" is the key. The producer computes the partition from a hash of the record key:

text

partition = hash(key) % numPartitions        // when a key is present
partition = sticky-batch round-robin         // when the key is null

So key = order_id puts every event about one order into one partition, in order, forever. That is the guarantee you actually get, and it is worth stating in the exact words you should use in a design review: Kafka gives you same-key ordering, never global ordering.

typescript

// Same-key ordering is a producer-side decision, not a broker feature.
await producer.send({
  topic: 'order-events',
  messages: [
    {
      // Every event for this order lands in the same partition, so a consumer
      // sees created -> paid -> shipped in that order. Drop this key and the
      // three events scatter across partitions and can be processed in any order.
      key: order.id,
      value: JSON.stringify(event),
    },
  ],
});

The cost of keying is a hot partition, and it is the same cost as Module F-14's partition-key problem wearing different clothes. Key by tenant_id in a system where one tenant is 40% of traffic and one partition is 40% of the load — on one broker, consumed by one consumer, which now sets the lag for the whole topic while eleven others idle. Module P-8's isolation problem, arriving through the message bus. The mitigations are the familiar ones: a composite key (tenant_id:order_id) if you only need ordering at the finer grain, or an explicit custom partitioner that splits the whale across several partitions and accepts that its events are no longer ordered relative to each other.

The other cost of keying is that it makes partition count nearly immutable, which is its own section below.

A guarantee people assume and do not have: ordering across topics. Two topics are unrelated logs. If your service writes payment-events and order-events and a consumer joins them, there is no ordering relationship between them at all — not even loosely. Systems that need cross-entity ordering need a single topic with a shared key, or a design that does not depend on the ordering (Module P-16's approach: make handlers order-insensitive rather than trying to enforce order).

Consumer Groups, Offsets, and Where "At-Least-Once" Comes From

A consumer group is a set of consumer instances sharing a group.id. The group's coordinator (a broker) divides the topic's partitions among the live members, and stores the group's committed offsets in an internal topic, __consumer_offsets. Two different groups reading the same topic are completely independent — separate offsets, separate lag, separate progress. That independence is the whole reason a log supports fan-out without duplicating data: your search indexer, your analytics loader, and your email service are three groups on one topic, not three copies of a queue.

The consumer commits its own offsets. Not the broker. This is the detail everything else in this section hangs off. An offset commit is a claim by the consumer that says "I have handled everything before position N," and the consumer decides when to make that claim. Where you put the commit relative to your processing is your delivery semantic:

typescript

for (const record of await consumer.poll()) {
  // COMMIT BEFORE  →  at-most-once
  // Offset advances first. Crash mid-handler and the record is never retried:
  // the group resumes after it. You lose messages, you never duplicate them.
  await consumer.commit(record.offset + 1);
  await handle(record);
}

for (const record of await consumer.poll()) {
  // COMMIT AFTER  →  at-least-once  (this is what you almost always want)
  // Crash between handle() and commit() and the record is redelivered on
  // restart, because the stored offset still points at it. Your handler
  // therefore has to tolerate being run twice on the same record.
  await handle(record);
  await consumer.commit(record.offset + 1);
}

	Commit before processing	Commit after processing
Semantic	At-most-once	At-least-once
Failure mode	Silent message loss	Duplicate processing
Handler requirement	None	Must be idempotent (Module P-16)
Suitable for	Metrics, sampled telemetry, replaceable data	Payments, orders, anything durable
How you find out it's wrong	A number is quietly low forever	A customer is charged twice

There is no third option that removes the problem, and it is worth being blunt about this because "exactly-once" is marketed aggressively. Kafka does have exactly-once semantics — an idempotent producer plus transactions that atomically commit produced records and consumer offsets, with isolation.level=read_committed on the reading side. It genuinely works, and its scope is genuinely narrow: it is exactly-once for Kafka-to-Kafka processing. The moment your handler charges a card, sends an email, or writes to Postgres, the external side effect is outside the transaction and you are back to at-least-once with duplicates. That is not a flaw in the implementation; it is what "no distributed transaction across two systems" means (Module A-3).

So: commit after processing, and make handlers idempotent. That is the standard posture, and Module P-16 is entirely about the second half of it.

One practical trap. Default enable.auto.commit=true commits periodically inside poll(), which means it commits offsets for records the previous poll returned — approximately commit-after, and usually fine. It stops being fine the moment your handler is asynchronous: if you hand records to a worker pool and return from the loop immediately, auto-commit marks them done while they are still in flight, and a crash loses them. You have silently built at-most-once while believing you had at-least-once. Manual commits after completion are the fix, and the cost is that you now own the commit cadence — commit too often and you add load to __consumer_offsets, commit too rarely and a crash reprocesses more.

Rebalance Storms: One Slow Consumer Stops Everyone

This is the outage shape you should be able to draw from memory, because it is Kafka's signature failure and because its feedback loop makes it self-worsening in exactly the way Module F-4's utilisation curve predicts.

The group coordinator tracks liveness two ways. A background thread sends a heartbeat every few seconds, and separately the consumer must call poll() again within max.poll.interval.ms — five minutes by default. The second one is a liveness check on your processing, not your network, and it is the one that fires.

text

1. A batch takes longer than usual to process. A downstream API got slow,
   a GC pause landed, max.poll.records handed the loop 500 records instead
   of 50, whatever.

2. The consumer misses max.poll.interval.ms. The coordinator concludes it
   is dead and removes it from the group.

3. A rebalance begins. Under the classic eager protocol this is
   stop-the-world: EVERY consumer in the group revokes ALL its partitions
   and waits for a new assignment. Consumption across the entire topic
   stops — not just on the slow consumer's partitions.

4. Reassignment completes. Consumers resume from last committed offsets,
   so every record processed-but-not-committed is processed again.

5. Meanwhile producers never stopped. The backlog is now larger than it
   was, and there is duplicate work on top of it.

6. Larger backlog means fuller batches means longer processing between
   polls means the next consumer misses its deadline. Go to step 2.

7. The group spends its time rebalancing instead of consuming. Lag climbs
   in a straight line. Restarting consumers triggers more rebalances.

Step 6 is why it is called a storm rather than a blip. The consequence of the pause is more load, and more load makes the pause recur — the same shape as Module F-4's queueing behaviour, where a system past the knee of the curve does not recover on its own.

Three mitigations, in the order you should reach for them:

Mitigation	What it changes	What it costs
Lower `max.poll.records`, raise `max.poll.interval.ms`	Fewer records per loop, longer permitted processing → the deadline stops firing	Lower per-poll throughput; a genuinely hung consumer now takes much longer to be evicted
Static membership (`group.instance.id` + a generous `session.timeout.ms`)	A restarting member reclaims its previous partitions instead of triggering a reassignment — deploys and pod restarts stop causing rebalances	A truly dead member's partitions stay unconsumed until the session timeout expires
Cooperative rebalancing (`partition.assignment.strategy` = cooperative-sticky)	Incremental rebalance: only the partitions that must move are revoked, and everyone else keeps consuming — no stop-the-world	Rebalances take more rounds; migrating a whole group to it requires a two-step rolling upgrade, not a single config flip

Two further habits that matter more than the configuration. First, do not do unbounded work inside the poll loop — if a handler calls a third-party API with no timeout, your consumer's liveness is that vendor's availability (Module A-6). Second, know why you are pausing. A consumer that legitimately needs a long time on a batch should call pause() on the partition and keep polling rather than blocking, so heartbeats and poll deadlines are satisfied while work proceeds.

Why this matters in production: the first instinct during a lag incident is to scale consumers out, and it is usually wrong on both counts. If you are already at partition count, the new instances get nothing. And each deployment change triggers another rebalance, adding pauses to a group that is already behind. Increasing throughput per consumer — smaller batches, a timeout on the slow dependency, cheaper handler work — beats adding instances almost every time.

Consumer Lag Is the Metric That Actually Tells You Something

Broker CPU, disk throughput, and produce latency are all worth graphing, and none of them answers the question your users care about. Consumer lag does: for each partition, the log end offset minus the group's committed offset. It is the number of records written but not yet processed.

Lag is the right metric because it is the only one that reflects the relationship between production and consumption rates. A cluster can look completely healthy — brokers idle, latency flat — while a consumer group is forty minutes behind and your users are seeing forty-minute-old data.

What to watch, and what each pattern means:

Lag by partition, not just the group total. A group total of 50,000 spread evenly is a capacity problem. A group total of 50,000 concentrated in one partition is a hot key or one stuck consumer, and adding instances will not help either.
The derivative, not the absolute value. Lag of 100,000 that is falling is a system recovering from a burst and needs no action. Lag of 5,000 that has risen steadily for an hour means your consumption rate is below your production rate and will not self-correct. Alert on sustained increase.
Lag in time, not only in records. "Two million records behind" means nothing to an incident commander. "Nineteen minutes behind" is actionable and maps directly to an SLO (Module O-2). Derive it from the timestamp of the record at the committed offset.
The retention cliff. Lag that approaches the retention window is a hard deadline. Cross it and the broker deletes records the consumer has not read: silent, permanent data loss, reported to the consumer as an offset-out-of-range condition it typically resolves by jumping to the earliest available offset — which looks like recovery while the gap is already gone. Alert on lag as a fraction of retention, well before the cliff.

Retention and Compaction: When the Log Becomes the State Store

Kafka's default retention is time- or size-based: keep records for seven days, or keep 100 GB per partition, then delete the oldest segments. This makes the log a buffer with a fixed memory.

cleanup.policy=compact changes the contract entirely. Under log compaction, Kafka retains at least the most recent record for every key, indefinitely, and garbage-collects superseded versions in the background. Old values for a key eventually disappear; the latest value for a key never does.

text

Before compaction:
  offset:  0        1        2        3        4        5
  key:     user-1   user-2   user-1   user-3   user-2   user-1
  value:   {v1}     {v1}     {v2}     {v1}     {v2}     {v3}

After compaction (earlier values for user-1 and user-2 collapsed away):
  offset:  3        4        5
  key:     user-3   user-2   user-1
  value:   {v1}     {v2}     {v3}

properties

# A compacted topic is a durable, replayable key-value store that happens
# to also be a change stream. Reading it from offset 0 reconstructs current
# state for every key; following the tail gives you every subsequent change.
cleanup.policy=compact
min.cleanable.dirty.ratio=0.5
# A record with a null value is a "tombstone": it deletes the key on
# compaction. delete.retention.ms is how long tombstones survive so that a
# consumer rebuilding state actually observes the deletion rather than
# silently keeping a value that was removed.
delete.retention.ms=86400000

That is the log-as-state-store idea, and it is the conceptual heart of event streaming. A compacted topic is simultaneously a change stream and a full snapshot: consume from the beginning to materialise current state, then keep consuming to stay current. It is how Kafka stores its own consumer offsets, how stream-processing frameworks restore local state after a restart, and the natural landing place for change data capture output (Module P-23).

The costs, and they are real:

Only the latest value survives. Compaction discards history for a key by design. If you need the full audit trail, you need a second non-compacted topic, or cleanup.policy=compact,delete with a retention window long enough for your audit requirement.
Compaction is asynchronous and best-effort. Duplicate keys are visible until the cleaner runs, so a consumer replaying from offset 0 may see several versions of a key and must be last-write-wins by offset — which it gets for free, since same-key records share a partition.
The tombstone window is a correctness bug waiting to happen. A consumer offline longer than delete.retention.ms can rebuild state, miss the tombstone, and resurrect deleted data. In a system with deletion obligations (Module O-9) that is not merely stale — it is a compliance failure.
State in a topic is state you now have to plan around. "Reconstruct from the log" is genuinely powerful and it is not instant. Replaying a 500 GB compacted topic to rebuild a cache is minutes to hours of cold, disk-bound reading that also evicts the page cache the caught-up consumers were relying on.

Partition Count Is Nearly Irreversible

Look again at partition = hash(key) % numPartitions. Change numPartitions and every key's destination changes.

Kafka lets you increase partitions on a topic. It cannot decrease them, and increasing them does not move existing data. So after going from 12 to 24 partitions:

New records for order-99 hash to a different partition than the old ones.
That order's history is now split across two partitions, read by two consumers, in no defined relative order.
Ordering is broken not just going forward but retroactively for every key in the topic, for anything that replays across the boundary.
A compacted topic is worse still: the same key now has live records in two partitions, and compaction — which operates per-partition — will never collapse them.

For an unkeyed topic, adding partitions is close to free. For a keyed topic where consumers depend on ordering, it is a migration, not a configuration change. The realistic options are to create a new topic with the target partition count and dual-read through the cutover, or to accept a defined period of unordered processing. Both are project-sized.

This is Module F-14's warning again — the partition key is the hardest thing to reverse — and it argues for the same posture: over-provision partitions at creation. If a topic might need 12 consumers, start with 48. The costs of doing so are concrete rather than theoretical:

Cost of more partitions	Mechanism
Broker file handles and memory	Each partition is open segment files plus in-memory index structures
Slower leader election on broker failure	Failover work scales with the number of partitions to reassign
Longer rebalances	More partitions to compute and distribute assignments for
Worse producer batching	Records spread thinner across more batches, so batches fill more slowly — smaller writes, more requests, worse compression
End-to-end latency	More partitions per broker means more concurrent sequential streams, which the disk sees as less-sequential

The judgement is to be generous but not absurd. Thousands of partitions on a small cluster is its own outage. Four times your expected consumer count is a defensible starting point.

The Honest Cost: You Now Operate a Distributed System

Everything above is the case for Kafka. Here is the invoice, and it should be read before adoption rather than after.

Kafka is a distributed, stateful, replicated storage system that your team now runs. Not a library, not a managed queue with a single API call. Self-hosted, that means:

Cluster membership and metadata — brokers, controllers, quorum health, and the failure modes of whichever coordination mechanism your deployment uses.
Replication and durability tuning that is genuinely subtle. acks=all with min.insync.replicas=2 on a replication factor of 3 gives you durability; acks=1 gives you throughput and a window in which an acknowledged record is lost if the leader dies before followers catch up. unclean.leader.election.enable=true keeps a partition available by promoting an out-of-sync replica, and silently discards committed records to do so. Availability and durability, priced explicitly — Module P-10's trade-off with a config file attached.
Disk capacity planning as an ongoing task. Retention times replication factor times ingest rate, and a broker that fills its disk does not degrade gracefully.
Partition rebalancing across brokers when you add capacity, which is a data-movement operation that competes with production traffic for network and disk.
Upgrades of a stateful cluster you cannot take offline, with client compatibility to reason about.
Skills. The number of engineers who can diagnose a rebalance storm at 3am is smaller than the number who can diagnose a slow query.

Managed Kafka removes much of the first list and none of the last one — you still choose partition counts, still design keys, still get rebalance storms, still own consumer lag. It converts an operations problem into a bill (Module O-8), and cross-AZ replication traffic on a high-throughput topic is a large line on it.

Which leads to the decision rule. Use a queue (Module P-14) when you have work to distribute. Use a log when you have facts to distribute. Concretely, Kafka earns its cost when at least two of these hold:

Multiple independent consumers need the same stream, and you would otherwise be fanning out to N queues.
Replay has real value — reprocessing history to fix a bug, backfilling a new service, rebuilding a search index (Module P-20).
Throughput is genuinely high — sustained tens of MB/s or more, where per-message broker bookkeeping starts to matter.
Ordering per key is a requirement, not a nicety.

If you have one producer, one consumer, and moderate volume, a queue is a smaller system that fails in fewer ways, and choosing it is the senior decision. Kafka is not the upgrade from a queue. It is a different tool with a different shape and a much larger operational footprint.

Where This Shows Up in Your Stack

PostgreSQL: the write-ahead log is the same idea — append-only, sequential, replayable — which is why Postgres logical replication is the natural source for streaming database changes into Kafka (Module P-23). It is also the destination side of the dual-write problem from Module F-14: writing to Postgres and publishing to Kafka in two steps has no atomicity, and the outbox pattern (Module A-3) is how you fix it.
Node.js: the consumer's poll loop is on the event loop, so any synchronous CPU-heavy work in a handler delays the heartbeat thread's cooperation and the next poll() — a JSON parse over large payloads is a plausible rebalance trigger. Keep handlers await-shaped with explicit timeouts, and move heavy transforms to worker threads.
Redis: Redis Streams offers the log shape — consumer groups, explicit acknowledgement, XAUTOCLAIM for stuck entries — at a much smaller operational cost, with in-memory capacity limits and weaker durability. For moderate volume with short retention it is often the right answer and does not require a new cluster (Module P-12).
Kafka Connect / stream processing: for moving data in and out, prefer a maintained connector over a bespoke consumer. Most consumers people write are re-implementations of a sink connector, complete with a fresh set of offset-commit bugs.

Summary

Kafka productises Module F-2's finding — that a sequential access pattern beats a random one by an order of magnitude in useful throughput — by supporting exactly one access pattern. The cost is that it removes almost every database feature to do it.
The primary abstraction is an append-only, retained, re-readable log, not a queue. Consumption advances an offset instead of destroying a message, which is what makes fan-out and replay possible — and what means there is no per-message retry, delay, or dead-letter behaviour.
Speed comes from sequential append plus the OS page cache plus zero-copy sendfile. Give brokers RAM for the kernel, not for the heap, and remember that TLS and format conversion break the zero-copy path and turn a network-bound cluster into a CPU-bound one.
Partitions are both the parallelism ceiling and the ordering boundary. A partition goes to one consumer per group, so consumers beyond the partition count idle. Keys route records via hash(key) % numPartitions, which buys same-key ordering and never global ordering — at the cost of hot partitions when the key is skewed.
The consumer commits its own offsets, and where the commit sits decides the semantic: commit before processing is at-most-once and loses records; commit after is at-least-once and duplicates them. Exactly-once is real but scoped to Kafka-to-Kafka, so any external side effect returns you to at-least-once.
Rebalance storms are the signature outage: one slow consumer misses max.poll.interval.ms, an eager rebalance stops consumption for the whole group, the backlog grows, and the larger backlog causes the next miss. Mitigate with smaller max.poll.records, static membership, and cooperative rebalancing — each of which trades faster failure detection for stability.
Consumer lag is the metric that matters, watched per partition, as a trend rather than a level, expressed in time, and alerted on as a fraction of the retention window before records are deleted unread.
Retention deletes by age; cleanup.policy=compact retains the latest record per key forever, making the log a durable state store you can replay to rebuild state. It costs you history, requires last-write-wins handling of not-yet-compacted duplicates, and can resurrect deleted keys if a consumer is offline past delete.retention.ms.
Partition count is nearly irreversible. It cannot be decreased, and increasing it rehashes every key and breaks ordering retroactively. Over-provision at creation, paying with file handles, slower failover, longer rebalances, and worse producer batching.
The real cost is operational: replication tuning, disk planning, broker rebalancing, stateful upgrades, and scarce expertise. Choose a log over a queue when you have multiple consumers, real replay value, high throughput, or a per-key ordering requirement — not by default.

The next module, Idempotency and Deduplication, picks up the loose end this one deliberately left. We concluded that at-least-once is the delivery semantic you should choose and that duplicates are therefore certain — not a rare failure but normal operation, generated by every rebalance and every retry. Module P-16 is how you make a handler that can be run twice on the same record and still be correct: idempotency keys, deduplication windows, natural idempotence in the data model, and the distinction between an operation that is safe to repeat and one that merely appears to be.

Knowledge Check

A consumer group with 12 partitions and 12 instances falls behind during a traffic spike. Lag climbs steadily, and the group's logs show repeated rebalances. An engineer scales the deployment from 12 to 30 instances to catch up. Lag gets worse. Why?

An inventory service consumes stock-events and occasionally applies a stock_decremented event before the stock_replenished event that preceded it, producing negative stock. The topic has 24 partitions; the producer sends events without a key. The team's design doc says "Kafka guarantees ordering." What is wrong and what is the fix, with its cost?

A payments consumer commits offsets after each handler completes. A pod is killed mid-batch during a deploy, and on restart three customers see duplicate charges. A senior engineer says the commit placement is correct and should not change. What is the reasoning, and what should actually change?

Test your knowledge with more question sets

Sign in to keep reading

The rest of this module is free — sign in with Google to unlock it and track your progress.

PreviousModule P-14: Message Queues Next Module P-16: Idempotency and Deduplication

Discussion

Join the discussion

Loading comments...