Node.js In-Depth
Module A-10·19 min read

When to decouple — the outbox pattern, datastore split strategies, and transitioning a write-heavy module out of the monolith without data loss.

Written by Jatin Jain Saraf · Architect of SupraScan

Module 9 — Pragmatic Microservice Deconstruction: Splitting Ingestion from Analytics

What this module covers: The Modulith from Module 8 is the right starting architecture. But eventually a specific module genuinely needs independent scaling, geographic distribution, or a different runtime. Extracting a module incorrectly causes data loss, split-brain state, and deployment coupling that defeats the purpose of the split. This module covers how to identify the correct seam to split, the outbox pattern for atomically publishing events during extraction, datastore split strategies, and the operational reality of running two services where there was one.

When to Split: The Three Legitimate Signals

The decision to split a module out of the Modulith should be data-driven. There are exactly three legitimate reasons:

Signal 1: Independent scaling requirement

A module needs 10× the compute of others. The analytics module needs 32 CPU cores for aggregation while ingestion needs 4. Scaling the whole Modulith to 32 cores wastes 28 cores of ingestion capacity.

// Measure per-module CPU cost
const { cpuUsage } = process;
const before = cpuUsage();
await analyticsModule.processTransaction(tx);
const usage = cpuUsage(before);
console.log(`Analytics CPU: ${usage.user}μs user, ${usage.system}μs system`);

If analytics consistently uses 15× more CPU than ingestion per transaction, that's a scaling signal.

Signal 2: Different availability requirements

Ingestion must be 99.99% available (data loss on downtime). Analytics can tolerate 99.9% (reports are slightly stale during an outage). Running them together means analytics bugs can take down ingestion — an unacceptable risk profile.

Signal 3: Genuine team autonomy need

A separate team owns analytics and deploys 10 times per day. Coupling their deployment to ingestion (which deploys once a week) creates a deployment bottleneck. Conway's Law applies.

Do NOT split for:

  • "Microservices are the modern way" (cargo cult)
  • The module is large (size is not a reason for distribution)
  • You want to use a different language (use a native addon instead)
  • "It might need to scale later" (YAGNI — split when the signal is real)

Identifying the Split Seam

The correct seam for splitting is where write contention and read contention diverge.

For a blockchain indexer:

Write path:    incoming block → parse → validate → INSERT into transactions table
Read path:     API queries → SELECT aggregations, lookups, analytics
 
Both paths hit the same PostgreSQL primary.
At 50K writes/sec + 10K reads/sec, the primary's write throughput is the constraint.
Reads on the primary compete with writes for buffer pool, WAL, and I/O.

The seam: separate the write service from the read/analytics service. The write service owns the primary. The analytics service reads from a read replica or a separate projection store.

Before split:
  [Modulith] → writes transactions → PostgreSQL primary
  [Modulith] → reads analytics ← PostgreSQL primary (competing I/O)
 
After split:
  [Ingestion Service] → writes transactions → PostgreSQL primary
  [Ingestion Service] → publishes events → Kafka topic
  [Analytics Service] → consumes Kafka → builds projections → PostgreSQL analytics (separate)
  [Analytics Service] → serves read API

Now writes and reads never compete. The ingestion service can sustain 50K writes/sec at full I/O. Analytics has its own PostgreSQL instance sized for reads.

Naming the approach: everything this module describes — carving analytics out from behind the same public interface it always had, routing its traffic through the outbox/Kafka path while the Modulith keeps running, then retiring the in-process code path once the new service is proven — is the strangler fig pattern. New functionality grows up around the old implementation and gradually takes over its load, rather than a big-bang rewrite-and-cutover. The name comes from the strangler fig vine, which grows around a host tree and eventually replaces it entirely without ever taking the tree down first.

The Outbox Pattern: Atomic Write + Event Publish

The most dangerous moment in a service split is when a write to the database and a publish to Kafka must be treated as a unit. If you write to the database and then publish to Kafka, a crash between the two leaves the database updated but Kafka unnotified — downstream services miss the event.

// BROKEN: non-atomic write + publish
async function ingestTransaction(tx) {
  await db.write(tx);              // step 1: DB write succeeds
  // CRASH HERE → Kafka never gets the event → analytics never updates
  await kafka.publish('transactions', tx);  // step 2: may never execute
}

The outbox pattern solves this by writing the event to an outbox table in the same database transaction as the business data. Atomicity is guaranteed by the database. A separate process reads the outbox and publishes to Kafka.

Think of the outbox row as a coat-check ticket: the coat is guaranteed to be in the room the instant you get the ticket (the DB commit), even if the runner who fetches it for you (the publisher) is a few minutes late. The event's existence is settled at commit time; its delivery to Kafka is merely a formality that catches up afterward.

// Step 1: Write transaction + outbox event in ONE database transaction
async function ingestTransaction(tx) {
  const client = await pool.connect();
  try {
    await client.query('BEGIN');
 
    // Write business data
    await client.query(
      'INSERT INTO transactions (hash, block_height, sender, amount) VALUES ($1, $2, $3, $4)',
      [tx.hash, tx.blockHeight, tx.sender, tx.amount]
    );
 
    // Write outbox event — same transaction, atomically
    await client.query(
      'INSERT INTO outbox (event_type, payload, created_at) VALUES ($1, $2, NOW())',
      ['TRANSACTION_INGESTED', JSON.stringify(tx)]
    );
 
    await client.query('COMMIT');
    // Either BOTH succeed or BOTH fail. Never one without the other.
  } catch (err) {
    await client.query('ROLLBACK');
    throw err;
  } finally {
    client.release();
  }
}
// Step 2: Outbox publisher — reads unprocessed events, publishes to Kafka
async function outboxPublisher() {
  while (true) {
    const { rows } = await pool.query(
      'SELECT id, event_type, payload FROM outbox WHERE published_at IS NULL ORDER BY id LIMIT 100 FOR UPDATE SKIP LOCKED'
    );
 
    if (rows.length === 0) {
      await sleep(100);  // no events, wait briefly
      continue;
    }
 
    // Publish to Kafka
    await kafka.sendBatch(rows.map(row => ({
      topic: 'transactions',
      messages: [{ key: row.id.toString(), value: row.payload }]
    })));
 
    // Mark as published
    await pool.query(
      'UPDATE outbox SET published_at = NOW() WHERE id = ANY($1)',
      [rows.map(r => r.id)]
    );
  }
}

Why FOR UPDATE SKIP LOCKED: Multiple outbox publisher instances can run safely. Each grabs a batch of rows that no other publisher is currently processing. No duplicate publishes.

The at-least-once guarantee: if the publisher crashes after publishing to Kafka but before marking the rows, it will re-publish on restart. The Kafka consumer must be idempotent — processing the same event twice must be safe.

Kafka Consumer: Idempotent Processing

Sign in to keep reading

The rest of this module is free — sign in with Google to unlock it and track your progress.

Sign in & Register
PreviousModule A-9: The Modern Hybrid Monolith: High-Throughput Modulith ArchitectureNext Module A-11: High-Performance IPC: gRPC, Kafka & Event Streams

Discussion

0

Join the discussion

Loading comments...
PortfolioBlog

© 2026 Jatin Jain Saraf (JJS). All rights reserved.