Module 4 — The HTTP/TCP Subsystem & Ingestion Backpressure

The Problem: Producers Outpacing Consumers

A blockchain indexer at peak load has an inherent mismatch:

• Producer: blockchain full node pushing 50,000 transaction events/second over TCP
• Consumer: PostgreSQL database accepting 8,000–12,000 writes/second

Without backpressure, the process does this:

javascript
// What happens without backpressure:
socket.on('data', (chunk) => {
  const transactions = parse(chunk);
  for (const tx of transactions) {
    db.write(tx);  // returns a Promise, does NOT wait for it
  }
});
// At 50K tx/sec ingest, 8K/sec db write:
// After 1 second: 42K tx queued in memory
// After 10 seconds: 420K tx queued → ~500MB memory
// After 30 seconds: OOM kill

The fix requires two layers of backpressure working together:

1. TCP flow control — the kernel tells the sender to slow down when the receive buffer is full
2. Stream backpressure — Node.js pauses the socket read when the downstream consumer is busy

Understanding how these two layers interact is the key to building a pipeline that never OOMs under load.

TCP Flow Control: The Kernel's Backpressure

TCP has backpressure built in via the receive window mechanism.

Every TCP ACK includes a window field — the number of bytes the receiver is willing to accept. When the kernel's receive buffer fills up, it advertises a smaller window to the sender. When the buffer is full, it advertises zero window — the sender must stop completely.

Normal flow:
  Sender → [50KB data] → Receiver
  Sender ← [ACK, window=65536] ← Receiver (plenty of buffer)
  Sender → [50KB more data] → Receiver

Buffer pressure:
  Sender → [50KB data] → Receiver (buffer 95% full)
  Sender ← [ACK, window=4096] ← Receiver (only 4KB left)
  Sender → [4KB data] → Receiver (respects window)

Buffer full:
  Sender → [data] → Receiver (buffer 100% full)
  Sender ← [ACK, window=0] ← Receiver (zero window — STOP)
  Sender: blocked. Waiting for window update.
  ... later, when Node.js drains the buffer ...
  Sender ← [window update, window=65536] ← Receiver
  Sender resumes.

The critical insight: TCP flow control automatically propagates backpressure upstream. When Node.js stops reading from the socket (because downstream is slow), the kernel receive buffer fills, the window shrinks to zero, and the blockchain full node is forced to stop sending. The backpressure is communicated all the way back to the data source — no data is dropped, it is simply slowed.

Node.js Streams: Application-Level Backpressure

Node.js streams implement the same backpressure mechanism at the JavaScript level, mirroring TCP flow control.

The highWaterMark (HWM)

Every writable stream has a highWaterMark — the maximum number of bytes (or objects) it is willing to buffer before signalling that it is full.

javascript
// Default HWMs:
// Byte streams: 16KB (16384 bytes)
// Object mode streams: 16 objects

// Custom HWM for a database write stream:
const dbWriteStream = new Writable({
  objectMode: true,
  highWaterMark: 1000,  // buffer up to 1000 objects before signalling full
  write(transaction, encoding, callback) {
    db.write(transaction)
      .then(() => callback())
      .catch(callback);
  }
});

The write() Return Value: The Backpressure Signal

When you write to a writable stream, it returns a boolean:

• true — buffer is below HWM, safe to continue writing
• false — buffer has reached HWM, you should stop writing

javascript
// The contract:
const canContinue = writable.write(chunk);

if (!canContinue) {
  // STOP writing. The stream has too much buffered.
  // Wait for the 'drain' event before writing more.
}

The drain Event: The Resume Signal

When the stream's internal buffer empties below HWM after being full, it emits drain. This is your signal to resume writing.

javascript
// Correct backpressure implementation for a DB write pipeline
function writeWithBackpressure(stream, data) {
  const canContinue = stream.write(data);
  if (!canContinue) {
    return new Promise(resolve => stream.once('drain', resolve));
  }
  return Promise.resolve();
}

// Usage in an ingestion loop
async function processIncoming(socket, dbStream) {
  for await (const chunk of socket) {
    const transactions = parseChunk(chunk);
    for (const tx of transactions) {
      await writeWithBackpressure(dbStream, tx);
      // If dbStream is full, this await suspends here
      // The for-await loop pauses, which pauses the socket read
      // Socket pause triggers TCP receive buffer to fill
      // TCP receive buffer full → zero window → sender pauses
      // Backpressure propagated all the way to the data source
    }
  }
}

readable.pause() and readable.resume()

For explicit backpressure control on a readable stream:

javascript
const socket = net.createConnection({ host, port });

let writesPending = 0;
const MAX_PENDING = 5000;

socket.on('data', (chunk) => {
  const transactions = parse(chunk);

  for (const tx of transactions) {
    writesPending++;

    db.write(tx).finally(() => {
      writesPending--;
      // If socket was paused due to queue depth, resume
      if (writesPending < MAX_PENDING / 2 && socket.isPaused()) {
        socket.resume();
      }
    });
  }

  // Pause if too many writes are in flight
  if (writesPending >= MAX_PENDING) {
    socket.pause();  // stops reading from TCP receive buffer
    // kernel receive buffer fills → window shrinks → sender slows
  }
});

The Danger of Unpaused Readable Streams

A net.Socket is a Readable stream. In Node.js, a readable stream that is not being consumed operates in two modes:

Flowing mode (default when a data event listener is attached): data is emitted as fast as it arrives. If you can't process it, it accumulates in memory.

Paused mode (after socket.pause()): data accumulates in the kernel receive buffer. No JavaScript memory consumption. TCP flow control does its job.

The correct pattern for a high-throughput ingestion socket: monitor write queue depth and pause/resume the socket accordingly.

HTTP Ingestion: IncomingMessage as a Stream

http.IncomingMessage (the req object in an HTTP server) is a Readable stream backed by the underlying TCP socket. The same backpressure rules apply.

javascript
// Wrong: buffering the entire body before processing
app.post('/ingest', (req, res) => {
  let body = '';
  req.on('data', chunk => body += chunk);  // accumulates entire body in memory
  req.on('end', () => {
    const transactions = JSON.parse(body);   // synchronous parse blocks event loop
    processAll(transactions);
    res.sendStatus(200);
  });
});

// Correct: streaming processing with backpressure
app.post('/ingest', (req, res) => {
  const parser = createStreamingParser();   // Transform stream
  const dbWriter = createDbWriteStream();   // Writable stream with HWM

  req
    .pipe(parser)       // parse chunks as they arrive
    .pipe(dbWriter)     // write parsed transactions, apply backpressure
    .on('finish', () => res.sendStatus(200))
    .on('error', (err) => res.status(500).json({ error: err.message }));
});

With .pipe(), backpressure is handled automatically:

• If dbWriter returns false from write(), parser pauses
• If parser is paused, req pauses
• If req is paused, the underlying TCP socket is paused
• TCP socket paused → kernel receive buffer fills → sender slows

writable.writableLength and writable.writableHighWaterMark

Real-time monitoring of stream buffer state:

javascript
// Monitor backpressure state in production
const dbStream = createDbWriteStream({ highWaterMark: 1000 });

setInterval(() => {
  const utilization = dbStream.writableLength / dbStream.writableHighWaterMark;
  console.log(`DB write buffer: ${(utilization * 100).toFixed(0)}% full`);

  if (utilization > 0.9) {
    console.warn('DB write stream near capacity — upstream should pause');
  }
}, 1000);

HTTP/2: Multiplexed Streams for Persistent Ingestion Connections

For blockchain indexers that maintain persistent connections to full nodes, HTTP/2 provides stream multiplexing over a single TCP connection.

javascript
import http2 from 'node:http2';

// HTTP/2 server for persistent block subscription connections
const server = http2.createSecureServer({ key, cert });

server.on('stream', (stream, headers) => {
  // Each HTTP/2 stream is independent and multiplexed over one TCP connection
  // Backpressure works per-stream, not per-connection

  stream.on('data', (chunk) => {
    const canContinue = processChunk(chunk);
    if (!canContinue) {
      stream.pause();  // pause this stream's flow
    }
  });

  stream.on('drain', () => stream.resume());
});

Benefit for blockchain indexers: instead of 1,000 separate TCP connections from 1,000 full nodes, use 10 HTTP/2 connections each multiplexing 100 streams. Fewer file descriptors, fewer TCP handshakes, same data throughput. Each stream has independent backpressure.

Handling Socket Floods: server.maxConnections

When a blockchain network goes through a major upgrade, every node may attempt to reconnect simultaneously (similar to the thundering herd from Module 3). server.maxConnections provides a hard cap:

javascript
const server = net.createServer(handleConnection);
server.maxConnections = 5000;   // refuse connections beyond this

// When maxConnections is reached:
// - New TCP SYN packets are rejected
// - Clients receive connection refused immediately
// - Cleaner than accepting and then hanging

server.listen(3000);

// Monitor connection count
setInterval(() => {
  server.getConnections((err, count) => {
    if (!err) {
      console.log(`Active connections: ${count}/${server.maxConnections}`);
    }
  });
}, 5000);

socket.setTimeout(): Eliminating Zombie Connections

Connections that are established but never send data consume file descriptors indefinitely without a timeout:

javascript
server.on('connection', (socket) => {
  // Kill connections that have been idle for 30 seconds
  socket.setTimeout(30_000);
  socket.on('timeout', () => {
    socket.destroy();  // force close — no graceful shutdown for idle sockets
  });

  // Reset timeout on activity
  socket.on('data', () => socket.setTimeout(30_000));
});

Rate Limiting at the TCP Layer

Before requests reach your application routing logic, you can enforce rate limits at the connection level using token bucket algorithms:

javascript
// Token bucket rate limiter per IP address
class RateLimiter {
  #buckets = new Map();
  #rate;      // tokens per second
  #capacity;  // maximum bucket depth

  constructor({ rate, capacity }) {
    this.#rate = rate;
    this.#capacity = capacity;
  }

  consume(ip) {
    const now = Date.now();
    let bucket = this.#buckets.get(ip);

    if (!bucket) {
      bucket = { tokens: this.#capacity, lastRefill: now };
      this.#buckets.set(ip, bucket);
    }

    // Refill based on elapsed time
    const elapsed = (now - bucket.lastRefill) / 1000;
    bucket.tokens = Math.min(
      this.#capacity,
      bucket.tokens + elapsed * this.#rate
    );
    bucket.lastRefill = now;

    if (bucket.tokens >= 1) {
      bucket.tokens -= 1;
      return true;   // allowed
    }
    return false;    // rate limited
  }
}

const limiter = new RateLimiter({ rate: 1000, capacity: 5000 });

server.on('connection', (socket) => {
  const ip = socket.remoteAddress;

  socket.on('data', (chunk) => {
    if (!limiter.consume(ip)) {
      socket.destroy();   // hard limit exceeded — drop connection
      return;
    }
    processData(chunk);
  });
});

The Production Incident: OOM from Missing Backpressure During Airdrop

Context: A UPI payment processing gateway ingesting payment events from multiple upstream aggregators. Normal throughput: 3,000 events/second. During a major e-commerce sale, 35,000 events/second arrived simultaneously.

The broken pipeline:

javascript
// Original ingestion handler — no backpressure
socket.on('data', (chunk) => {
  const payments = parsePayments(chunk);
  payments.forEach(payment => {
    // db.write returns a Promise — not awaited
    db.write(payment).then(() => emitConfirmation(payment));
  });
});

At 35,000 events/sec with 8,000 db writes/sec capacity:

• Second 1: 27,000 writes queued in memory
• Second 5: 135,000 writes queued → ~270MB
• Second 12: ~380,000 writes queued → ~760MB
• Second 14: Process OOM-killed by the OS

The Kubernetes pod restarted. On restart, the TCP connections reconnected, and the same flood resumed. The pod crashed again in 14 seconds. Kubernetes kept restarting it. This cycle continued for 8 minutes until the upstream aggregators hit their own connection retry limits.

The fix — stream pipeline with backpressure:

javascript
// Step 1: Create a write stream with appropriate HWM
const paymentWriteStream = new Writable({
  objectMode: true,
  highWaterMark: 500,  // buffer max 500 payments
  write(payment, _, callback) {
    db.write(payment)
      .then(() => {
        emitConfirmation(payment);
        callback();
      })
      .catch(callback);
  }
});

// Step 2: Create a parse transform stream
const parseStream = new Transform({
  readableObjectMode: true,
  transform(chunk, _, callback) {
    const payments = parsePayments(chunk);
    payments.forEach(p => this.push(p));
    callback();
  }
});

// Step 3: Wire with pipe — backpressure automatic
socket
  .pipe(parseStream)
  .pipe(paymentWriteStream);

// pipe() handles pause/resume automatically:
// paymentWriteStream full → parseStream pauses
// parseStream paused → socket pauses
// socket paused → TCP receive buffer fills
// TCP buffer fills → window = 0 → upstream aggregator slows

Result: Under 35,000 events/sec, the pipeline automatically throttled to 8,000/sec — the database's write capacity. Memory stabilized at ~12MB (the 500-object buffer × ~24KB per object). No OOM. The upstream aggregator experienced slightly higher latency during the peak, but no data was lost.

Backpressure Monitoring

javascript
// Emit backpressure metrics for observability
const backpressureCounter = new Counter({
  name: 'nodejs_stream_backpressure_total',
  help: 'Number of times backpressure was applied to incoming socket'
});

socket.on('data', (chunk) => {
  const canContinue = parseStream.write(chunk);
  if (!canContinue) {
    backpressureCounter.inc();
    socket.pause();
    parseStream.once('drain', () => socket.resume());
  }
});

Tracking nodejs_stream_backpressure_total over time tells you exactly how often your pipeline is capacity-constrained — invaluable for right-sizing your database write pool and scaling decisions.

Summary

Backpressure prevents memory exhaustion from the outside in. Module 5 goes inside — how to process gigabytes of data without touching the V8 heap at all, using off-heap Buffers, Transform streams, and streaming pipelines for blockchain transaction log processing.

Next: Module 5 — Native Streams & Off-Heap Buffer Storage →