Module 5 — Native Streams & Off-Heap Buffer Storage

What this module covers: Processing gigabytes of blockchain transaction logs or UPI settlement files without crashing the V8 runtime requires understanding the boundary between V8-managed memory and memory that lives completely outside it. Buffer in Node.js allocates from the OS directly, bypassing V8's heap and garbage collector entirely. Custom Transform streams process that memory in chunks, keeping your heap stable regardless of input size. This module covers the precise memory model for off-heap storage, the correct implementation of Transform streams for production ingestion pipelines, and streaming pipeline composition with pipeline().

The Memory Boundary: V8 Heap vs Off-Heap

In Module 1 you learned that V8 manages its own heap — New Space for short-lived objects, Old Space for long-lived ones. All standard JavaScript objects live here: arrays, plain objects, strings, closures.

Node.js adds a second memory region: off-heap memory — memory allocated directly from the OS via malloc, completely outside V8's control. V8 has no visibility into it, no GC responsibility for it, and cannot trigger GC based on its size.

This distinction is critical for ingestion pipelines:

When you process a 500MB blockchain block file with Buffer, V8's heap stays flat. The data never touches the GC. The only heap allocation is a small Buffer JavaScript object (~100 bytes) that wraps the off-heap memory pointer.

Buffer Internals: Three Allocation Methods

Buffer.alloc(size)

Allocates size bytes of off-heap memory and zeros it. Safe for security-sensitive operations where you cannot allow previous memory contents to be readable. Slower due to the memset.

Use when: the buffer will be read before being fully written (zero prevents reading stale data), or the data is security-sensitive.

Buffer.allocUnsafe(size)

Allocates size bytes of off-heap memory without zeroing. The memory may contain previous data from other allocations. Faster — no memset.

For sizes ≤ 8KB: allocates from an 8KB pooled slab that libuv pre-allocates. Multiple small Buffer.allocUnsafe calls share the same underlying memory slab, reducing syscall overhead.

Use when: you will immediately write to the buffer before reading, and the data is not security-sensitive. Most ingestion pipelines fall into this category — you write parsed transaction data into the buffer immediately.

Buffer.allocUnsafeSlow(size)

Allocates size bytes of off-heap memory bypassing the 8KB slab pool. Each call requests memory directly from the OS (via malloc). Slower than allocUnsafe for small buffers, but avoids fragmentation for large allocations.

Use when: allocating large buffers (> 8KB) that will live for a long time. Avoids consuming the shared slab, which would be a waste for large chunks.

Buffer vs TypedArray vs ArrayBuffer

For ingestion pipelines: use Buffer. It has the convenience methods (read/write integers at offsets, copy, compare) needed for binary protocol parsing. TypedArray is better for numeric computation where V8's typed array optimization applies.

Transform Streams: The Core of Every Ingestion Pipeline

A Transform stream is both a Readable and a Writable. It receives data chunks in, transforms them, and emits transformed chunks out. It is the correct abstraction for:

• Decoding raw binary transaction data into JavaScript objects
• Parsing line-delimited JSON from a log file
• Decompressing compressed data streams
• Normalizing inconsistent data formats

The _transform Method

The _flush Method

_flush is called when the upstream has ended and there is no more data coming. It is your chance to emit any final output from accumulated state. Always call callback() — forgetting it hangs the pipeline.

pipeline() vs .pipe(): Always Use pipeline()

.pipe() has a fatal flaw: it does not handle errors correctly. If a stream in the chain errors, other streams are not automatically destroyed. You get resource leaks.

pipeline() with Async Generators

The most flexible form of pipeline() uses async generators as transform stages:

Off-Heap Buffer Pooling for Zero-Allocation Hot Paths

When your hot path processes millions of transactions per second, even the overhead of Buffer.allocUnsafe() accumulates. Pre-allocate a pool of reusable buffers:

Processing Multi-Gigabyte Settlement Files: A Complete Example

UPI settlement files are batch files containing all payment transactions for a settlement period. A large payment processor might generate 2–5GB settlement files every hour. Processing these requires streaming — loading the full file into memory would require 2–5GB of heap.

Memory profile during this operation on a 3GB file:

• V8 heap: ~40–80MB (the pipeline stream objects, parsed records in flight)
• Off-heap: up to 4MB (the read HWM) + zlib decompression buffers
• Total process memory: < 200MB regardless of file size

zlib Streaming: Decompressing Large Blockchain Archives

zlib uses off-heap buffers internally. The decompression window (32KB by default) lives outside V8's heap. You can tune it:

Summary

Off-heap Buffers and streams handle data throughput. Module 6 handles CPU throughput — how to saturate all cores on a large server using cluster and worker_threads, and how to minimize the IPC overhead that typically defeats the purpose of multi-process scaling.

Next: Module 6 — Core Scaling: Multi-Process Clustering & IPC Latency →