Module A-21·13 min read

Structured JSON diagnostic reports at the exact moment of failure — integrating into SRE pipelines without the overhead of full core dump analysis.

JJS

Written by Jatin Jain Saraf · Senior Software Engineer

Module 20 — Automated Post-Mortem Diagnostics: `process.report`

Q: How can you configure Node.js to automatically capture the V8 heap state and open file descriptors exactly when the process crashes from an Out of Memory (OOM) error?

Enable `process.report.reportOnFatalError = true` at application startup. — Setting `process.report.reportOnFatalError = true` automatically generates a structured JSON diagnostic report (containing the heap state, libuv handles, native stacks, etc.) exactly at the moment the process encounters a fatal error like an OOM crash.

Q: What does the presence of hundreds of identical outgoing TCP connection entries (e.g., `{ "type": "tcp", "address": "db.internal", "port": 5432 }`) in the `libuv` section of a `process.report` most likely indicate?

The application has a connection leak, opening database connections without returning them to the pool. — The `libuv` section lists all open file descriptors and active handles. Hundreds of open outbound TCP connections to the same database port strongly indicates a connection leak, where the app opens connections but fails to close or pool them properly.

Q: What is the recommended way to reliably capture and preserve a `process.report` when Kubernetes shuts down a pod via a `SIGTERM` signal?

Listen for the `SIGTERM` event, call `process.report.writeReport()`, manually upload the file to persistent storage, and then gracefully shut down. — Because Kubernetes pod storage is ephemeral, simply writing the report to disk when the pod shuts down means the data is lost. Manually calling `process.report.writeReport()` inside a `SIGTERM` handler allows you to await the upload of the file to a persistent location (like S3) before finishing the shutdown process.

What this module covers: When a Node.js process crashes with an uncaught exception, OOM kill, or fatal error, you typically have one chance to capture diagnostic information before the process terminates. process.report generates a structured JSON diagnostic report containing the V8 heap state, libuv handle and request queues, native C++ call stacks, environment variables, and system information — captured at the exact moment of failure. This module covers configuring diagnostic reports for production, integrating report generation into SRE alert pipelines, and reading the output to diagnose crashes that are otherwise invisible.

What a Diagnostic Report Contains

process.report is Node's flight data recorder — you never want to need it, but when the process goes down, it's the only artifact that survives to tell you what the engines looked like in the final second.

A process.report output is a JSON file with these sections:

json

{
  "header": {
    "reportVersion": 3,
    "event": "OOMError",
    "trigger": "OutOfMemory",
    "filename": "report.20260517.143022.json",
    "dumpEventTime": "2026-05-17T14:30:22.441Z",
    "processId": 12847,
    "cwd": "/app",
    "commandLine": ["node", "--max-old-space-size=2048", "dist/app.js"],
    "nodeVersion": "v22.3.0",
    "release": { "name": "node", "lts": "Jod" }
  },
  "javascriptStack": {
    "message": "JavaScript heap out of memory",
    "stack": "FATAL ERROR: Reached heap limit Allocation failed..."
  },
  "nativeStack": [
    "#0 0x10f3c node::MakeCallback",
    "#1 0x23a1 uv__io_poll",
    "..."
  ],
  "javascriptHeap": {
    "totalMemory": 2147483648,
    "usedMemory": 2145001472,
    "externalMemory": 8192000,
    "heapSpaces": {
      "new_space": { "memorySize": 33554432, "committedMemory": 33521664, "usedMemory": 33501200 },
      "old_space": { "memorySize": 2097152000, "committedMemory": 2097152000, "usedMemory": 2095874000 }
    }
  },
  "uvthreadResourceUsage": { "userCpuSeconds": 847.23, "kernelCpuSeconds": 12.44 },
  "libuv": [
    { "type": "tcp", "address": "0.0.0.0", "port": 3000, "fd": 10, "is_active": true, "is_referenced": true },
    { "type": "tcp", "address": "db.internal", "port": 5432, "fd": 22, "is_active": true, "sends_size": 0 }
  ],
  "workers": [],
  "environmentVariables": {
    "DATABASE_URL": "postgres://indexer_svc:Sup3rSecretPW@db.internal:5432/txns",
    "NODE_ENV": "production",
    "UV_THREADPOOL_SIZE": "16"
  },
  "resourceUsage": { "rss": 2478080000, "heapTotal": 2147483648, "heapUsed": 2145001472 }
}

In one file, you have: the exact error, the V8 heap state at failure time, every open file descriptor (TCP connections, sockets), environment variables, and CPU usage — without needing to attach a debugger or reproduce the crash.

Operational caveat: report generation can itself fail under severe OOM. Writing a report requires allocating memory to serialize the heap and handle state to JSON — if the process is already at the absolute edge of its memory limit when the fatal error fires, there may not be enough headroom left to produce the report at all, or the report may be truncated mid-write (a partial JSON file, or one missing the javascriptHeap section entirely). This is not a reason to skip reportOnFatalError — a truncated report with a stack trace is still far more useful than nothing — but don't assume the report is guaranteed to be complete during the worst OOM events, which are exactly the events where you need it most.

This is a real risk, not a hypothetical one. process.report does NOT redact anything by default — environmentVariables above is the raw, unmodified process.env, verbatim. If your process has DATABASE_URL, API_KEY, JWT secrets, or any other credential in its environment (which is the normal way to configure a production Node.js service), every diagnostic report contains those credentials in plaintext. A report generated on OOM or uncaught exception, written to disk and then uploaded to S3 or attached to a ticket, is a credential leak waiting to happen unless you explicitly redact it first — see the redaction step below, which must run before the report is persisted or shipped anywhere.

Configuration: Triggering Reports Automatically

javascript

// Configure automatic report generation at application startup

// Report on any fatal error (OOM, uncaught exception, SIGTERM)
process.report.reportOnFatalError = true;

// Report on uncaught exceptions (before process exits)
process.report.reportOnUncaughtException = true;

// Report on SIGUSR2 signal (manual trigger without process restart)
process.report.reportOnSignal = true;
process.report.signal = 'SIGUSR2';  // default

// Where to write reports
process.report.directory = '/app/diagnostic-reports';
process.report.filename = 'report.{date}.{time}.{pid}.json';
// Variables available: {date}, {time}, {pid}, {tid}, {hostname}, {timestamp}

bash

# Trigger a report manually without restarting
kill -SIGUSR2 $(pgrep -f "node dist/app.js")
# Generates: /app/diagnostic-reports/report.20260517.143022.12847.json
# Process continues running — zero impact on production traffic

CLI-Flag Equivalents

Every option above can also be set as a startup flag, which is often preferable in containerized deployments where you want the behavior locked in from process launch — before any application code has a chance to run (or fail to run):

bash

# Equivalent to setting the properties programmatically at startup, but active
# from the very first tick — including crashes during module initialization,
# before your own code would have had a chance to set process.report.* at all
node \
  --report-uncaught-exception \
  --report-on-fatalerror \
  --report-on-signal \
  --report-directory=/app/diagnostic-reports \
  --report-filename=report.{date}.{time}.{pid}.json \
  --report-compact \
  dist/app.js

Flag	Equivalent to
`--report-uncaught-exception`	`process.report.reportOnUncaughtException = true`
`--report-on-fatalerror`	`process.report.reportOnFatalError = true`
`--report-on-signal`	`process.report.reportOnSignal = true`
`--report-directory=<dir>`	`process.report.directory = '<dir>'`
`--report-compact`	Single-line JSON (no pretty-printing) — smaller files, easier to pipe into log aggregators that expect one JSON object per line

For a Dockerized indexer, baking these flags into the container's ENTRYPOINT/CMD (or a NODE_OPTIONS env var) guarantees report generation is armed regardless of how the application code initializes — including if the crash happens before your own process.report.reportOnFatalError = true line would have executed.

Redacting Secrets Before a Report Ever Leaves the Box

Because environmentVariables is raw process.env, every report must be scrubbed before it is written anywhere durable or uploaded anywhere. Never skip this step — an unredacted report is a plaintext credential dump.

javascript

// redact-report.js
import { readFileSync, writeFileSync } from 'node:fs';

// Keys matched case-insensitively. Extend this list for your own secret naming conventions.
const SENSITIVE_KEY_PATTERNS = [
  /_SECRET$/i, /_KEY$/i, /_TOKEN$/i, /_PASSWORD$/i, /^PASSWORD$/i,
  /_URL$/i,        // catches DATABASE_URL, REDIS_URL, etc. (may embed creds)
  /_DSN$/i, /_CREDENTIALS?$/i, /^AWS_/i, /_API_KEY$/i,
];

function isSensitiveKey(key) {
  return SENSITIVE_KEY_PATTERNS.some((pattern) => pattern.test(key));
}

// Redact sensitive process.env entries inside a parsed report object, in place.
function redactReportObject(report) {
  if (report.environmentVariables && typeof report.environmentVariables === 'object') {
    for (const key of Object.keys(report.environmentVariables)) {
      if (isSensitiveKey(key)) {
        report.environmentVariables[key] = '[REDACTED]';
      }
    }
  }
  return report;
}

// Read a report from disk, redact it, and overwrite it with the scrubbed version.
// Call this immediately after process.report.writeReport() — before any upload,
// log line, or ticket attachment.
function redactReportFile(reportPath) {
  const report = JSON.parse(readFileSync(reportPath, 'utf8'));
  redactReportObject(report);
  writeFileSync(reportPath, JSON.stringify(report, null, 2));
  return reportPath;
}

export { redactReportFile, redactReportObject };

Kubernetes Integration: Report Before OOM Kill

When Kubernetes sends SIGTERM before OOM killing a pod, trigger a report in the SIGTERM handler — redact it before it's copied anywhere:

javascript

import { redactReportFile } from './redact-report.js';

process.on('SIGTERM', async () => {
  // Generate diagnostic report BEFORE shutdown
  const reportFilename = process.report.writeReport();

  // Scrub secrets from process.env BEFORE the report leaves this process
  redactReportFile(reportFilename);
  logger.info({ reportFilename }, 'Diagnostic report written and redacted before shutdown');

  // Copy the REDACTED report to persistent storage (pod storage is ephemeral)
  await uploadToS3(reportFilename, `reports/${process.pid}-${Date.now()}.json`);

  // Then graceful shutdown
  await gracefulShutdown('SIGTERM');
});

Production story: during a UPI settlement spike, indexer pods started restarting repeatedly under memory pressure — but the logs showed nothing. Kubernetes was sending SIGKILL directly, with no SIGTERM grace period configured, which meant the process was terminated before its logger could flush the last lines explaining what was going wrong. Every restart looked identical from the outside: pod dies, pod restarts, no error in the logs, repeat. The crash was completely unreproducible in staging because staging never carried the same memory pressure.

The fix was exactly the pattern above: process.report.reportOnFatalError = true plus a SIGTERM handler that writes and uploads a redacted report to S3 before the pod's ephemeral storage disappears. Once wired up, the very next crash produced a report showing hundreds of leaked Redis client handles in the libuv section — a subscription client was being re-created on every reconnect attempt without closing the previous one. What had been an unreproducible, silent, recurring crash became a five-minute diagnosis from a single artifact.

Reading a Diagnostic Report: The Key Sections

Diagnosing OOM from a Report

javascript

// From the report's javascriptHeap section:
{
  "heapSpaces": {
    "old_space": {
      "memorySize": 2147483648,   // 2GB allocated
      "usedMemory": 2145001472,   // 2GB used = 99.9% full → OOM
    },
    "new_space": {
      "usedMemory": 33501200,     // new space mostly full too
    }
  }
}

// Diagnosis: Old Space completely full. Likely cause: memory leak.
// Next step: compare with two heap snapshots (Module 12 runbook)

Diagnosing Resource Leaks from `libuv` Section

javascript

// From the report's libuv section:
[
  { "type": "tcp", "address": "10.0.0.45", "port": 5432, "fd": 22 },
  { "type": "tcp", "address": "10.0.0.45", "port": 5432, "fd": 23 },
  // ... 487 more entries for the same host:port combination ...
]

// 489 TCP connections to db.internal:5432
// Database pool configured for max: 50
// → Connection leak: connections opened but never returned to pool

Diagnosing Event Loop Stall

javascript

// From the report's header + uvthreadResourceUsage:
{
  "header": { "event": "Signal", "trigger": "SIGUSR2" },
  "uvthreadResourceUsage": {
    "userCpuSeconds": 847,
    "kernelCpuSeconds": 0.1  // very low kernel time
  }
}

// High user CPU, very low kernel CPU:
// Node.js is burning CPU in JavaScript (not I/O)
// Combined with ELU > 0.95: pure JavaScript computation blocking the loop
// Next step: clinic flame to identify the function

SRE Pipeline Integration

Automatic Report Upload on Crash

javascript

// Configure crash reporting pipeline
import { S3Client, PutObjectCommand } from '@aws-sdk/client-s3';
import { createReadStream } from 'fs';
import { redactReportFile } from './redact-report.js';

const s3 = new S3Client({ region: 'ap-south-1' });

async function uploadDiagnosticReport(filename) {
  // Redact secrets from process.env BEFORE the file is ever read for upload.
  // This must happen before createReadStream — the file on disk itself is
  // rewritten scrubbed, so nothing unredacted is ever sent to S3.
  redactReportFile(filename);

  const key = `diagnostic-reports/${process.env.SERVICE_NAME}/${Date.now()}-${filename}`;
  await s3.send(new PutObjectCommand({
    Bucket: 'ops-diagnostic-reports',
    Key: key,
    Body: createReadStream(filename),
    ContentType: 'application/json',
    Metadata: {
      'service': process.env.SERVICE_NAME,
      'pod': process.env.HOSTNAME,
      'node-version': process.version,
    },
  }));
  logger.info({ s3Key: key }, 'Redacted diagnostic report uploaded to S3');
  return key;
}

// Hook into process events
process.on('uncaughtException', async (err) => {
  logger.error({ err }, 'Uncaught exception — generating diagnostic report');
  const filename = process.report.writeReport();
  await uploadDiagnosticReport(filename).catch(() => {});
  // Don't prevent normal exit — just ensure the redacted report is uploaded first
});

Alert on Crash with Report Link

javascript

// Send a PagerDuty/Slack alert with the report location
async function alertWithReport(err, reportS3Key) {
  await fetch(process.env.PAGERDUTY_WEBHOOK, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
      routing_key: process.env.PAGERDUTY_KEY,
      event_action: 'trigger',
      payload: {
        summary: `Node.js crash: ${err.message}`,
        severity: 'critical',
        source: process.env.HOSTNAME,
        custom_details: {
          diagnostic_report: `s3://ops-diagnostic-reports/${reportS3Key}`,
          node_version: process.version,
          service: process.env.SERVICE_NAME,
          heap_used_mb: Math.round(process.memoryUsage().heapUsed / 1024 / 1024),
        }
      }
    })
  });
}

Programmatic Report Inspection

Every example so far has used process.report.writeReport(), which writes the report to disk as a file — useful when you need a persistent artifact to upload or attach to a ticket. But when all you want is to inspect or triage the current process's diagnostic state in-process, writing to disk and immediately reading it back is an unnecessary round trip. process.report.getReport() returns the same report as a plain JavaScript object, entirely in memory:

javascript

// Direct, in-memory alternative to writeReport() + readFileSync() + JSON.parse()
// — useful for a health-check endpoint or an in-process triage function that
// doesn't need a file on disk at all
function analyzeCurrentProcess() {
  const report = process.report.getReport();  // no disk I/O, no file to clean up

  return {
    crashReason: report.header.event,
    heapUsedPct: Math.round(
      report.javascriptHeap.usedMemory / report.javascriptHeap.totalMemory * 100
    ),
    openTcpConnections: report.libuv.filter(h => h.type === 'tcp').length,
  };
}

// e.g. exposed on an internal diagnostics endpoint for live triage,
// without ever touching the filesystem
app.get('/internal/diagnostics', (req, res) => {
  res.json(analyzeCurrentProcess());
});

Use getReport() for live, in-process inspection (health endpoints, ad-hoc debugging in a REPL, in-process alerting logic). Use writeReport() when you need a durable file — for post-mortem analysis after the process has already died, writeReport() (typically paired with reportOnFatalError/reportOnSignal) is the only option, since there's no "current process" left to call getReport() on. The CI/CD analysis below reads a report that was already written to disk by a previous, separate process run — that's the case where re-parsing a file is unavoidable, not a missed opportunity to use getReport().

javascript

// Read and parse a report for automated analysis
import { readFileSync } from 'fs';

function analyzeReport(reportPath) {
  const report = JSON.parse(readFileSync(reportPath, 'utf8'));

  const analysis = {
    crashReason: report.header.event,
    heapUsedPct: Math.round(
      report.javascriptHeap.usedMemory / report.javascriptHeap.totalMemory * 100
    ),
    openTcpConnections: report.libuv.filter(h => h.type === 'tcp').length,
    openFileHandles: report.libuv.filter(h => h.type === 'fs_event' || h.type === 'pipe').length,
    cpuTimeSeconds: report.uvthreadResourceUsage.userCpuSeconds,
  };

  // Automated triage
  if (analysis.heapUsedPct > 95) analysis.likelyCause = 'OOM / memory leak';
  if (analysis.openTcpConnections > 200) analysis.likelyCause = 'Connection leak';
  if (analysis.cpuTimeSeconds > 3600) analysis.likelyCause = 'Long-running process with CPU issue';

  return analysis;
}

// Use in CI/CD to detect memory leaks during load tests
const analysis = analyzeReport('./diagnostic-reports/latest.json');
if (analysis.heapUsedPct > 80) {
  throw new Error(`Memory leak detected: heap at ${analysis.heapUsedPct}%`);
}

Summary

Concept	Key Takeaway
`process.report`	Structured JSON at the moment of failure. V8 heap, libuv handles, call stacks, raw (unredacted) env vars.
`reportOnFatalError`	Captures report on OOM, segfault, other fatal errors. No code change needed after configuration.
`reportOnSignal`	SIGUSR2 triggers a report in a running process. Zero impact on traffic.
`writeReport()`	Programmatic report generation in signal handlers and error handlers.
Heap section	`usedMemory / totalMemory` > 95% = OOM. Old Space full = memory leak.
libuv section	400 TCP connections to one host = connection leak. Unexpected handles = resource leak.
CPU section	High user CPU, low kernel CPU = JavaScript blocking the event loop.
Redact before persisting	`environmentVariables` is raw `process.env` — never pre-redacted. Run `redactReportFile()` before writing to disk, uploading, or linking in alerts.
S3 upload before exit	Persist redacted reports from ephemeral Kubernetes pods before they disappear.
Automated analysis	Parse reports in CI/CD load tests to detect memory leaks before production.

Next: HTTP/2, gRPC Transport, and Protocol Selection →

Knowledge Check

How can you configure Node.js to automatically capture the V8 heap state and open file descriptors exactly when the process crashes from an Out of Memory (OOM) error?

What does the presence of hundreds of identical outgoing TCP connection entries (e.g., { "type": "tcp", "address": "db.internal", "port": 5432 }) in the libuv section of a process.report most likely indicate?

What is the recommended way to reliably capture and preserve a process.report when Kubernetes shuts down a pod via a SIGTERM signal?

Test your knowledge with more question sets

Sign in to keep reading

The rest of this module is free — sign in with Google to unlock it and track your progress.

PreviousModule A-20: The Modern Web Standard Shift: Undici, fetch, and Web Crypto Next Module A-22: HTTP/2, gRPC Transport, and Protocol Selection

Discussion

Join the discussion

Loading comments...

Module 20 — Automated Post-Mortem Diagnostics: process.report

What a Diagnostic Report Contains

Configuration: Triggering Reports Automatically

CLI-Flag Equivalents

Sign in to keep reading

Discussion

Module 20 — Automated Post-Mortem Diagnostics: `process.report`