Module 20 — Automated Post-Mortem Diagnostics: process.report

For a Dockerized indexer, baking these flags into the container's ENTRYPOINT/CMD (or a NODE_OPTIONS env var) guarantees report generation is armed regardless of how the application code initializes — including if the crash happens before your own process.report.reportOnFatalError = true line would have executed.

Redacting Secrets Before a Report Ever Leaves the Box

Because environmentVariables is raw process.env, every report must be scrubbed before it is written anywhere durable or uploaded anywhere. Never skip this step — an unredacted report is a plaintext credential dump.

// redact-report.js
import { readFileSync, writeFileSync } from 'node:fs';
 
// Keys matched case-insensitively. Extend this list for your own secret naming conventions.
const SENSITIVE_KEY_PATTERNS = [
  /_SECRET$/i, /_KEY$/i, /_TOKEN$/i, /_PASSWORD$/i, /^PASSWORD$/i,
  /_URL$/i,        // catches DATABASE_URL, REDIS_URL, etc. (may embed creds)
  /_DSN$/i, /_CREDENTIALS?$/i, /^AWS_/i, /_API_KEY$/i,
];
 
function isSensitiveKey(key) {
  return SENSITIVE_KEY_PATTERNS.some((pattern) => pattern.test(key));
}
 
// Redact sensitive process.env entries inside a parsed report object, in place.
function redactReportObject(report) {
  if (report.environmentVariables && typeof report.environmentVariables === 'object') {
    for (const key of Object.keys(report.environmentVariables)) {
      if (isSensitiveKey(key)) {
        report.environmentVariables[key] = '[REDACTED]';
      }
    }
  }
  return report;
}
 
// Read a report from disk, redact it, and overwrite it with the scrubbed version.
// Call this immediately after process.report.writeReport() — before any upload,
// log line, or ticket attachment.
function redactReportFile(reportPath) {
  const report = JSON.parse(readFileSync(reportPath, 'utf8'));
  redactReportObject(report);
  writeFileSync(reportPath, JSON.stringify(report, null, 2));
  return reportPath;
}
 
export { redactReportFile, redactReportObject };

Kubernetes Integration: Report Before OOM Kill

When Kubernetes sends SIGTERM before OOM killing a pod, trigger a report in the SIGTERM handler — redact it before it's copied anywhere:

import { redactReportFile } from './redact-report.js';
 
process.on('SIGTERM', async () => {
  // Generate diagnostic report BEFORE shutdown
  const reportFilename = process.report.writeReport();
 
  // Scrub secrets from process.env BEFORE the report leaves this process
  redactReportFile(reportFilename);
  logger.info({ reportFilename }, 'Diagnostic report written and redacted before shutdown');
 
  // Copy the REDACTED report to persistent storage (pod storage is ephemeral)
  await uploadToS3(reportFilename, `reports/${process.pid}-${Date.now()}.json`);
 
  // Then graceful shutdown
  await gracefulShutdown('SIGTERM');
});

Production story: during a UPI settlement spike, indexer pods started restarting repeatedly under memory pressure — but the logs showed nothing. Kubernetes was sending SIGKILL directly, with no SIGTERM grace period configured, which meant the process was terminated before its logger could flush the last lines explaining what was going wrong. Every restart looked identical from the outside: pod dies, pod restarts, no error in the logs, repeat. The crash was completely unreproducible in staging because staging never carried the same memory pressure.

The fix was exactly the pattern above: process.report.reportOnFatalError = true plus a SIGTERM handler that writes and uploads a redacted report to S3 before the pod's ephemeral storage disappears. Once wired up, the very next crash produced a report showing hundreds of leaked Redis client handles in the libuv section — a subscription client was being re-created on every reconnect attempt without closing the previous one. What had been an unreproducible, silent, recurring crash became a five-minute diagnosis from a single artifact.

---

Reading a Diagnostic Report: The Key Sections

Diagnosing OOM from a Report

// From the report's javascriptHeap section:
{
  "heapSpaces": {
    "old_space": {
      "memorySize": 2147483648,   // 2GB allocated
      "usedMemory": 2145001472,   // 2GB used = 99.9% full → OOM
    },
    "new_space": {
      "usedMemory": 33501200,     // new space mostly full too
    }
  }
}
 
// Diagnosis: Old Space completely full. Likely cause: memory leak.
// Next step: compare with two heap snapshots (Module 12 runbook)

Diagnosing Resource Leaks from libuv Section

// From the report's libuv section:
[
  { "type": "tcp", "address": "10.0.0.45", "port": 5432, "fd": 22 },
  { "type": "tcp", "address": "10.0.0.45", "port": 5432, "fd": 23 },
  // ... 487 more entries for the same host:port combination ...
]
 
// 489 TCP connections to db.internal:5432
// Database pool configured for max: 50
// → Connection leak: connections opened but never returned to pool

Diagnosing Event Loop Stall

// From the report's header + uvthreadResourceUsage:
{
  "header": { "event": "Signal", "trigger": "SIGUSR2" },
  "uvthreadResourceUsage": {
    "userCpuSeconds": 847,
    "kernelCpuSeconds": 0.1  // very low kernel time
  }
}
 
// High user CPU, very low kernel CPU:
// Node.js is burning CPU in JavaScript (not I/O)
// Combined with ELU > 0.95: pure JavaScript computation blocking the loop
// Next step: clinic flame to identify the function

---

SRE Pipeline Integration

Automatic Report Upload on Crash

// Configure crash reporting pipeline
import { S3Client, PutObjectCommand } from '@aws-sdk/client-s3';
import { createReadStream } from 'fs';
import { redactReportFile } from './redact-report.js';
 
const s3 = new S3Client({ region: 'ap-south-1' });
 
async function uploadDiagnosticReport(filename) {
  // Redact secrets from process.env BEFORE the file is ever read for upload.
  // This must happen before createReadStream — the file on disk itself is
  // rewritten scrubbed, so nothing unredacted is ever sent to S3.
  redactReportFile(filename);
 
  const key = `diagnostic-reports/${process.env.SERVICE_NAME}/${Date.now()}-${filename}`;
  await s3.send(new PutObjectCommand({
    Bucket: 'ops-diagnostic-reports',
    Key: key,
    Body: createReadStream(filename),
    ContentType: 'application/json',
    Metadata: {
      'service': process.env.SERVICE_NAME,
      'pod': process.env.HOSTNAME,
      'node-version': process.version,
    },
  }));
  logger.info({ s3Key: key }, 'Redacted diagnostic report uploaded to S3');
  return key;
}
 
// Hook into process events
process.on('uncaughtException', async (err) => {
  logger.error({ err }, 'Uncaught exception — generating diagnostic report');
  const filename = process.report.writeReport();
  await uploadDiagnosticReport(filename).catch(() => {});
  // Don't prevent normal exit — just ensure the redacted report is uploaded first
});

Alert on Crash with Report Link

// Send a PagerDuty/Slack alert with the report location
async function alertWithReport(err, reportS3Key) {
  await fetch(process.env.PAGERDUTY_WEBHOOK, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
      routing_key: process.env.PAGERDUTY_KEY,
      event_action: 'trigger',
      payload: {
        summary: `Node.js crash: ${err.message}`,
        severity: 'critical',
        source: process.env.HOSTNAME,
        custom_details: {
          diagnostic_report: `s3://ops-diagnostic-reports/${reportS3Key}`,
          node_version: process.version,
          service: process.env.SERVICE_NAME,
          heap_used_mb: Math.round(process.memoryUsage().heapUsed / 1024 / 1024),
        }
      }
    })
  });
}

Programmatic Report Inspection

Every example so far has used process.report.writeReport(), which writes the report to disk as a file — useful when you need a persistent artifact to upload or attach to a ticket. But when all you want is to inspect or triage the current process's diagnostic state in-process, writing to disk and immediately reading it back is an unnecessary round trip. process.report.getReport() returns the same report as a plain JavaScript object, entirely in memory:

// Direct, in-memory alternative to writeReport() + readFileSync() + JSON.parse()
// — useful for a health-check endpoint or an in-process triage function that
// doesn't need a file on disk at all
function analyzeCurrentProcess() {
  const report = process.report.getReport();  // no disk I/O, no file to clean up
 
  return {
    crashReason: report.header.event,
    heapUsedPct: Math.round(
      report.javascriptHeap.usedMemory / report.javascriptHeap.totalMemory * 100
    ),
    openTcpConnections: report.libuv.filter(h => h.type === 'tcp').length,
  };
}
 
// e.g. exposed on an internal diagnostics endpoint for live triage,
// without ever touching the filesystem
app.get('/internal/diagnostics', (req, res) => {
  res.json(analyzeCurrentProcess());
});

Use getReport() for live, in-process inspection (health endpoints, ad-hoc debugging in a REPL, in-process alerting logic). Use writeReport() when you need a durable file — for post-mortem analysis after the process has already died, writeReport() (typically paired with reportOnFatalError/reportOnSignal) is the only option, since there's no "current process" left to call getReport() on. The CI/CD analysis below reads a report that was already written to disk by a previous, separate process run — that's the case where re-parsing a file is unavoidable, not a missed opportunity to use getReport().

// Read and parse a report for automated analysis
import { readFileSync } from 'fs';
 
function analyzeReport(reportPath) {
  const report = JSON.parse(readFileSync(reportPath, 'utf8'));
 
  const analysis = {
    crashReason: report.header.event,
    heapUsedPct: Math.round(
      report.javascriptHeap.usedMemory / report.javascriptHeap.totalMemory * 100
    ),
    openTcpConnections: report.libuv.filter(h => h.type === 'tcp').length,
    openFileHandles: report.libuv.filter(h => h.type === 'fs_event' || h.type === 'pipe').length,
    cpuTimeSeconds: report.uvthreadResourceUsage.userCpuSeconds,
  };
 
  // Automated triage
  if (analysis.heapUsedPct > 95) analysis.likelyCause = 'OOM / memory leak';
  if (analysis.openTcpConnections > 200) analysis.likelyCause = 'Connection leak';
  if (analysis.cpuTimeSeconds > 3600) analysis.likelyCause = 'Long-running process with CPU issue';
 
  return analysis;
}
 
// Use in CI/CD to detect memory leaks during load tests
const analysis = analyzeReport('./diagnostic-reports/latest.json');
if (analysis.heapUsedPct > 80) {
  throw new Error(`Memory leak detected: heap at ${analysis.heapUsedPct}%`);
}

---

Summary

Next: HTTP/2, gRPC Transport, and Protocol Selection →

---