A-15 — Production Observability and Runbooks

Who this is for: Architects responsible for a Next.js application in production — the ones who get paged at 3am. This module is about building the observability infrastructure that turns "the site is down" into "the database connection pool exhausted at 03:14 UTC, triggered by a deployment that removed the connection limit from the Prisma config, here's the fix." That level of precision comes from traces, metrics, logs, and runbooks built before the incident, not during it.

The Three Pillars of Observability

Observability is the ability to understand a system's internal state from its external outputs. The three pillars:

Traces — the journey of a single request through your system. A trace for a product page request shows: Middleware execution (2ms), Server Component render (8ms), database query for product (45ms), database query for reviews (120ms), response sent. Traces answer "why was this specific request slow?"

Metrics — aggregated measurements over time. Request rate, error rate, p50/p95/p99 response times, cache hit rate, database connection pool size. Metrics answer "is the system healthy overall, and are things getting worse?"

Logs — discrete events. "User 123 purchased product 456." "Cache miss for key products:page:1." "Database query failed: connection timeout." Logs answer "what happened?"

None of these is sufficient alone. A slow request shows up in metrics (rising p99), is diagnosed via traces (database query taking 2s), and confirmed by logs (connection pool exhausted). The triad is the diagnostic workflow.

OpenTelemetry in Next.js

OpenTelemetry (OTel) is the industry standard for trace and metric instrumentation. Next.js 13+ has built-in OTel support.

getNodeAutoInstrumentations automatically instruments:

• HTTP requests (both incoming and outgoing)
• Prisma queries (via @prisma/instrumentation)
• DNS lookups
• setTimeout / setInterval for tracking async work

The result: every request automatically generates a trace showing all the work it triggered. No manual span creation required for the common cases.

Custom Spans for Business Logic

The auto-instrumentation covers infrastructure — database, HTTP. For business logic, add custom spans:

Custom spans appear nested inside the auto-instrumented HTTP span in your trace UI. You can see exactly where within a request the business logic executed, how long it took, and whether it threw.

Sentry for Error Tracking

OpenTelemetry traces tell you about slow requests. Sentry tells you about broken requests — the uncaught exceptions, the unhandled rejections, the React hydration mismatches.

The wizard configures Sentry automatically. What it sets up:

• instrumentation.ts with Sentry SDK initialisation
• Error boundary integration for React
• Next.js specific configuration in next.config.ts
• Source map upload for production

The global-error.tsx catches errors that bubble past all error.tsx boundaries — the last resort error handler. Without it, uncaught root-level errors show a blank page.

Metrics and Alerting

The metrics that matter for a Next.js application:

The alerting philosophy: alert on symptoms, not causes. "Error rate > 1%" is a symptom — it tells you users are experiencing failures. "Database CPU > 80%" is a cause — useful for investigation but not an emergency on its own. Symptom-based alerting reduces alert fatigue.

A minimal alert set for most applications:

1. Error rate (5xx) > 1% for 5 minutes → P1 incident
2. p99 response time > 5s for 10 minutes → P2 incident
3. Health check endpoint returning non-200 → P1 incident
4. Error rate > 0.1% for 30 minutes → P3 (monitor, not wake someone up)

The Runbook Template

A runbook is a document that answers: "what do I do when alert X fires?" Writing runbooks before incidents means the on-call engineer isn't making decisions under pressure for the first time.

The format matters less than the content. What every runbook needs: the alert trigger, immediate triage steps, common causes with specific fixes, and escalation paths.

The "High Error Rate" runbook above is the generic shape. In practice, your on-call rotation needs runbooks for the specific failure modes that actually page people — the ones that don't reduce cleanly to "check Sentry, check deploys, check the connection pool." Here are four that come up constantly in Next.js production operation.