A-15 — Production Observability and Runbooks

Who this is for: Architects responsible for a Next.js application in production — the ones who get paged at 3am. This module is about building the observability infrastructure that turns "the site is down" into "the database connection pool exhausted at 03:14 UTC, triggered by a deployment that removed the connection limit from the Prisma config, here's the fix." That level of precision comes from traces, metrics, logs, and runbooks built before the incident, not during it.

The Three Pillars of Observability

Observability is the ability to understand a system's internal state from its external outputs. The three pillars:

Traces — the journey of a single request through your system. A trace for a product page request shows: Middleware execution (2ms), Server Component render (8ms), database query for product (45ms), database query for reviews (120ms), response sent. Traces answer "why was this specific request slow?"

Metrics — aggregated measurements over time. Request rate, error rate, p50/p95/p99 response times, cache hit rate, database connection pool size. Metrics answer "is the system healthy overall, and are things getting worse?"

Logs — discrete events. "User 123 purchased product 456." "Cache miss for key products:page:1." "Database query failed: connection timeout." Logs answer "what happened?"

None of these is sufficient alone. A slow request shows up in metrics (rising p99), is diagnosed via traces (database query taking 2s), and confirmed by logs (connection pool exhausted). The triad is the diagnostic workflow.

OpenTelemetry in Next.js

OpenTelemetry (OTel) is the industry standard for trace and metric instrumentation. Next.js 13+ has built-in OTel support.

getNodeAutoInstrumentations automatically instruments:

• HTTP requests (both incoming and outgoing)
• Prisma queries (via @prisma/instrumentation)
• DNS lookups
• setTimeout / setInterval for tracking async work

The result: every request automatically generates a trace showing all the work it triggered. No manual span creation required for the common cases.

Custom Spans for Business Logic

The auto-instrumentation covers infrastructure — database, HTTP. For business logic, add custom spans:

Custom spans appear nested inside the auto-instrumented HTTP span in your trace UI. You can see exactly where within a request the business logic executed, how long it took, and whether it threw.

Sentry for Error Tracking

OpenTelemetry traces tell you about slow requests. Sentry tells you about broken requests — the uncaught exceptions, the unhandled rejections, the React hydration mismatches.

The wizard configures Sentry automatically. What it sets up:

• instrumentation.ts with Sentry SDK initialisation
• Error boundary integration for React
• Next.js specific configuration in next.config.ts
• Source map upload for production

The global-error.tsx catches errors that bubble past all error.tsx boundaries — the last resort error handler. Without it, uncaught root-level errors show a blank page.

Metrics and Alerting

The metrics that matter for a Next.js application:

The alerting philosophy: alert on symptoms, not causes. "Error rate > 1%" is a symptom — it tells you users are experiencing failures. "Database CPU > 80%" is a cause — useful for investigation but not an emergency on its own. Symptom-based alerting reduces alert fatigue.

A minimal alert set for most applications:

1. Error rate (5xx) > 1% for 5 minutes → P1 incident
2. p99 response time > 5s for 10 minutes → P2 incident
3. Health check endpoint returning non-200 → P1 incident
4. Error rate > 0.1% for 30 minutes → P3 (monitor, not wake someone up)

The Runbook Template

A runbook is a document that answers: "what do I do when alert X fires?" Writing runbooks before incidents means the on-call engineer isn't making decisions under pressure for the first time.

The format matters less than the content. What every runbook needs: the alert trigger, immediate triage steps, common causes with specific fixes, and escalation paths.

Structured Logging

Structured logs (JSON format) are parseable by log aggregation services (Datadog, Grafana Loki, CloudWatch Logs Insights). They're queryable: "show me all logs where userId is 123 and level is error."

The traceId in every log entry is the correlation key — in your observability platform, you can click from a log entry to the trace that generated it. This is the debugging superpower: see the error in Sentry, look up the trace in your tracing service, correlate the logs via traceId, understand exactly what happened.

Congratulations — You've Completed the Course

You've made it through all three phases:

Foundation — the mental model. Routing, rendering, data fetching, the core components. The vocabulary for everything that followed.

Practitioner — the production toolkit. Authentication, database integration, caching, middleware, SEO, testing, deployment. Everything you need to ship a real application.

Architect — the internals. React Flight, request lifecycle, caching mechanics, PPR, streaming SSR, Server Action security, advanced routing, state management, edge compute, build systems, performance engineering, security, infrastructure, observability. The layer that separates engineers who understand Next.js from engineers who just use it.

The last piece of advice: read the Next.js changelog with each release. The framework moves fast. The architectural understanding from this course is what lets you evaluate each new feature and know whether it changes anything for your specific application. The mechanics change; the mental model for reasoning about them doesn't.