The standard is W3C trace context: a traceparent header carrying trace ID, span ID and flags, generated at the edge if absent and propagated everywhere.

It must survive asynchronous boundaries. This is where nearly every system's tracing breaks:

• Queue and topic messages must carry the trace context in the message envelope, not just the payload, and the consumer must restore it (Modules P-14, P-15). Otherwise the trace ends at the producer and the work done by the consumer is an orphan.
• Background jobs must store the trace context on the job row and restore it when claimed (Module P-19), so "why did this user's export fail" links back to the request that requested it.
• Outbox events and CDC should carry it too (Module P-23), so a downstream effect is attributable to its cause.
• Webhook deliveries should include it, so a partner's report of a problem can be traced (Module P-17).
• In-process async context needs a mechanism — AsyncLocalStorage in Node — or the ID is lost the first time you await inside a callback chain.

And a trick worth knowing: attach the trace ID to your database queries as an SQL comment. Postgres records it in pg_stat_activity and slow-query logs, so a slow query found in the database can be traced back to the request that issued it, which is otherwise a genuinely hard correlation:

---

Logs: Structured, Scrubbed, Sampled, Budgeted

Structured, not printf. JSON with consistent field names, so logs are queryable rather than greppable. log.info({ orderId, tenantId, durationMs }, "order created") is a data record; log.info("Created order " + id + " in " + ms + "ms") is a sentence you will regret writing a parser for.

Levels used meaningfully. ERROR means someone should look; WARN means it is notable and handled. A codebase where ERROR fires for expected validation failures has trained everyone to ignore errors, which is the most expensive possible outcome of a logging decision.

Sample deliberately. Keep every error, sample successful requests (1–10% is common at volume), and keep all logs for a sampled trace so a retained trace has its detail intact. Log volume grows with traffic and with every developer who adds a line, which makes it one of the largest observability line items — and it is a cost that appears without anyone deciding (Module O-8).

Never log secrets or personal data, and treat that as a compliance control rather than hygiene. Logs get shipped to third-party providers, retained for months, and replicated across regions — so a request body in a log is a PII store with residency and retention obligations (Modules A-9, O-9). Use an allowlist of fields to log rather than a denylist of fields to redact, because a denylist fails silently the first time someone adds a field. Scrub at the collector as a second layer.

---

Traces: Where the Time Went, and How to Sample Them

A trace is a tree of spans, and its value is the waterfall:

Nothing else in your toolkit makes that legible. It is also how you find the serial chain that should be parallel and the fan-out amplification of Module A-4.

Sampling is the cost control, and the choice matters:

• Head-based: decide at the edge (keep 1%). Cheap, simple, and it discards most errors and slow requests — precisely the traces you wanted.
• Tail-based: buffer a whole trace in a collector, then decide once its outcome is known. Keep 100% of errors, 100% of traces above a latency threshold, and 1% of the rest. This is what you actually want, and it costs a collector tier with memory proportional to trace duration times throughput.

Instrument boundaries automatically (OpenTelemetry auto-instrumentation covers HTTP, database drivers and queue clients) and add manual spans for the business-meaningful steps that framework instrumentation cannot know about — "reserve inventory," "run ranking model." Put high-cardinality identifiers on span attributes, where they are free, rather than on metric labels, where they are ruinous.

---

The Golden Signals, and the Synthesis: Every Mechanism Has One Metric

Start with the four golden signals per service and per dependency: latency (as a distribution), traffic, errors, and saturation — with saturation the leading indicator, since it moves before the other three (Module A-8).

Then the payoff. Every resilience and correctness mechanism built in the previous three phases is silent by default, and each has exactly one metric that proves it is working. This table is the observability plan for everything in this course:

The pattern across every row: the mechanisms that protect you do so by converting a loud failure into a quiet one. That is their purpose, and it is exactly why each needs a metric. A system with good resilience and no observability is a system that degrades permanently without telling anyone.

---

Controlling the Cost

Observability commonly runs 10–30% of infrastructure spend, and occasionally exceeds the cost of the system it observes. The levers, in order of effect:

• Cardinality limits and enforcement at the collector, so one bad label cannot take out the backend or the budget.
• Tail-based sampling for traces, and log sampling of successes.
• Retention tiers: hot for 7 days, warm for 30, then aggregated rollups or raw data in object storage (Modules P-22, O-8). Almost nobody queries raw traces from four months ago; many people pay to keep them.
• Delete unused metrics. In most systems a large majority of time series are never queried by any dashboard or alert. Auditing that is a quick, sizeable saving.
• Aggregate at the edge before shipping, where the raw detail has no consumer.

Why this matters in production: the failures observability prevents are the ones already listed in the table — a breaker open for three weeks, a dead-letter queue nobody drained, a CDC slot filling a disk, a cache silently refusing writes, a payment reconciliation nobody read. But observability also causes incidents, in two recognisable ways: a metrics backend killed by a user_id label added in a routine PR, and a log pipeline that costs more than the application and gets cut in an emergency, removing your ability to debug at exactly the wrong moment. Both are cardinality and volume problems, and both are cheaper to prevent with a limit at the collector than to fix under pressure.

---

Where This Shows Up in Your Stack

• OpenTelemetry: the vendor-neutral instrumentation layer, and the reason to adopt it is portability — instrument once, change backend without re-instrumenting. The collector is where sampling, scrubbing, cardinality limiting and routing belong, because doing it in application code means a deploy per change.
• Prometheus: pull-based, and its memory is a function of active series, so cardinality is a hard operational constraint rather than a cost curve. Use recording rules to precompute expensive queries, histogram_quantile over summed buckets (never averaged percentiles), and relabelling to drop labels you do not query.
• Node.js: the runtime metrics that actually matter are event-loop delay (the saturation signal, Module A-8), heap usage and GC pause time, and active handles. Use AsyncLocalStorage to carry trace context across await boundaries, or correlation IDs silently disappear inside async call chains.
• PostgreSQL: pg_stat_statements gives normalised query fingerprints with total and mean time — the single most useful database view there is. Add the trace ID as an SQL comment so a slow query in pg_stat_activity can be attributed to a request. Watch pool wait time from the application side, replica lag, and retained WAL per replication slot (Module P-23).
• Redis: INFO gives evicted_keys, keyspace_misses, mem_fragmentation_ratio and used_memory versus maxmemory — the four numbers that explain almost every Redis surprise (Module P-12). SLOWLOG catches the KEYS someone ran in production.
• Kafka: consumer group lag per partition is both the health metric and the backpressure signal (Modules P-15, A-8), and per-partition rather than aggregate, because one lagging partition is invisible in a total.

---

Summary

• The test is whether you can answer an unanticipated question without deploying code. Pre-aggregated data cannot be sliced by a dimension you discarded at write time, so the shape of what you collect matters as much as the volume.
• Three signals with three different limits: metrics cannot explain a single request, logs cannot aggregate cheaply, and traces are sampled so the request a customer is complaining about may not exist. Wide structured events increasingly replace the split, because metrics can be derived from events and events cannot be derived from metrics — at the cost of storage and a backend that aggregates high-cardinality fields at query time. Metrics remain the right substrate for alerting.
• Cardinality is the budget: cost is unique label combinations, not data points. Adding user_id to an 18,000-series metric is a 36-billion-series outage, not a gradual increase. Metrics take bounded dimensions only; high-cardinality dimensions belong in logs, traces and events — which makes path normalisation mandatory.
• Histograms multiply cardinality by their bucket count, and buckets must be chosen in advance — you cannot learn about a 4-second p99.9 from buckets that stop at 1 second. Choose boundaries straddling your SLO thresholds.
• You cannot average percentiles. Merge histogram buckets across instances and compute the quantile once. Also measure client-perceived latency rather than only handler time, and look at the distribution, since a bimodal latency has a p99 that describes neither mode.
• A correlation ID on every log line, span and downstream call is the highest-value cheap thing here — and it must survive asynchronous boundaries: queue message envelopes, background job rows, outbox events, webhook deliveries, and in-process await chains via AsyncLocalStorage. This is the gap most systems have, and it is why traces so often end at the producer. Attaching the trace ID as an SQL comment lets a slow query be traced back to its request.
• Logs: structured JSON with consistent fields, levels that mean something (an ERROR for expected validation failures trains everyone to ignore errors), keep all errors and sample successes, and allowlist the fields you log rather than denylisting the ones you redact — because logs are shipped to third parties, retained for months and replicated across regions, which makes them a PII store with residency obligations.
• Traces make the N+1 and the accidental serial chain visible instantly, and tail-based sampling is what you want — buffer the trace, then keep 100% of errors and slow traces and a small fraction of the rest — at the cost of a collector tier. Put high-cardinality identifiers on span attributes, where they are free.
• Start with the four golden signals per service and per dependency, with saturation as the leading indicator.
• Every resilience mechanism from the previous three phases is silent by default, and each has exactly one metric that proves it works: breaker state and time-open, degraded-serve rate, retry-budget exhaustion, shed rate and queue wait time, oldest-unstarted-job age per queue, pool wait time, replica and CDC lag, retained WAL per slot, cache evictions, search recall and zero-result rate, unreconciled payment breaks and a ledger that sums to zero, per-endpoint webhook age, fan-out latency, LLM groundedness, ntp_offset, per-principal limiter rejections — and a region dimension on all of it.
• The pattern is that protective mechanisms work by converting a loud failure into a quiet one, which is exactly why each needs a metric. Good resilience without observability is a system that degrades permanently without telling anyone.
• Control the cost with cardinality limits at the collector, tail-based and log sampling, retention tiers, and deleting the majority of series nothing queries. Observability can cause its own incidents — a metrics backend killed by a label added in a routine PR, or a log bill cut in an emergency, removing your ability to debug precisely when you need it.

The next module, SLIs, SLOs and Error Budgets, turns this data into decisions. Having metrics is not the same as knowing whether you are meeting a promise, and an alert on every anomaly is how a team learns to ignore its pager. O-2 covers choosing indicators users can actually feel, setting targets you can defend, spending an error budget deliberately, and alerting on symptoms with burn rates rather than on every cause.

---