---

Retry Amplification, and the Failure You Cannot Exit by Removing Its Cause

This is the most important section in the module.

Retries multiply through layers. Consider a request path where each layer retries three times:

Each layer is individually reasonable. Together they are an 81× amplifier, and it engages precisely when the deepest component is failing — because that is what triggers the retries. A database at a 50% error rate receiving three times its normal load will not recover; it will produce more errors, which produce more retries, which produce more errors.

That produces a property worth naming, because it is unintuitive and it is why some outages last hours:

A metastable failure state is one that sustains itself after its trigger is gone.

A 30-second network blip causes a wave of retries. The retries push load above capacity. Being above capacity produces timeouts, which produce more retries. The network is fine now — and the system does not recover, because the load keeping it down is the load generated by its own failure. Removing the original cause changes nothing. You exit only by reducing load: shedding traffic (Module A-8), opening circuit breakers (Module A-7), or, in the field, taking the service out of rotation until queues drain.

Two rules follow.

Retry at one layer, not at every layer. Pick the layer that knows whether the operation is safe to retry — usually the one closest to the business logic, which holds the idempotency key — and make every other layer pass failures through. If a mesh (Module A-5) is retrying for you, the application must not also retry, or you have silently rebuilt the multiplier.

Bound amplification with a retry budget, which is the mechanism most systems lack and the one that actually works:

A retry budget changes the shape of the failure completely. With a per-request retry count, a 100% error rate produces 3× (or 81×) load. With a 10% budget, a 100% error rate produces 1.1× load: the system fails fast, cheaply, and leaves the dependency enough headroom to recover. It converts retries from a mechanism that can destroy a dependency into one that cannot.

Why this matters in production: retry amplification is the most common way a small incident becomes a long one. The initial trigger is usually mundane — a failover, a deploy, a brief network partition — and the duration is set by whether the system can shed the load its own retries created. Teams add retries because they improve success rates in testing (where the dependency is healthy) and discover their real behaviour during an incident, when the amplification is indistinguishable from a traffic surge. A retry budget and a circuit breaker cost an afternoon each and are the difference between a five-minute blip and a two-hour incident.

---

Deadline Propagation: Do Not Do Work Nobody Is Waiting For

Timeouts are usually configured per hop, independently. That is wrong, and the reason is the zombie-work problem.

The edge gives a request a 3-second budget. Service A calls B calls C. Two seconds in, the edge has given up and the user has seen an error. But B and C have their own 5-second timeouts and know nothing about it, so they keep working — holding connections, running queries, consuming CPU — on a result no one will read. During an overload, a large fraction of a system's capacity goes into computing answers for callers that left, which is another route into the metastable state.

The fix is to propagate a deadline (an absolute time) rather than a timeout (a duration), and to have every hop shrink its own budget to what remains:

Three properties of a correct implementation:

• The deadline is absolute and passed unchanged, so every hop measures against the same instant. gRPC does this natively (grpc-timeout); over HTTP you need a header convention, and the emerging one worth adopting is Deadline/grpc-timeout-style propagation through your gateway.
• Every hop reserves a little for itself — its own processing and its response serialisation — so it does not hand a downstream a budget that leaves no time to return the answer.
• Deadline exceeded is a terminal outcome, never retried. There is no budget to retry inside, and retrying is the definition of doing work nobody wants.

And note the clock caveat from Module A-1: an absolute deadline compares wall clocks across machines, so it is only as good as your NTP discipline. In practice a few milliseconds of skew is irrelevant against a 3-second budget, but a badly skewed host will reject everything as already-expired — which is one of the many silent symptoms clock drift produces.

Cancellation has to actually cancel

A timeout on the client does not stop the server. This matters most at the database, and it is the specific detail that surprises people:

A Node query timeout abandons the response. The Postgres backend keeps executing the query, holding the connection, the snapshot and its locks.

So a client-side timeout on a slow query gives you the worst of both: the client has moved on (and may retry, doubling the work), while the server does the original query to completion. The enforcement has to be server-side:

Likewise, an AbortController only helps if the code path checks the signal — a long synchronous loop ignores it entirely — and an HTTP server does not stop a handler because the client disconnected unless the handler looks.

---

Hedged Requests: Buying Tail Latency With Load

The tail-latency technique worth knowing, because it addresses something retries cannot. If a dependency's p50 is 10 ms and its p99 is 400 ms, that tail is usually one unlucky server — a GC pause, a cold cache, a noisy neighbour — not a systemic slowness. So:

Send the request. If no response arrives by the p95 (say 40 ms), send a second request to a different instance and take whichever answers first.

Because only 5% of requests trigger a hedge, the extra load is about 5%, and the p99 collapses towards the p50. It is a deliberate purchase of tail latency with a small, bounded amount of load.

The constraints are strict:

• Idempotent operations only — you are creating a duplicate on purpose.
• Bound the hedge rate with a budget, exactly like retries, or a systemic slowdown makes every request hedge and you have added 100% load during an incident.
• Cancel the loser where the protocol allows, or you pay for both.
• Route the hedge elsewhere — a second request to the same slow instance is pointless.

Do not confuse hedging with retrying. A retry follows a failure; a hedge follows slowness, with the first attempt still in flight. Retries improve success rate; hedges improve tail latency.

---

Where This Shows Up in Your Stack

• PostgreSQL: the client timeout is not the bound — statement_timeout is, and without it a cancelled client leaves a query running to completion while holding a connection (Module P-4). Set statement_timeout per role and per workload (short for the web role, generous for the reporting role, Module P-22), lock_timeout so a DDL statement cannot queue and block every reader behind it (Module O-4), and idle_in_transaction_session_timeout to kill transactions abandoned across a network call. Retry only the errors that are retryable: 40001 serialisation failure and 40P01 deadlock are designed to be retried (Module P-1); a constraint violation is not.
• Node.js: there is no default overall timeout on fetch, so add AbortSignal.timeout() on every outbound call — this is the highest-value line of resilience code in most codebases. Know that undici distinguishes headersTimeout from bodyTimeout, and neither is a total deadline. AbortController is cooperative: a synchronous loop or an unresponsive library ignores it. And a long synchronous handler blocks the event loop, so your own timeouts will not fire on schedule either (Module P-19).
• Redis: operations are sub-millisecond, so timeouts should be tight (tens of milliseconds) — a Redis call taking 2 seconds means something is wrong and waiting will not help. But because Redis is on the path of nearly every request, a slow Redis is an every-endpoint problem, so a tight timeout plus a circuit breaker plus a defined degraded behaviour (serve without the cache) is the pattern (Module A-7).
• Kafka: delivery.timeout.ms is the total bound on a produce (including retries), request.timeout.ms bounds one attempt, and the interaction to know is that retries with max.in.flight.requests.per.connection > 1 can reorder messages unless enable.idempotence=true — so the safe configuration is idempotent producer plus generous delivery timeout rather than a hand-tuned retry count (Module P-15).
• Service mesh: a mesh applies retries and timeouts at the proxy (Module A-5), which means it can double whatever the application does. Decide which layer owns retry policy, configure the other to pass through, and remember the proxy can only safely retry what you have told it is idempotent.

---

Summary

• No timeout is a decision to fail together. By Little's Law, a dependency's service time rising 600× raises your concurrency 600× at unchanged traffic, exhausting pools and request slots and degrading endpoints that never touched that dependency. A timeout converts someone else's latency into your bounded, attributable error rate.
• Four timeouts exist and only the total one bounds you. Connect, time-to-first-byte and idle are useful diagnostics; a response trickling a byte every four seconds defeats all three. Always set the overall deadline.
• Derive the timeout from the p99.9 of the successful distribution, not the average. Too short converts would-have-succeeded requests into retry load — a load amplifier disguised as safety. Too long holds resources through a detectable failure.
• A retry policy must fit inside the overall budget: per-attempt timeout is remaining / max_attempts. A 3-second attempt retried three times inside a caller's 5-second budget means attempts two and three are work for nobody.
• Retry only what is safe. Connection refused and 503/429 are safe; timeouts and 500s are only safe when the operation is idempotent, because both are the ambiguous failure; 4xx content rejections never are. HTTP method semantics are a hint, not a guarantee — decide from what the operation does and whether it carries an idempotency key.
• Exponential backoff fixes frequency; jitter fixes synchronisation. Without jitter, a recovering dependency is knocked over by the first aligned burst. Use full jitter, cap the exponential, bound the attempts, and make the first retry near-immediate for a connection refusal since that costs the dependency nothing.
• Retry amplification multiplies through layers: three retries at four layers is 81 attempts per user request, engaged exactly when the deepest component is failing. Retry at one layer only — the one that knows the operation is idempotent — and if a mesh retries for you, the application must not.
• A metastable failure state sustains itself after its trigger is gone. Retries push load above capacity, being above capacity produces timeouts, timeouts produce retries. Removing the original cause changes nothing; you exit only by reducing load — shedding (Module A-8), opening breakers (Module A-7), or removing the service from rotation until queues drain.
• A retry budget is the mechanism that bounds amplification and the one most systems lack: cap retries at ~10% of successful volume, or use adaptive client-side throttling driven by the accept rate. With per-request retry counts, a 100% error rate produces 3× or 81× load; with a 10% budget it produces 1.1×, which leaves the dependency enough headroom to recover.
• Propagate an absolute deadline, not a per-hop timeout. Otherwise an overloaded system spends its capacity computing answers for callers that have already given up. Pass the deadline unchanged, shrink each hop's budget to what remains minus a reserve for its own work, and treat deadline-exceeded as terminal — never retried, since there is no budget to retry inside. Absolute deadlines depend on clock discipline (Module A-1).
• Cancellation must be enforced where the work happens. A client-side query timeout does not stop the Postgres backend, which keeps executing while holding a connection, a snapshot and locks — statement_timeout is the real bound. AbortController is cooperative and a synchronous loop ignores it.
• Hedged requests buy tail latency with load: send a second request to a different instance after the p95 and take the first answer, which affects ~5% of traffic and collapses p99 towards p50. Idempotent operations only, with a hedge budget so a systemic slowdown does not double all traffic, and cancel the loser. A retry follows a failure; a hedge follows slowness with the first attempt still in flight.

The next module, Circuit Breakers, Bulkheads, and Graceful Degradation, is the other half of this policy. Timeouts and budgets bound the cost of each call to a failing dependency; a circuit breaker stops making the calls at all once it is clear they will fail, a bulkhead stops one dependency from consuming the resources every other request needs, and degradation decides which features to switch off so the ones that matter keep working.

---