Module F-4·20 min read

p50 vs p99 vs p999, why averages hide outages, and Little's Law — the equation that tells you how many servers you actually need.

JJS

Written by Jatin Jain Saraf · Senior Software Engineer

Module F-4 — Latency, Throughput, and Tail Latency

Q: A page assembles its content from 100 backend calls, each with an independent 1% chance of falling into its own p99 tail. Leadership argues the p99 is "only 1% of requests" and not worth engineering effort. What does the module say?

At 100 calls the probability that at least one is slow is about 63%, so the p99 path is the majority experience for page loads — which is why tail latency is an architectural concern rather than a tuning detail. — This is the fan-out amplification table: 1 − 0.99¹⁰⁰ ≈ 63%. Because a page needs *all* its calls to be fast, per-call rarity does not survive fan-out — the module cites this as the core argument of "The Tail at Scale" and the reason Modules A-6 through A-8 exist. Option B commits the exact error the section is written against: confusing a share of requests with a share of user experiences. Option C inverts how the probabilities compose — requiring every call to succeed quickly makes the outcome worse with more calls, not smoother. Option D changes what gets recorded rather than what users experience; a longer timeout makes the slow request slower still, and hides it from monitoring at the same time.

Q: A service handles 1,000 req/s at 200ms average latency, using a connection pool capped at 250. A dependency slows down and latency rises to 400ms, with traffic unchanged. What does Little's Law predict, and why does it matter?

Concurrency doubles from 200 to 400 in-flight requests, exceeding the 250-connection pool — so the system begins failing because latency rose, not because traffic did, which is the mechanism behind most cascading failures. — L = λ × W gives 1,000 × 0.4 = 400 concurrent requests, up from 200, with no change in arrival rate. The module highlights this direction of the law specifically because it's the one people don't use: concurrency is what consumes resources, so a downstream slowdown silently doubles the demand for connections, threads, and sockets. Crossing the pool limit is where it turns into visible failure. Option A confuses arrival rate with resource occupancy — the same requests per second each occupy a slot twice as long. Option C describes what a system with admission control would deliberately do (Module A-8), not what happens automatically; without it, requests pile up. Option D has the causality backwards: shrinking the pool constrains your own concurrency and does nothing to the dependency's speed.

Q: An SRE proposes running latency-sensitive services at 85–90% CPU utilisation to reduce cost, arguing that unused capacity is waste. What does the module's queueing result say?

Response time grows roughly as S ÷ (1 − ρ), so ~90% utilisation implies about 10× the service time versus 2× at 50% — the idle headroom *is* the latency budget, and 75% already sits one traffic increment from the cliff. — The module gives W = S ÷ (1 − ρ) and the accompanying table — 2× at 50%, 5× at 80%, 10× at 90%, 100× at 99% — and stresses that the exact figures depend on the model while the hyperbolic *shape* is universal. Its stated conclusions are to target 50–70% for latency-sensitive work, to treat idle capacity as the latency budget rather than waste, and to watch the trend because the knee sits close to the cliff. Option D asserts linearity, which is precisely the intuition the section corrects. Option B is wrong: the result is a general queueing property and applies to any resource requests wait for — threads, connections, the Node.js event loop. Option A denies the relationship altogether, and is the belief that produces "it was fine until it wasn't" incidents. Worth adding: real workloads have high variance in service time, which makes queueing worse at the same utilisation than the formula suggests.

What this module covers: The average response time is the most quoted and least useful number in production. This module replaces it: what percentiles actually mean, why a p99 that "only affects 1% of requests" affects a large fraction of users, how fan-out amplifies the tail until the rare becomes routine, the real sources of tail latency, why you can't average percentiles, Little's Law as the bridge from QPS to server count, and the queueing result that explains why a system at 90% utilisation has 10× the latency of the same system at 50%. Plus latency budgets — how to divide a 300ms promise across the hops that have to keep it.

The Average Is the Number That Hides the Outage

Consider two services. Both report a mean response time of 120ms.

	Service A	Service B
p50	115 ms	40 ms
p90	130 ms	90 ms
p99	150 ms	2,400 ms
p99.9	180 ms	14,000 ms
Mean	120 ms	120 ms

Service A is healthy and predictable. Service B has a subset of requests taking fourteen seconds — timeouts, abandoned carts, retry storms, support tickets — and its dashboard says it's identical to A. The mean is not slightly misleading here; it's actively concealing the only thing worth knowing.

This happens because latency distributions are not symmetric. There's a hard floor (the work genuinely takes some minimum time) and no ceiling (a request can queue, retry, hit a GC pause, or wait on a lock for effectively unbounded time). The distribution is right-skewed with a long tail, and an average of a long-tailed distribution tells you almost nothing about its shape.

Analogy: the mean is a group photo where everyone's face is blended into one composite. It looks like a person. It tells you nothing about whether anyone in the group is having a bad day.

Why this matters in production: every incident of the form "the dashboards were green but customers were complaining" is this. Averages stay green while the tail burns, because the tail is by definition a small share of requests — and a small share of requests is a large share of users, for reasons the next section makes uncomfortable.

Percentiles, Stated Precisely

p99 = 250ms means: of all requests in the window, 99% completed in 250ms or less, and 1% took longer. It says nothing about how much longer — that's what p99.9 and the maximum are for.

Which percentiles to actually watch:

p50 (median) — the typical experience. Good for capacity trends, bad for user-visible quality.
p90/p95 — where degradation first becomes visible to real people.
p99 — the standard SLO target for user-facing APIs. Your unluckiest requests, but frequent enough to be a routine event at any real volume.
p99.9 — where infrastructure problems hide: GC pauses, failovers, single bad nodes, noisy neighbours.
max — usually one weird thing, but occasionally the early warning for a systemic problem. Never alert on it alone.

The trap: 1% of requests is not 1% of users. If loading one page issues 20 API calls, a user needs all twenty to be fast. With an independent 1% chance of each being slow:

text

P(all 20 fast)  = 0.99²⁰ ≈ 0.82
P(at least one slow) ≈ 18%

Nearly one in five page loads hits the p99 path. Your "1% problem" is an 18% problem in the only unit that matters. And at higher fan-out it gets worse fast:

Backend calls per page	Chance at least one is a p99 request
1	1%
10	10%
20	18%
100	63%
500	99.3%

At 100 calls per request — an entirely normal number for a microservices page assembly or a feed hydration — the p99 case is the majority case. This is the core argument of Dean and Barroso's "The Tail at Scale," and it's the reason tail latency is treated as an architectural concern rather than a tuning detail: at scale, you don't get to treat rare events as rare. Modules A-6, A-7, and A-8 are collectively the response to this table.

Where the Tail Actually Comes From

Tail latency is rarely "the code is slow." The usual causes, roughly in order of how often they turn out to be the answer:

Queueing. The dominant cause, and the subject of the utilisation section below. The request wasn't slow; it waited — for a thread, a connection from the pool, a database lock, a disk I/O slot.
Connection pool exhaustion. A request that waits 2 seconds for a free database connection has 2 seconds of latency with no slow query anywhere in the trace. Module P-4.
Garbage collection and runtime pauses. A stop-the-world pause freezes every in-flight request on that instance. This is a classic p99.9 signature: periodic, correlated, invisible in the mean.
Cold caches. A cache miss costs the full backend path. After a deploy or an eviction wave, the miss rate spikes and so does the tail — Module P-12.
Noisy neighbours. On shared infrastructure, someone else's workload steals CPU, disk, or network. You'll see it as unexplained variance you cannot reproduce.
Retries and timeouts. A client that retries after 1s turns one slow request into two, adding load precisely when the system is struggling. Module A-6.
Network-level events. A TCP retransmit adds hundreds of milliseconds; TLS renegotiation, DNS re-resolution, and connection re-establishment all appear as inexplicable single-request spikes (Modules F-5, F-6).
Skew in the data, not the system. The user with 40,000 followers, the tenant with 12 million rows, the order with 900 line items. Same code path, 100× the work. Module P-8's whale accounts.

Why this matters in production: notice how many of these are invisible from inside application code. The request handler did nothing slow; it queued, or the runtime paused, or the pool was full. This is exactly why Module O-1 insists on distributed tracing — a trace shows the gap between spans, and the gap is where the tail lives.

You Cannot Average Percentiles

This one causes real, quiet damage in monitoring setups.

If instance A reports p99 = 100ms and instance B reports p99 = 300ms, the fleet's p99 is not 200ms. Percentiles aren't linear — you can't average them, and you can't sum them across hops either. To compute a true fleet-wide p99 you need the underlying distributions merged (which is what histogram-based metrics such as Prometheus histograms and HDR-histogram-style structures are designed for), not the per-instance percentiles combined.

Two consequences worth acting on:

Prefer histogram or sketch-based metrics over pre-computed per-instance percentiles, so aggregation stays mathematically valid.
A rising fleet p99 can come entirely from one bad instance. Always be able to break the percentile down by instance, availability zone, and deploy version — otherwise "p99 doubled" leads to a fleet-wide investigation when the answer is one host with a failing disk.

There's a related subtlety that bites in load testing: coordinated omission. A load generator that waits for each response before sending the next request stops sending during a stall — so the requests that would have been slow are never issued and never measured. The measured p99 looks fine while the real one is catastrophic. Fixed-rate load generation is the fix, and Module O-5 covers it properly.

Little's Law: Turning QPS Into Servers

Little's Law is one equation and it applies to every queueing system, with no assumptions about distributions:

text

L = λ × W

L = average number of requests in the system (concurrency)
λ = arrival rate (requests per second)
W = average time in the system (latency, in seconds)

Worked:

text

1,000 requests/second, average latency 200ms (0.2s)
L = 1,000 × 0.2 = 200 requests in flight, on average

That number is the one that sizes things, because concurrency is what consumes resources. 200 in-flight requests means:

~200 threads if your model is thread-per-request (plus headroom — call it 300).
~200 database connections if each request holds one for its whole duration, which is why Module P-4 exists.
In Node.js, 200 concurrent async operations — cheap in memory, but 200 simultaneous open sockets and pending callbacks.

The most useful thing about Little's Law is the direction people don't use it in: if latency goes up, concurrency goes up at the same arrival rate. Latency doubling from 200ms to 400ms doubles in-flight requests from 200 to 400 with no change in traffic. If your pool holds 250, you have just crossed a hard limit and the system will start failing — not because traffic grew, but because something downstream got slower. This is the mechanism behind most cascading failures, and it's why Modules A-7 and A-8 push you to shed load early instead of accumulating it.

Inverted, it also tells you your throughput ceiling:

text

Max throughput = concurrency limit ÷ latency
A pool of 100 connections, each held 50ms → 100 ÷ 0.05 = 2,000 TPS, maximum.

No amount of extra application instances raises that ceiling. Only shortening the hold time or widening the pool does — and widening the pool has its own cost at the database (Module P-4 again).

Utilisation: Why 90% Busy Means 10× Slower

Here's the result that explains most "it was fine until it wasn't" incidents. For a simple queueing model (M/M/1 — random arrivals, one server), average response time relates to utilisation ρ as:

text

W = S ÷ (1 − ρ)

S = service time (how long the work itself takes)
ρ = utilisation (0 to 1)

Utilisation	Response time vs service time
50%	2×
70%	3.3×
80%	5×
90%	10×
95%	20×
99%	100×

Read the shape, not the exact figures — real systems have multiple servers and non-random arrivals, so the numbers differ, but the curve is the same: latency doesn't degrade linearly with load, it degrades hyperbolically, and it goes vertical near saturation.

Three practical conclusions:

Target 50–70% utilisation for latency-sensitive services, not 90%. The idle capacity isn't waste; it's the entire latency budget. This is the honest answer to "why are we paying for servers that are half idle" — Module O-5 turns it into a defensible headroom target, and Module O-8 prices it.
The knee is close to the cliff. A service at 75% utilisation looks healthy and is one traffic increment from 90%. Alerting on utilisation trend beats alerting on a threshold.
Variance makes it worse than the table suggests. The formula assumes exponentially distributed service times. Real workloads have a mix — the 900-line-order alongside the 2-line-order — and higher variance in service time means more queueing at the same utilisation. Uneven work is more expensive than the average suggests.

Latency Budgets: Dividing the Promise

If the product requirement is "search results in under 300ms at p99," that number has to be allocated, because every hop spends some of it:

text

Total budget:                    300 ms (p99)
├─ TLS + network, client → edge:  40 ms   (Modules F-5, F-6)
├─ Edge → app, same region:        5 ms
├─ Auth check (cached):            5 ms
├─ Search query:                 120 ms   ← the actual work
├─ Result hydration (batched):    60 ms   ← N+1 here would blow the budget alone
├─ Serialisation + compression:   20 ms
└─ Unallocated headroom:          50 ms

Writing it down does three things a single target can't. It shows immediately whether the design is even feasible — if the search engine's own p99 is 400ms, no amount of optimisation elsewhere saves you, and the requirement or the engine has to change. It makes the sequential-call cost visible, which is why hydration is written as batched: twenty sequential 5ms calls is 100ms of budget, while one batched call is 5ms. And it turns timeout selection into arithmetic rather than folklore — a downstream timeout larger than its share of the budget is a guarantee the whole request blows through the total (Module A-6's deadline propagation).

The rule worth internalising: budget in p99s, not averages. Summing average latencies across five hops and calling it the total is how a 300ms design ships and measures 900ms in production.

Where This Shows Up in Your Stack

PostgreSQL: pg_stat_statements reports mean_exec_time and total time, and the query that dominates total time is often not the slowest one — it's a fast query called constantly. Meanwhile a query with a bimodal distribution (fast on cached pages, slow on cold ones) has a mean that describes neither case.
Node.js: the event loop is a queue, so Little's Law applies directly to it. One CPU-heavy synchronous operation blocks the loop and adds its full duration to the latency of every request already queued — a textbook tail-latency generator that leaves no slow span in any individual trace.
Redis: single-threaded command execution means one O(n) command (KEYS, a huge LRANGE) delays every other client. Redis's p99 is often a story about one badly-shaped command rather than about Redis.
Load balancers: least-connections routing generally beats round-robin for tail latency, because round-robin keeps sending work to an instance that's already stuck. Module F-11 covers why, and what health checks miss.
Serverless: cold starts are a tail-latency phenomenon by construction — a small fraction of invocations pay initialisation cost. Your p50 can be excellent while p99 includes a full container boot. Module F-10.

Summary

Averages hide outages. Two services with identical means can differ by 100× at p99. Latency distributions are right-skewed with unbounded tails.
Watch p50, p95, p99, p99.9 — p99 for user-facing SLOs, p99.9 for infrastructure problems.
1% of requests is not 1% of users. With 100 backend calls per page, 63% of pages hit a p99 path. Fan-out turns rare into routine.
The tail usually comes from waiting, not from slow code: queueing, pool exhaustion, GC pauses, cold caches, noisy neighbours, retries, and data skew.
Percentiles can't be averaged or summed. Use histogram-based metrics, and always be able to slice by instance and deploy.
Little's Law: L = λ × W. Concurrency is what consumes resources — and rising latency raises concurrency at constant traffic, which is how cascading failures start.
Latency degrades hyperbolically with utilisation — roughly 10× service time at 90% busy. Idle headroom is the latency budget.
Allocate a latency budget across hops, in p99s, and let it tell you whether the design is feasible before you build it.

The next module, The Anatomy of a Request, spends that budget concretely: DNS, TCP handshake, TLS, and HTTP, hop by hop, with the milliseconds attributed — so when a request takes 800ms you know which layer to interrogate first.

Knowledge Check

A page assembles its content from 100 backend calls, each with an independent 1% chance of falling into its own p99 tail. Leadership argues the p99 is "only 1% of requests" and not worth engineering effort. What does the module say?

A service handles 1,000 req/s at 200ms average latency, using a connection pool capped at 250. A dependency slows down and latency rises to 400ms, with traffic unchanged. What does Little's Law predict, and why does it matter?

An SRE proposes running latency-sensitive services at 85–90% CPU utilisation to reduce cost, arguing that unused capacity is waste. What does the module's queueing result say?

Test your knowledge with more question sets

Sign in to keep reading

The rest of this module is free — sign in with Google to unlock it and track your progress.

PreviousModule F-3: Back-of-the-Envelope Estimation Next Module F-5: The Anatomy of a Request

Discussion

Join the discussion

Loading comments...