The cache layers from browser to CDN to app to database, hit ratio as the only number that matters, TTL choice, and sizing a cache to the working set instead of the dataset.
What this module covers: Caching is the highest-leverage technique in system design and the one with the most easily-forgotten cost. This module covers the full cache hierarchy from browser to disk, why hit ratio is the only number that matters and why the last percentage point is worth the most, sizing to the working set instead of the dataset, choosing a TTL as an explicit staleness budget, eviction policies and how a single analytics scan can destroy a cache, cache key design and the key-explosion failure that silently drops your hit rate to zero, and the cases where caching is the wrong answer. It closes with the four metrics that tell you whether a cache is working.
Caching Trades Staleness for Latency and Cost
That's the whole deal, and it deserves to be stated as a trade rather than an improvement. You get faster reads and less load on the origin; you pay by serving data that may be out of date, plus the memory it occupies and the invalidation logic you now maintain.
The important consequence: "how stale can this be?" is a product question, not an infrastructure question. Module F-1 listed consistency tolerance as one of the four numbers that change every answer, and a cache is where that number gets spent. If nobody has said it out loud, you've made a product decision inside an infrastructure change — and the bug report ("I changed my name and it still shows the old one") will arrive as a mystery instead of an expected consequence.
Analogy: a cache is a photocopy on your desk instead of the original in the archive. Enormously faster to consult, and the entire risk is that someone edited the original since you copied it. Every question about caching reduces to: how out of date can this photocopy be before it does damage?
Why this matters in production: most cache incidents aren't "the cache was slow." They're "the cache served something wrong for longer than anyone expected," or "the cache disappeared and the origin couldn't take the load it had been shielded from." Both are consequences of the trade being made implicitly.
The Hierarchy: Every Layer Is a Cache
A single read can be answered at any of these, and each hit means every layer below it does nothing at all:
Layer
Typical latency
Scope
Who controls it
Browser memory / HTTP cache
~0 ms
One user
Cache-Control headers
CDN / edge
5–50 ms
All users near a PoP
Headers + purge API
Reverse proxy (nginx, Varnish)
1–5 ms
All users
Proxy config
Application in-process
~0 ms
One instance
Your code
Distributed cache (Redis)
0.5–2 ms
All instances
Your code
Database buffer pool
~0 ms (RAM hit)
All queries
shared_buffers
OS page cache
~0 ms
All file reads
The kernel
Disk
50 µs – 8 ms
—
—
Two principles fall out of the table:
Cache as far out as you can tolerate. A CDN hit never touches your infrastructure — no compute, no database, no egress from your origin. An in-process hit is faster than Redis but is per-instance (Module F-9: N instances means N caches, N× memory, lower combined hit rate, and instances that can disagree). Pushing a cache outward multiplies its value and reduces your control, which is exactly the tension.
You already have caches you didn't configure. Postgres's shared_buffers and the OS page cache are doing enormous work invisibly. This is why a query is 2ms on the second run and 200ms on the first, and why a database restart produces a slow period that looks like a regression. Before adding a cache layer, check whether the existing ones are simply undersized.
Now the non-obvious part, and it's the most useful idea in the module: the marginal value of hit ratio increases as hit ratio increases.
Going 50% → 80% removes 60% of origin load.
Going 90% → 95% removes half the remaining origin load.
Going 98% → 99% removes half again.
Each additional percentage point near the top halves the traffic your database sees. So a cache at 90% is not "nearly optimal" — the work to reach 98% would reduce database load five-fold. This inverts the usual diminishing-returns intuition, and it's why tuning an already-good cache is often better value than optimising the origin.
The mirror image is the risk: at 99% hit ratio, your origin is sized for 1% of traffic. Lose the cache — an eviction wave, a restart, a failover, a bad deploy that changes key formats — and the origin instantly receives 100× its normal load. This is why the cache is a critical dependency and not an optimisation, and it's the entire subject of Module P-12's stampede section.
Sizing: Working Set, Not Dataset
You don't need to cache everything; you need to cache what's actually being read (Module F-3's fourth calculation). Access in real systems is heavily skewed — a small fraction of items absorbs most requests.
The right approach is to measure rather than assume:
sql
If the top 1,000 items serve 85% of reads, caching 1,000 items buys an 85% hit ratio for a trivial amount of memory. That's the number to design around — and it also tells you when caching won't help, because a flat access distribution (every item read about equally, rarely) has no working set to exploit.
Then add headroom. A cache sized exactly to the working set will evict hot keys whenever the set shifts — a new campaign, a viral item, a batch job — and the hit ratio collapses precisely when traffic is highest.
TTL: Writing Down the Staleness Budget
A TTL is a promise about how stale data may be. Choose it from three inputs:
How often the data actually changes. A product's description changes weekly; its stock count changes constantly.
How much damage staleness does. A slightly stale view count is invisible; a stale permission check is a security incident; a stale price is a financial and legal problem.
How expensive a miss is. An origin computation costing 2 seconds justifies a longer TTL than one costing 5ms.
Rough anchors:
Data
TTL
Reasoning
Immutable assets (hashed filenames)
1 year
Content can't change; the URL changes instead
Reference data (countries, categories)
hours–days
Changes rarely, staleness harmless
Product catalogue
minutes
Changes occasionally, brief staleness acceptable
Aggregates and counts
seconds–minutes
Approximate by nature
Inventory, pricing
seconds, or event-invalidated
Staleness has direct financial cost
Permissions, sessions
very short, or never cached
Staleness is a security bug
Two techniques worth knowing now:
TTL jitter. If a thousand keys are written at once with an identical TTL, they all expire at once and produce a synchronised stampede. Adding randomness (ttl = base + random(0, base × 0.1)) spreads the expiry. This is the same jitter reasoning as retry backoff (Module A-6) — correlated timing is the enemy.
Stale-while-revalidate. Serve the stale value immediately and refresh in the background. The user gets cache latency, the data converges shortly after, and no request ever pays the miss cost. It's a strictly better trade than a hard TTL for most read-heavy content, and it's built into HTTP (stale-while-revalidate) and into most CDN configurations.
LFU (least frequently used) — better when popularity is stable and you don't want a brief burst to displace a long-term favourite. Redis's allkeys-lfu implements this with frequency counters.
TTL-only — evict on expiry; nothing evicted early. Requires that the total set genuinely fits.
Random — surprisingly acceptable, and very cheap. Redis offers it, and under a skewed distribution it usually evicts a cold key.
The failure mode worth internalising is sequential flooding, and it's LRU's weakness:
text
Nothing malfunctioned. LRU did exactly what it promises, and the promise was wrong for a scan. The defences:
Don't populate the cache from analytical or batch reads. Give those queries a path that bypasses caching entirely — often the cleanest fix.
Use LFU where scans are unavoidable, so a single access doesn't outrank a thousand.
Separate caches (or Redis databases/prefixes with their own budgets) for workloads with different access patterns. That's a bulkhead in Module A-7's sense.
Route analytics elsewhere entirely. Module P-22's whole argument is that OLAP work does not belong on the OLTP path — and this is one of the reasons.
Cache Keys: The Silent Hit-Rate Killer
A cache key must include everything that changes the response and nothing that doesn't. Both halves fail in practice.
Missing a dimension means serving one user's data to another. If a personalised response is keyed only by URL, you will eventually cache user A's dashboard and serve it to user B. This is not a hypothetical class of bug; it is one of the more serious real-world caching incidents, and it happens at CDNs when Vary or the cache key configuration doesn't account for an auth cookie.
Including too much means the hit ratio quietly goes to zero:
typescript
Two habits worth adopting permanently:
Version your keys. A version segment lets a deploy that changes the response shape invalidate everything atomically — the old keys age out and no consumer ever sees a mismatched shape. Far safer than a mass purge, and it makes rollback trivial.
Watch key cardinality as a metric. A sudden rise in distinct keys with a falling hit ratio is the unmistakable signature of an accidental key dimension, and it's invisible if you only monitor latency.
When Not to Cache
Caching is not free, and these cases cost more than they return:
Data with no reuse. Highly personalised content read once per user has no hit ratio to gain. You've added a dependency and a write for nothing.
Cheap origin reads. A 1ms indexed primary-key lookup is already about as fast as a Redis round trip. Caching it adds a network hop, a serialisation cost, and an invalidation problem to save nothing.
Data where staleness is a correctness bug. Permissions, account balances, inventory at the point of committing a sale. If you cache these, you need event-driven invalidation with real guarantees, not a TTL and hope.
As a substitute for fixing the query. A cache in front of a query that takes 4 seconds because it's missing an index means every miss takes 4 seconds — and misses are exactly what happens during a stampede, a deploy, or an eviction wave. Fix the query, then consider the cache. Caching a bad query hides it until the worst possible moment.
That last one is worth saying twice: a cache converts a performance problem into an availability problem. The slow query stops being visible day to day and becomes a cliff you fall off when the cache is cold.
Measuring a Cache
Four metrics, and none of them is latency:
Hit ratio, per cache and per key prefix. A fleet-wide 92% can hide one prefix at 30%.
Miss latency — what a miss actually costs, since that's your worst-case path and the thing that determines survival during a cold start.
Eviction rate — steady evictions with a good hit ratio means the working set is near capacity, so you're one shift away from a collapse.
Key cardinality — the early warning for key explosion.
And measure the thing hit ratio can't tell you: what happens with the cache empty. Load-test the cold path deliberately (Module O-5), because that's the state you'll be in during an incident, and the version of your system that runs without a cache is the version you'll actually have to survive.
Where This Shows Up in Your Stack
Next.js: the framework layers several caches — the client router cache, the full-route cache, and the data cache — and most "why is my page stale" confusion is not knowing which layer answered. revalidate is a TTL; revalidateTag is tag-based invalidation (Module P-13), and each has a distinct staleness budget.
PostgreSQL: compare heap_blks_hit and heap_blks_read in pg_statio_user_tables for the buffer-pool hit ratio. A poor ratio usually means shared_buffers is undersized or the working set exceeds RAM — and increasing memory is often better value than adding an application cache layer.
Redis:INFO stats gives keyspace_hits/keyspace_misses; maxmemory-policy is your eviction choice, and allkeys-lru vs volatile-lru differ in whether keys without a TTL can be evicted — a real source of surprise when a cache silently stops evicting and starts refusing writes.
CDNs: the cache key configuration is the highest-stakes setting you own. Including an auth cookie is safe and lowers the hit ratio; excluding one when responses are personalised leaks data between users.
Summary
Caching trades staleness for latency and cost. Name the staleness budget out loud, or you've made a product decision inside an infrastructure change.
Every layer is a cache — browser, CDN, proxy, in-process, Redis, buffer pool, page cache. Cache as far out as you can tolerate; a CDN hit costs you nothing at all.
You already have caches you didn't configure. Check whether shared_buffers and the page cache are undersized before adding a layer.
Hit ratio's marginal value increases with hit ratio: 90% → 95% halves origin load, and 98% → 99% halves it again. The corollary is that at 99% your origin is sized for 1% of traffic.
Size to the measured working set, plus headroom — a cache sized exactly to it collapses whenever the set shifts.
Choose TTL from change rate, staleness damage, and miss cost. Add jitter to avoid synchronised expiry; prefer stale-while-revalidate for read-heavy content.
LRU dies to sequential floods. One nightly scan can evict the entire hot set; bypass caching for batch reads, or use LFU, or separate the caches.
Cache keys must include every varying dimension and nothing else. Missing one leaks data between users; adding a spurious one drops the hit ratio to zero. Version your keys; monitor cardinality.
Don't cache low-reuse data, already-cheap reads, correctness-critical values, or a slow query you haven't fixed — caching a bad query converts a performance problem into an availability problem.
Measure hit ratio per prefix, miss latency, eviction rate, and key cardinality — and load-test the cold path.
The next module, Database Fundamentals for Architects, goes to the layer the cache is protecting: schema modelling, B-tree indexes, what the query planner is deciding, and how one slow query becomes an upstream outage.
Knowledge Check
A cache is at 90% hit ratio. One engineer argues further tuning is low-value since "we're already catching most requests." What does the module's arithmetic say?
A service's cache runs at 95% hit ratio all day, but every morning the first twenty minutes are slow with near-zero hit rate. A nightly export job scans several million rows. What's happening?
A report takes 4 seconds because a query is missing an index. A team puts a Redis cache in front of it, and p50 improves dramatically. What does the module say about this?
Test your knowledge with more question sets
Sign in to access a wider variety of questions and get notified when new practice sets are added to this module.
-- Which fraction of rows serves most of the reads? Derive it from access logs-- or an events table rather than guessing at 80/20.WITH ranked AS(SELECT resource_id,COUNT(*)AS hits, ROW_NUMBER()OVER(ORDERBYCOUNT(*)DESC)AS rank
FROM access_log
WHERE occurred_at >now()-interval'7 days'GROUPBY resource_id
)SELECTCOUNT(*)AS distinct_resources,SUM(hits)AS total_hits,SUM(hits) FILTER (WHERE rank <=1000)*100.0/SUM(hits)AS pct_from_top_1000;
Cache holds 10,000 hot items, hit ratio 95%.
A nightly export job scans 5 million rows, and each read populates the cache.
LRU dutifully treats each freshly-scanned row as "most recently used."
All 10,000 hot items are evicted.
The morning traffic arrives to an empty cache — 0% hit ratio, 100× origin load.
// ✗ Key explosion: the timestamp makes every key unique. Hit ratio ≈ 0,// and the cache steadily fills with garbage nobody will ever read again.const key =`product:${id}:${locale}:${Date.now()}`;// ✗ Also broken: a session ID in the key for a response that isn't personalised.// One entry per session for identical content.const key =`products:list:${sessionId}`;// ✓ Only the dimensions the response actually varies by.const key =`product:${id}:${locale}:${currency}:v3`;// ^^ schema version: bump this to// invalidate every entry at once after a shape change, instead of purging.