What this module covers: A complete worked design for a home timeline at global scale, and the phase's mirror image of Module P-24 — where a payment system put correctness above availability, a feed puts availability above consistency, and almost every decision inverts as a result. The fan-out choice, worked with real arithmetic: write-time fan-out gives cheap reads and cannot survive a celebrity, read-time fan-out gives cheap writes and an unpredictable 200-way scatter-gather, and the hybrid bounds both. A cache hierarchy that earns its complexity because timelines and posts have opposite sharing profiles. Ranking as a two-stage system with a latency budget, and the pagination bug that ranking creates. The celebrity problem in detail, including the inversion where you push to caches rather than to timelines. Multi-region where the only thing that must not be stale is your own post. And deletion, where the lesson is that a denormalised view is filtered at read rather than corrected at write.
Requirements, and the Priority That Inverts
Functional:
A home timeline: recent posts from accounts a user follows, ranked.
Post and delete; posts may be public or restricted.
Engagement counts (likes, replies) visible on each post.
Blocks and mutes must be honoured.
An indication that new content is available.
Non-functional:
Target
Read:write ratio
~100:1
Feed load latency
p99 < 200 ms
Freshness
seconds to ~a minute is acceptable
Availability
99.99% for reads
Correctness of counts
approximate is fine
Own-post visibility
immediate, always
And the priority statement, which is the whole reason this capstone reads differently from the last one:
Availability and latency outrank consistency here. A feed that is thirty seconds stale is not a defect. A feed that takes two seconds to load, or fails, is. A like count that is off by three is invisible; a like count that costs a database write per like is an outage.
Module P-24 said the opposite — fail the payment rather than risk a double charge. Both are correct, and the discipline is to state which one you are building before choosing mechanisms, because nearly every decision below follows from that single line. The one exception is the last row: a user must always see their own post immediately, and that exception drives a specific piece of design.
Analogy: a newspaper's editorial process versus its printing. The bank's ledger from Module P-24 is double-entry bookkeeping — every entry balanced, nothing overwritten. A newspaper is the opposite: a hundred editions printed and distributed to newsagents in advance, each slightly different, none of them corrected once printed. If a story is pulled, you do not send drivers to retrieve every copy from every shop — you stop selling it and the next edition omits it. That is exactly the delete design below, and the reason is the same: the copies are cheap and disposable, and chasing them costs more than the error does.
Estimation: the Numbers That Decide the Architecture
Quantity
Value
Registered users
500 M
Daily active users
200 M
Posts per DAU per day
~0.5
Posts per second
100 M/day ≈ 1,200/s average, ~5,000/s peak
Feed loads per DAU per day
~15
Feed reads per second
3 B/day ≈ 35,000/s average, ~100,000/s peak
Median follows
~200
Largest account
~100 M followers
Two derived numbers carry the design:
Write-time fan-out volume:1,200 posts/s × 200 followers = 240,000 timeline writes/second on average, and around a million per second at peak. That is a very large number of very small writes — which is achievable, and it tells you the timeline store must be a cheap append-oriented structure, not a relational table with indexes.
The celebrity number: one post by a 100 M-follower account is 100 million timeline writes. At a sustained 500,000 writes/second that is 200 seconds — during which it consumes the entire fan-out capacity and every ordinary user's post is delayed behind it (Module A-8's head-of-line problem, at feed scale). This single number is why pure write-time fan-out is not an option, and it is worth computing early because it invalidates the simplest design outright.
Storage: posts are small (a few hundred bytes of text plus references), so 100 M/day is tens of gigabytes per day — trivial. Timelines are the volume: 200 M active users × 800 cached post IDs × ~16 bytes ≈ 2.5 TB of timeline data, which fits in a Redis cluster of a reasonable size and, crucially, is rebuildable — a property we will design for deliberately.
The Central Decision: Fan-Out on Write or on Read
text
On write (push)
On read (pull)
Write cost
O(followers) — huge
O(1)
Read cost
O(1) list read
O(followees) scatter-gather + merge
Read latency
predictable, low
variable, tail-dominated
Wasted work
fan-out to users who never log in
none
Celebrity post
catastrophic
free
Storage
a materialised list per user
none
For a 100:1 read-heavy workload, push is obviously right — you move work to the rare operation. Except for the celebrity row, and except that fanning out to 500 M registered users when only 200 M are active wastes 60% of the writes.
So the design is hybrid, and the rule is:
Push for ordinary accounts (below a follower threshold, say 100,000): their posts are written into follower timelines at post time.
Pull for large accounts: their posts are not fanned out. At read time, the feed service also fetches recent posts from the small set of large accounts this user follows, and merges.
Push only to active users (seen in the last ~30 days). Inactive users' timelines are rebuilt on next login from the pull path.
Why the hybrid bounds both sides: the push cost is now bounded by followers of ordinary accounts, which has a ceiling; the pull cost is bounded by how many large accounts one user follows, which is a handful. Neither side has an unbounded term, and that is the test of a good hybrid.
The threshold is a tuning parameter with a real trade: lower it and more accounts are pulled, making reads more expensive for everyone; raise it and a single post's fan-out gets larger and slower. Around 10⁵ followers is a defensible starting point, adjusted by measurement.
Data Model
Posts — the source of truth, immutable once written:
sql
A Snowflake-style ID matters here for two reasons from Module P-3: it is generated without coordination across shards, and it is time-sortable — so a timeline can be ordered and paginated by ID alone, with no secondary index and no timestamp comparison.
The follow graph — the hot metadata:
text
Both directions are needed: followees(user) for the pull path and the initial rebuild, followers(author) for fan-out. It is read constantly and changes slowly, which makes it ideal for aggressive caching.
Timelines — per user, capped, and holding IDs only:
text
Three deliberate properties. IDs, not content — otherwise an edit or a delete would need to reach millions of copies, and storage would be 100× larger. Capped, because nobody scrolls past a few hundred items and an uncapped list is an unbounded storage commitment. And ZADD keyed on the post ID is naturally idempotent — it is an absolute set-membership operation, not an increment — so a retried or duplicated fan-out job is harmless, which is Module P-16's absolute-versus-relative distinction paying off exactly where at-least-once delivery is guaranteed.
Counters — approximate by design. A like must never be UPDATE posts SET likes = likes + 1: that is a relative operation on a single hot row, which serialises every like on a popular post and produces a dead tuple per like (Modules P-16, P-24). Instead: append like events to a log, aggregate periodically into a counter cache, and accept that the displayed number lags by seconds. For the hottest posts, shard the counter across N keys and sum on read.
The Write Path
text
Step 3 is the exception the requirements demanded: the author's own timeline is written synchronously, in the request, so the user always sees their own post immediately regardless of fan-out lag. It is one extra write and it removes the most-reported class of complaint.
The fan-out worker is Module P-19's job system with three specifics:
Separate queues by fan-out size. A 5-million-follower fan-out must not occupy the workers that ordinary posts need — that is head-of-line blocking, and the answer is the same as Module P-19's: separate queues with dedicated pools, so a large fan-out is slow without making everything slow.
Batch and pipeline aggressively. Ten thousand individual ZADDs is ten thousand round trips; batches of a thousand in a pipeline is ten. This is the difference between the fan-out rate being feasible and not.
Checkpoint by follower cursor. A fan-out of two million followers is a long job, so it records how far it has progressed (Module P-19's cursor) and resumes rather than restarting — and because ZADD is idempotent, an overlap after a crash costs nothing.
The SLI for this path is post-to-timeline latency at p99, and queue depth plus oldest-job age as the leading indicators (Modules P-19, A-8). Fan-out lag is the metric users actually feel as "the feed is behind."
The Read Path and a Cache Hierarchy That Earns Its Complexity
A feed load looks like this:
text
Four to six round trips total, not twenty — because everything is a batch read by ID. That shape is what makes a 200 ms p99 achievable.
Now the insight that justifies a layered cache rather than one big one:
Timelines are private and posts are shared, so they have opposite cache economics.
Timeline (timeline:{user})
Post object (post:{id})
Read by
one user
up to millions of users
Cross-user hit ratio
zero
extremely high
Size
200 M distinct keys
far fewer hot keys
Best home
Redis cluster, sharded by user
in-process cache and Redis
TTL
long; rebuildable from the graph
long — content is immutable
So the hierarchy is not ceremony:
L1, in-process — hot post objects and author profiles. A celebrity post read a million times a minute is served from local memory at ~100 ns rather than 0.5 ms across the network (Module F-12's hierarchy argument at its most favourable). Bounded size, LRU, short TTL to bound staleness.
L2, Redis — timelines (sharded by user), post objects, counters, block lists.
L3, the datastore — posts and the graph in Postgres (partitioned), or a wide-column store for timelines at the largest scale (Module P-9).
The CDN caches media and immutable post content by content-addressed key (Module A-11) — it cannot cache a personalised feed, but the payload bytes are the bulk of the transfer.
And design the timeline store as a cache, not as a source of truth. Losing the Redis cluster should be a degradation, not a data loss: timelines are rebuildable from follows plus posts, and rebuilding a user's timeline on demand is a bounded query. That single property turns a catastrophic dependency into a recoverable one, and it is worth accepting some rebuild cost to preserve it.
Ranking as a System
Chronological is simple, cheap, cacheable, and what many users say they prefer. Ranked feeds exist because they measurably increase engagement, and they change the architecture:
text
The latency budget is the constraint. Scoring 500 candidates inside ~50 ms means the model must be cheap and, more importantly, feature lookup must be precomputed — per-user and per-post features fetched as one batch read from a feature cache, never computed per candidate. The usual first mistake is a scoring function that issues a query per candidate; at 500 candidates that is the whole budget spent on round trips.
Ranking makes caching harder. A ranked feed is a function of (user, candidate set, model version, time), so it is far less cacheable than a chronological list. The workable split: cache the candidate set (which changes only when new posts arrive) and either compute the ranking fresh or cache the ranked page for a short window.
And ranking creates a pagination bug worth naming. If the ranking is recomputed on each page request, items shift between pages, so a user scrolling sees duplicates and misses posts entirely. The fix is a session-scoped pagination token that pins the candidate set and the ranking seed for the duration of the scroll — a cursor over a snapshot, which is Module P-20's keyset-cursor argument with an added requirement that the ordering be stable, not just the position.
The Celebrity Problem, in Detail
One account, 100 million followers, one post. Four separate problems, and each needs its own answer.
1. Fan-out is impossible at write time. 100 M writes at 500 k/s is 200 seconds of exclusive capacity. So this account is on the pull path: its post is written once, and its followers pick it up at read time from a small per-account "recent posts" list. A user following twelve large accounts does twelve small reads.
2. The read-time cache stampede. The moment the post is published, millions of readers request the same post object, all miss, and all hit the datastore — Module P-12's thundering herd at its worst. Two mechanisms:
Request coalescing / single-flight: the first miss fetches, everyone else waits on that one fetch. Per instance this is a few lines and removes almost all of the amplification.
Pre-warm at publish time, which is the pleasing inversion: instead of pushing the post ID into 100 million timelines, push the post object into the caches — a few thousand writes rather than a hundred million, and it eliminates the stampede entirely. The push is to the caches, not to the timelines.
3. Engagement counters go hot. Millions of likes on one post is a single hot key, so shard the counter across N sub-keys (likes:{post}:{0..15}), increment a random shard, and sum on read — trading a slightly more expensive read for a write path that does not serialise (Module P-6's hot-partition mechanism, at the key level).
4. Its followers' timelines are unbalanced. A user following one celebrity and 199 ordinary accounts has a feed dominated by whichever path is cheaper. Re-ranking with author-diversity constraints is what stops the pull path's posts monopolising the merged result.
Multi-Region, Where Only One Thing Must Not Be Stale
Following Module A-9's topology: reads local, writes to the author's home region.
text
The design point that saves the most money: replicate the post store and the follow graph, then compute timelines locally in each region. Replicating the timeline writes themselves would be 240,000 small cross-region writes per second, which is an enormous egress bill (Module O-8) for data that is derivable. Replicate the inputs, derive the outputs locally — a general principle worth carrying beyond feeds.
Cross-region replication lag becomes feed staleness of a few seconds, which the requirements already accepted. That acceptance is what makes this whole topology cheap, and it is the payoff for stating the availability-over-consistency priority up front.
The one exception is read-your-own-writes (Module P-11). A user posting from region B must see their post immediately even if the write went to region A. Two mechanisms together: the synchronous write into the author's own timeline (step 3 of the write path), and pinning the author's session to read their own recent posts from an authoritative source for a short window. Every other kind of staleness is acceptable; this one produces "my post disappeared" and must not happen.
Deletes, Blocks, and Privacy: Filter at Read, Do Not Correct at Write
A post is deleted. It exists in the timelines of millions of followers. You cannot chase it — finding and removing it from 40 million sorted sets is a fan-out as expensive as the original, arriving unpredictably.
So the timeline stores IDs, and hydration is where the truth is applied:
text
A deleted post simply fails to hydrate. Blocks and mutes are a small per-user list checked at hydration. A visibility change from public to followers-only is evaluated at hydration. Nothing is corrected in the materialised view; stale IDs are lazily removed when encountered.
The general lesson: a denormalised, materialised view should be filtered at read time rather than corrected at write time, because correcting it costs a second fan-out with worse timing. It is the same reason Module A-11 stores an immutable object and a mutable pointer, and it is why the timeline holds IDs rather than content.
Note the requirement it creates: fetch more IDs than you need (over-fetch by a margin), because filtering removes some, and a page that returns twelve items when the user expected twenty is a visible bug.
For a genuine erasure request (Module O-9), the post row is deleted, the tombstone remains long enough for the lazy cleanup, and the media is crypto-shredded by deleting the per-subject key — which reaches every copy, including caches and backups, without rewriting any of them.
Failure Modes and the Degradation Ladder
Failure
Behaviour
Fan-out workers lag
feed is stale; alert on p99 post-to-timeline latency and queue age
Redis timeline cluster lost
rebuild on demand from graph + posts — degraded latency, no data loss, because timelines were designed as a cache
Ranking service down
serve chronological (a genuinely good fallback, Module A-7)
Post store degraded
serve from cache with a stale marker; shed the cold tail (Module A-8)
Counter service down
omit counts rather than failing the feed
One region down
route reads elsewhere; authors in that region cannot post (accepted)
The degradation ladder is the deliverable: ranked → chronological → cached-and-stale → a small "pull to refresh" state. Each step is a defined behaviour with a metric, not a stack trace. And note that "counts are missing" is a far better outcome than "the feed failed," which is the whole argument of Module A-7 in a product setting.
Why this matters in production: the feed failures that actually happen are fan-out lag nobody alerted on, a celebrity post that stalled every other user's fan-out for three minutes because the queues were shared, a like counter that took a hot row lock and made a popular post un-likeable, a delete that a support team believed had worked because they checked the post store rather than a follower's feed, and a ranked-pagination bug that showed users the same post four times. None of them is exotic; each maps to a decision in this design.
What We Deliberately Did Not Build
Notifications — a separate fan-out system with different semantics (per-user, must-not-miss, so it needs the durable delivery of Modules P-19 and A-10 rather than a disposable timeline).
Direct messages — a different data model, different privacy properties, and often end-to-end encryption.
Search and trending — Module P-20's territory, fed by the same event stream.
Integrity and moderation — a classification pipeline plus a read-time filter, which slots into the hydration step this design already has.
Ads and injected content — another candidate source in the ranking stage.
Video — Module A-11's pipeline, with the feed holding only references.
Summary
State the priority first: availability and latency over consistency. A thirty-second-stale feed is fine; a slow or failed feed is not; approximate counts are invisible. This is the exact inversion of Module P-24, and nearly every decision here follows from that line — with one exception, that a user must always see their own post immediately.
Compute the celebrity number early. One 100 M-follower post is 100 million timeline writes, or ~200 seconds of exclusive fan-out capacity, which invalidates pure write-time fan-out outright.
Fan out on write for ordinary accounts, pull at read for large ones, and merge. The test of a good hybrid is that neither side has an unbounded term: push is bounded by followers-of-ordinary-accounts, pull by large-accounts-followed-by-one-user. Also push only to active users and rebuild an inactive user's timeline on login.
Timelines hold post IDs, capped at a few hundred, in a sorted set scored by a time-ordered Snowflake ID — so ordering and pagination need no secondary index (Module P-3), storage stays small, and edits or deletes never have to reach millions of copies. ZADD keyed on the post ID is absolute and therefore idempotent, which makes at-least-once fan-out harmless (Module P-16).
Never UPDATE ... likes = likes + 1. Append like events and aggregate; shard the counter for the hottest posts. A relative update on one hot row serialises every like and produces a dead tuple each time.
The write path is: post row plus outbox in one transaction, respond, then CDC to a queue and fan-out workers — with the author's own timeline written synchronously in the request, which is one extra write that removes the most-reported complaint. Separate queues by fan-out size so a large account cannot starve ordinary posts, batch ZADDs in pipelines of ~1,000, and checkpoint by follower cursor.
A feed load is four to six batch round trips, not twenty, because every step is a multi-get by ID. That shape is what makes a 200 ms p99 possible.
The cache hierarchy earns its complexity because timelines are private and posts are shared: timelines have zero cross-user hit ratio and belong in a sharded Redis cluster; post objects have an extremely high hit ratio and belong in an in-process cache and Redis, with the CDN carrying media and immutable content.
Design the timeline store as a cache, not a source of truth. Timelines are derivable from the graph plus posts, so losing the cluster becomes degraded latency rather than data loss — a property worth paying rebuild cost to keep.
Ranking is two-stage — candidate generation then scoring then re-rank — and its constraint is the latency budget. Features must be precomputed and fetched in one batch; a query per candidate spends the entire budget on round trips. Cache the candidate set rather than the ranked result, and pin the ranking with a session-scoped pagination token, or scrolling shows duplicates and skips posts.
The celebrity post has four distinct problems: fan-out (use the pull path), a cache stampede at publish (request coalescing, plus the inversion of pre-warming the post object into caches — thousands of writes instead of a hundred million), hot engagement counters (shard and sum), and feed imbalance (author-diversity re-ranking).
Multi-region: replicate the inputs and derive the outputs locally. Replicating 240,000 timeline writes per second across regions is an enormous egress bill for derivable data; replicate the post store and follow graph, and compute timelines in each region. Accept the lag as staleness — except for read-your-own-writes, which needs the synchronous own-timeline write plus a short session pin.
Filter at read, do not correct at write. A deleted, blocked, muted or newly-private post simply fails to hydrate, and stale IDs are cleaned lazily — because chasing 40 million materialised copies is a second fan-out with worse timing. Over-fetch IDs by a margin so filtering does not shorten the page. Erasure is a row delete plus crypto-shredding the media key, which reaches caches and backups without rewriting them.
The degradation ladder is the deliverable: ranked → chronological → cached-and-stale → an honest empty state, with counts omitted rather than the feed failed. And post-to-timeline p99 is the SLI users actually feel.
That closes Phase 3. Architect has been about coordination: agreeing on a value, taking exclusive action, changing several things at once, drawing boundaries, bounding failure, and spreading across the planet. What it has not covered is running any of it — proving the breakers fire, knowing when you are violating a promise, migrating a schema under load, surviving the loss of a datacentre, funding the whole thing, and staying inside the law.
Phase 4, Production & Governance, is that. It opens with observability, because every mechanism in the last three phases has a metric that decides whether it is working, and none of them tell you on their own.
Knowledge Check
A team implements pure fan-out-on-write. It performs well in testing and in early production. When a 40-million-follower account joins the platform, every user's feed falls minutes behind whenever that account posts, though no component errors and the fan-out workers are at full utilisation. What does the module prescribe, and what secondary optimisation applies to the same account?
A user deletes a post that has already been fanned out to 40 million follower timelines. An engineer proposes a cleanup job that removes the post ID from every affected timeline. What does the module say, and what secondary requirement does the recommended approach create?
The team plans multi-region expansion. One proposal replicates the Redis timeline writes to the second region so both regions have identical materialised timelines. Another replicates only the post store and follow graph, computing timelines independently in each region. Which does the module prefer, and what single consistency exception must be preserved either way?
Test your knowledge with more question sets
Sign in to access a wider variety of questions and get notified when new practice sets are added to this module.
FAN-OUT ON WRITE (push) FAN-OUT ON READ (pull)
post → write post ID into every post → store once
follower's timeline list
feed → for each of 200 followees,
feed → read ONE list. Done. fetch recent posts, merge, sort
CREATETABLE posts ( id bigintPRIMARYKEY,-- Snowflake: time-ordered, shard-safe (Module P-3) author_id bigintNOTNULL, body text, media_keys text[],-- object-store keys (Module A-11) visibility smallintNOTNULL,-- public / followers / restricted created_at timestamptz NOTNULL, deleted_at timestamptz -- tombstone; see the delete section)PARTITIONBY RANGE (created_at);
follows: (follower_id, followee_id, created_at) -- indexed both directions
counters: follower_count per account -- drives the push/pull decision
Redis sorted set per user: timeline:{user_id}
member = post_id, score = post_id (Snowflake → chronological)
ZADD timeline:{u} <post_id> <post_id>
ZREMRANGEBYRANK timeline:{u} 0 -801 ← cap at 800 entries
POST /posts
1. write the post row (durable, source of truth)
2. write an outbox event in the same transaction (Module A-3 — no dual write)
3. respond 201, and insert the post into the AUTHOR'S OWN timeline immediately
────────────────────────────────────────────────────────────────────
4. CDC/outbox relay → Kafka topic, partitioned by author_id (Module P-23)
5. fan-out workers:
- look up follower_count
- if above threshold → do nothing (pull path handles it)
- else → page through followers, filter to active,
pipeline ZADD in batches of ~1,000
6. pre-warm the post object into the caches
7. notify connected clients that new content exists (SSE — Module A-10)
1. read timeline:{user} → 20 post IDs (1 Redis call)
2. read pull-path posts → recent posts from large accounts followed
3. merge + rank → ordered 20 IDs
4. MGET post objects → 20 post bodies (1 multi-get)
5. MGET counters + author profiles → (1–2 multi-gets)
6. filter: deleted, blocked, muted, visibility
Region A (author's home) Region B
post write → local post store replica (async)
fan-out → local timelines follow-graph replica
timelines computed LOCALLY from the replica
timeline gives 20 IDs → MGET post objects
→ drop any that are tombstoned, blocked, muted,
or no longer visible to this viewer
→ if fewer than 20 remain, fetch more IDs