Module A-14·32 min read

Fan-out on write vs on read, the celebrity problem, ranking, multi-region reads, and a cache hierarchy that earns its complexity.

JJS
Written by Jatin Jain Saraf · Senior Software Engineer

Module A-14 — Capstone: Design a Global News Feed

What this module covers: A complete worked design for a home timeline at global scale, and the phase's mirror image of Module P-24 — where a payment system put correctness above availability, a feed puts availability above consistency, and almost every decision inverts as a result. The fan-out choice, worked with real arithmetic: write-time fan-out gives cheap reads and cannot survive a celebrity, read-time fan-out gives cheap writes and an unpredictable 200-way scatter-gather, and the hybrid bounds both. A cache hierarchy that earns its complexity because timelines and posts have opposite sharing profiles. Ranking as a two-stage system with a latency budget, and the pagination bug that ranking creates. The celebrity problem in detail, including the inversion where you push to caches rather than to timelines. Multi-region where the only thing that must not be stale is your own post. And deletion, where the lesson is that a denormalised view is filtered at read rather than corrected at write.


Requirements, and the Priority That Inverts

Functional:

  1. A home timeline: recent posts from accounts a user follows, ranked.
  2. Post and delete; posts may be public or restricted.
  3. Engagement counts (likes, replies) visible on each post.
  4. Blocks and mutes must be honoured.
  5. An indication that new content is available.

Non-functional:

Target
Read:write ratio~100:1
Feed load latencyp99 < 200 ms
Freshnessseconds to ~a minute is acceptable
Availability99.99% for reads
Correctness of countsapproximate is fine
Own-post visibilityimmediate, always

And the priority statement, which is the whole reason this capstone reads differently from the last one:

Availability and latency outrank consistency here. A feed that is thirty seconds stale is not a defect. A feed that takes two seconds to load, or fails, is. A like count that is off by three is invisible; a like count that costs a database write per like is an outage.

Module P-24 said the opposite — fail the payment rather than risk a double charge. Both are correct, and the discipline is to state which one you are building before choosing mechanisms, because nearly every decision below follows from that single line. The one exception is the last row: a user must always see their own post immediately, and that exception drives a specific piece of design.

Analogy: a newspaper's editorial process versus its printing. The bank's ledger from Module P-24 is double-entry bookkeeping — every entry balanced, nothing overwritten. A newspaper is the opposite: a hundred editions printed and distributed to newsagents in advance, each slightly different, none of them corrected once printed. If a story is pulled, you do not send drivers to retrieve every copy from every shop — you stop selling it and the next edition omits it. That is exactly the delete design below, and the reason is the same: the copies are cheap and disposable, and chasing them costs more than the error does.


Estimation: the Numbers That Decide the Architecture

QuantityValue
Registered users500 M
Daily active users200 M
Posts per DAU per day~0.5
Posts per second100 M/day ≈ 1,200/s average, ~5,000/s peak
Feed loads per DAU per day~15
Feed reads per second3 B/day ≈ 35,000/s average, ~100,000/s peak
Median follows~200
Largest account~100 M followers

Two derived numbers carry the design:

Write-time fan-out volume: 1,200 posts/s × 200 followers = 240,000 timeline writes/second on average, and around a million per second at peak. That is a very large number of very small writes — which is achievable, and it tells you the timeline store must be a cheap append-oriented structure, not a relational table with indexes.

The celebrity number: one post by a 100 M-follower account is 100 million timeline writes. At a sustained 500,000 writes/second that is 200 seconds — during which it consumes the entire fan-out capacity and every ordinary user's post is delayed behind it (Module A-8's head-of-line problem, at feed scale). This single number is why pure write-time fan-out is not an option, and it is worth computing early because it invalidates the simplest design outright.

Storage: posts are small (a few hundred bytes of text plus references), so 100 M/day is tens of gigabytes per day — trivial. Timelines are the volume: 200 M active users × 800 cached post IDs × ~16 bytes ≈ 2.5 TB of timeline data, which fits in a Redis cluster of a reasonable size and, crucially, is rebuildable — a property we will design for deliberately.


The Central Decision: Fan-Out on Write or on Read

text
On write (push)On read (pull)
Write costO(followers) — hugeO(1)
Read costO(1) list readO(followees) scatter-gather + merge
Read latencypredictable, lowvariable, tail-dominated
Wasted workfan-out to users who never log innone
Celebrity postcatastrophicfree
Storagea materialised list per usernone

For a 100:1 read-heavy workload, push is obviously right — you move work to the rare operation. Except for the celebrity row, and except that fanning out to 500 M registered users when only 200 M are active wastes 60% of the writes.

So the design is hybrid, and the rule is:

  • Push for ordinary accounts (below a follower threshold, say 100,000): their posts are written into follower timelines at post time.
  • Pull for large accounts: their posts are not fanned out. At read time, the feed service also fetches recent posts from the small set of large accounts this user follows, and merges.
  • Push only to active users (seen in the last ~30 days). Inactive users' timelines are rebuilt on next login from the pull path.

Why the hybrid bounds both sides: the push cost is now bounded by followers of ordinary accounts, which has a ceiling; the pull cost is bounded by how many large accounts one user follows, which is a handful. Neither side has an unbounded term, and that is the test of a good hybrid.

The threshold is a tuning parameter with a real trade: lower it and more accounts are pulled, making reads more expensive for everyone; raise it and a single post's fan-out gets larger and slower. Around 10⁵ followers is a defensible starting point, adjusted by measurement.


Data Model

Posts — the source of truth, immutable once written:

sql

A Snowflake-style ID matters here for two reasons from Module P-3: it is generated without coordination across shards, and it is time-sortable — so a timeline can be ordered and paginated by ID alone, with no secondary index and no timestamp comparison.

The follow graph — the hot metadata:

text

Both directions are needed: followees(user) for the pull path and the initial rebuild, followers(author) for fan-out. It is read constantly and changes slowly, which makes it ideal for aggressive caching.

Timelines — per user, capped, and holding IDs only:

text

Three deliberate properties. IDs, not content — otherwise an edit or a delete would need to reach millions of copies, and storage would be 100× larger. Capped, because nobody scrolls past a few hundred items and an uncapped list is an unbounded storage commitment. And ZADD keyed on the post ID is naturally idempotent — it is an absolute set-membership operation, not an increment — so a retried or duplicated fan-out job is harmless, which is Module P-16's absolute-versus-relative distinction paying off exactly where at-least-once delivery is guaranteed.

Counters — approximate by design. A like must never be UPDATE posts SET likes = likes + 1: that is a relative operation on a single hot row, which serialises every like on a popular post and produces a dead tuple per like (Modules P-16, P-24). Instead: append like events to a log, aggregate periodically into a counter cache, and accept that the displayed number lags by seconds. For the hottest posts, shard the counter across N keys and sum on read.


The Write Path

text

Sign in to keep reading

The rest of this module is free — sign in with Google to unlock it and track your progress.

Sign in & Register

Discussion

0

Join the discussion

Loading comments...

© 2026 Jatin Jain Saraf (JJS). All rights reserved.