Designing a News Feed: Who Pays for the Delivery?

One post can trigger millions of database writes. Here is how to build a hybrid architecture that survives the blast radius.

(One megaphone, and a shadow that splits into millions — a single post landing in every follower timeline the moment it publishes.)

---

500 followers and 50 million, same button

Your feed has to handle a publish from an account with 500 followers exactly the same way it handles one with 50 million. You can make the write path cheap, or you can make the read path cheap. You cannot do both, for every user, at once.

Either way, the post has to reach everyone who follows the author. That part is fixed. What you actually get to decide is who pays for the delivery — the writer, or the reader.

On a previous project, we decided the writer would pay. That worked fine right up until the first large account posted. The fan-out queue backed up behind that single write, and every ordinary user's timeline update sat in line behind a celebrity.

Delivery and ranking are two different jobs

Delivery means getting a post in front of a follower quickly, while it is still fresh. Ranking is the other half of the problem: out of everything a reader could see right now, which posts are worth putting on the screen, decided inside a tight time budget.

You need ranking at any scale. Celebrities just make both jobs harder at the same time, and most of the design ends up bending around that.

People open their feed constantly and post rarely, so reads outnumber writes by a wide margin. A reader feels read latency on every single scroll. Write cost is quieter — it turns up on the monthly infrastructure bill, where nobody is sitting and watching a spinner.

Fan-out on write vs fan-out on read

Fan-out on write copies the post ID into every follower's timeline the moment you publish, so each read is a cheap lookup. Reads become trivial, which feels like good engineering. Most teams start here, us included.

Fan-out on read is the mirror image. The post is written once to the author's own outbox — the author's personal, chronological list of everything they've posted — and each timeline is assembled live, pulling from everyone the reader follows. Writes stay cheap. Now every scroll uses a scatter-gather pattern: firing a fan of parallel requests out to hundreds of outboxes, then stitching all the results back together.

Whichever you pick, you're really just choosing which side gets the bill. Write fan-out's cost scales with the size of the author's audience. Read fan-out's cost scales with how many accounts each reader follows. Neither one survives a lopsided social graph. Write fan-out falls apart when a single account has millions of followers. Read fan-out falls apart when one person follows thousands of accounts.

The celebrity problem: one post, millions of writes

The first thing we got right was the data model, and it quietly saved us later. Every post gets a time-sorted ID (like Twitter's Snowflake architecture): a 64-bit integer that climbs with time. Sorting a timeline becomes sorting integers, with no timestamp column involved at all — as long as the ID generators don't drift out of sync across regions.

That choice paid off a second time at read time. Because the IDs are time-ordered, a reader's precomputed timeline and the outboxes of the few celebrities they follow can be combined with a plain k-way merge: walk several already-sorted lists at once and repeatedly take the highest ID off the front. The timeline itself stores only post IDs, never the bodies — those get hydrated (fetched from the post store) at read time — so the timeline stays a cheap list of numbers.

In production we never ran fan-out inside the request. Publishing dropped a job on a queue, and a pool of workers did the pushing, with retries and backpressure (slowing the producer down when the queue starts falling behind) so a burst couldn't take the write path down with it. That was fine for ordinary accounts.

Then a large account posted, and I watched one person's fan-out turn into millions of cache writes while every normal post queued up behind it. It was nowhere near the scale of the Twitter fan-out stories everyone quotes, but it was the same shape of failure, and it cost us an afternoon.

(The two base strategies. You are not choosing which is faster — you are choosing whether the write path or the read path absorbs the cost of delivery.)

So we split the graph by follower count, and that split is the whole trick. Below a threshold, an account fans out on write like everyone else. Above it, we skip the follower fan-out entirely and let readers pull that account's posts live.

The one rule we never broke: every post lands in the author's own outbox first, because the outbox is the source of truth. Fan-out is only an optimisation layered on top of it.

In pseudo-code, that hybrid write path looks roughly like this:

def fan_out(user_id, post_id):

author_outbox.append(user_id, post_id)   # source of truth: every post lands here first

if follower_count(user_id) >= CELEBRITY_THRESHOLD:

return                               # celebrities skip fan-out; readers pull them live

normal accounts: a worker runs this loop off the fan-out queue

for follower_id in get_followers(user_id):

timeline_cache.push(follower_id, post_id)   # one write per follower

(Read-time merge. Normal accounts were pushed to you; the few huge accounts you follow are pulled live and merged by ID, so no single post ever fans out to millions.)

That threshold reaches into everything downstream — the queues, the read path, the cache layout, even pagination — so it is not a value you casually tune on a Friday. Two distinct failure modes bit us.

1. Accounts hovering at the line.

They flipped between push and pull on every single post. The fix was hysteresis: promote an account to pull-only once it crosses the threshold, but only demote it again once it falls well below.

2. Pagination.

Merging a pushed timeline with a live celebrity pull means the cursor (the marker tracking what page you are on) has to hold a position in both sources, not just the last post ID it served. Get it wrong and the failure is completely silent. If a celebrity's outbox gains three posts between page one and page two, a naive cursor walks straight past them — and nobody files a bug for posts they never knew existed.

Ranking a news feed inside 300ms

Ranking decides whether the feed is worth opening at all, and it matters as much as delivery. A fast feed full of things you don't care about fails just as hard as a slow one. So ranking gets a real budget rather than whatever time happens to be left over. Instagram dropped the chronological feed back in 2016, and Twitter open-sourced its ranking stack in 2023.

The whole pipeline has to land inside a 300ms p99 — meaning even the slowest one percent of requests come back within 300 milliseconds. That window is tighter than it sounds.

When someone opens the feed, you pull around 1,000 candidate post IDs from the merged timeline, plus a few injected recommendations, sized to the fetch budget. A cheap, stateless filter cuts that down: it dedupes using a Bloom filter (a compact structure that answers "have I already shown this?" quickly, at the cost of an occasional false positive) and drops anything older than a week, leaving a few hundred. That filter also runs a read-time eligibility check — remember that, because it's where I later shipped a bug. Whatever survives, maybe 100 posts, goes to a heavy ranker, usually a two-tower machine learning model: one side creates a mathematical profile of the user, the other creates a profile of the post, and comparing the two (the dot product) gives you the relevance score. The top 20 come back.

(The ranking funnel and its budget. Every narrowing stage runs inside one 300ms p99 window, with a chronological branch for the moment the ranker times out.)

You don't hit a 300ms p99 by accident. You buy it, stage by stage. On a good day the budget might split something like this:

• 50ms to fetch candidates
• 40ms to filter
• 120ms in the ranker
• 60ms to hydrate post bodies
• 30ms of margin for the bad days

Your numbers will differ. Three things keep the budget honest:

• The post-side profiles (embeddings) are precomputed, so at request time the ranker only has to build the profile for the user.
• Every stage carries its own timeout, and that deadline propagates upward. A hung candidate fetch kills the fetch alone, not the whole window.
• A cheap first-pass scorer runs before the heavy model, so the expensive network only ever sees the finalists.

The eligibility check, and the bug I put in it

Which brings me back to that eligibility check. I got this one wrong, and it came out of a decision I still think was correct: the timeline is just a cache of post IDs.

Delete a post, or block a user, after fan-out has already run, and the underlying row disappears while the ID keeps sitting in millions of cached timelines. Filtering at write time can't help you, because the write already happened. I hadn't wired up a read-time re-check, so the feed carried on serving IDs for content that no longer existed. It stayed invisible until someone was staring at a post that should have been gone, and by then the dead ID was everywhere — there was no single row to go and delete.

The fix has two halves:

• Tombstones. A deletion writes a tombstone (a marker recording that the post is gone, instead of just removing the data), and the merge step checks for it, dropping any ID carrying one before it ever reaches ranking.
• Read-time block checks. Blocks get checked at read time against a carefully cached block list.

When the ranker times out

The ranker will time out eventually. A model server stalls, a feature store hangs. The fallback has to exist before you need it, not during the incident: if ranking throws, sort the candidates by ID and ship a chronological feed.

try:

feed = heavy_ranker.rank(candidates, per_stage_deadline)

except (Timeout, RankerError):

Snowflake IDs sort by time, so descending order gives a fresh-first chronological feed

feed = sorted(candidates, key=lambda c: c.post_id, reverse=True)

When to walk away from the hybrid model

Before writing any delivery code, I'd get honest about four questions:

• What is the real read-to-write ratio? Is it lopsided enough that precomputing timelines actually pays for the extra writes?
• Does the graph have a genuine heavy tail — a few accounts that dwarf the rest — or am I building for celebrities I don't have?
• Will the ranker fit inside the read budget with headroom on a bad day, not just a good one?
• When the ranker times out, which it will, what happens? Does a chronological fallback take over, or does the reader get a spinner?

That last one matters most, because without a fallback the ranking model's worst day becomes your feed's worst day.

The honest answer for most teams: if you're a small network with modest read traffic and nobody with a runaway following, plain fan-out on read is enough, and you should stop right there. The hybrid only earns its keep once the tail is real and reads clearly outweigh writes. Everything past that point is cost you've chosen to take on.

The tax you sign up for

Say you build it anyway. A read that used to be one cache GET now does considerably more work. It:

• merges two sources
• re-checks eligibility
• hydrates the post bodies
• calls the ranker
• falls back when the ranker dies

That's five places a page can come from, where there used to be one. Meanwhile the threshold goes stale as your biggest accounts keep growing. And the two delivery paths drift apart at the seam where they meet: the integer merge keeps the ordering clean, so what actually drifts is subtler — duplicates and gaps, wherever the live pull's window overlaps or misses the pushed cache.

So match the machinery to your graph, and then own it. Before launch, put a named on-call owner against each of:

• threshold drift
• the ranker's p99
• the eligibility filter
• fallback health

Those four hold your pager for the life of the product. There's no version of this where you ship the hybrid and walk away from it — someone will be tending that threshold for years.

---

Thank you for reading. If you found this useful, follow PracHub for more practical deep dives on engineering, system design, and interview preparation written to help you build skills that last beyond a single article.