Loading next page...
🎉 Launch Sale: Get 30% off annual plans with code LAUNCH30
A user with 50 followers posts a tweet. Your system writes 50 timeline entries — one into each follower's precomputed feed. Simple, fast, and the read path is a single sorted-set lookup. Now a celebrity with 20 million followers posts a tweet. Your system must write 20 million timeline entries. At peak, if 10 celebrities post within the same minute, that is 200 million timeline writes in 60 seconds — 3.3 million writes per second from just 10 tweets. Meanwhile, a normal user's tweet generating 50 writes waits in the same queue. The celebrity's fanout is starving the system.
This is the fundamental tension in feed systems: precompute the timeline (fast reads, expensive writes) or assemble it on demand (cheap writes, expensive reads). Neither extreme works at Twitter's scale. The answer is a hybrid — and the interesting part is where you draw the line and how you merge the two approaches at read time.
This post walks through designing a home timeline system from first principles. We will start with a correct but slow baseline, add fanout-on-write for normal users, introduce the hybrid celebrity strategy, and layer on ranking and caching. Every decision will be driven by the read/write asymmetry that defines this system: 1.85 million feed reads per second versus 46,000 tweets per second.
Let us begin.
The home timeline is the core product surface of Twitter. When a user opens the app, they expect to see recent tweets from people they follow, ordered by time (or relevance), loading in under 200 milliseconds. Behind that expectation is a system that must handle extreme read volume, asymmetric write amplification, and the celebrity skew problem.
In scope: tweet posting, follow graph, home timeline assembly, fanout strategy, caching, basic ranking.
Out of scope: search/discovery, trending topics, ads injection, direct messages, notifications.
Now we need to understand the asymmetry. The ratio of reads to writes determines whether we precompute or assemble on demand.
You reached the preview limit. Sign in to unlock the remaining sections.
Converting to per-second:
The read-to-write ratio is ~40:1. This is the defining characteristic: the system does 40× more reads than writes. Any architecture that makes reads expensive (like assembling the timeline from scratch on every request) will be crushed. Precomputation is not optional.
When a tweet is posted, fanout-on-write copies it into every follower's timeline. For normal users (median 50 followers), this is 50 writes per tweet — trivial. But the distribution is wildly skewed:
At peak, if full fanout-on-write applies to every tweet: 46,000 tweets/s × 300 average followers = ~13.8 million timeline writes per second. But this average masks the skew. A single celebrity tweet generates 20 million writes — equivalent to the write volume of 400,000 normal tweets. Ten celebrities posting in the same minute can overwhelm the entire fanout pipeline.
Each timeline entry stores a tweet reference: tweetId (8 bytes), authorId (8 bytes), scoreTs timestamp (8 bytes), sourceType flag (1 byte), plus Redis sorted set overhead per entry (~55 bytes for skiplist node, score, and member encoding). Total: ~80 bytes per timeline entry.
We limit each user's timeline to the most recent 800 entries (roughly 2-3 days of tweets for an active user following 300 accounts). Older entries fall off and are loaded from persistent storage on deep scroll.
With these numbers, let us define the data model.
A feed system has two fundamentally different data categories: source-of-truth data (tweets, follow edges — the canonical records that must be durable and correct) and serving-model data (precomputed timelines — derived, ephemeral, and optimized for read speed). Mixing these would be like storing both a database and its cache in the same system — you lose the ability to rebuild one from the other.
Tweet(tweetId, authorId, createdAt, text, mediaRef?, replyToId?)The canonical, immutable record of a tweet. Stored in the tweet store (durable, append-optimized). Every other representation — timeline entries, cached feeds, search indexes — is derived from this.
FollowEdge(followerId, followeeId, createdAt, isActive)The social graph. isActive supports soft-delete on unfollow — the edge is marked inactive rather than physically deleted, preserving history for audit and enabling fast reactivation. The follow graph drives two critical operations: fanout (who receives a new tweet?) and timeline assembly (whose tweets should appear in my feed?).
HomeTimeline(userId, tweetId, authorId, scoreTs, sourceType)The precomputed serving model — one entry per tweet per follower. This is what the feed service reads when a user opens the app. It lives in Redis (sorted by scoreTs for recency ordering) with persistent backing for durability.
sourceType distinguishes between PUSHED (fanout-on-write — the entry was placed here by a fanout worker) and PULLED (fanout-on-read — the entry was fetched at read time from a celebrity's tweet list). This distinction matters for cache invalidation: pushed entries are explicitly written and deleted; pulled entries are assembled fresh on each read and do not need invalidation.
Tweet is the durable record; fanout workers distribute it to HomeTimeline.FollowEdge controls who receives future tweets and whose past tweets appear.HomeTimeline is the precomputed serving layer; read path merges pushed entries with pulled celebrity tweets.These are v1 entities. After the hybrid fanout strategy is designed, the model will evolve.
{ "text": "Just setting up my twttr", "mediaRef": null }The response returns the tweetId and createdAt. The tweet is persisted to the tweet store synchronously (the author sees it immediately in their own profile). Fanout to followers' timelines happens asynchronously — the author does not wait for 50 (or 20 million) timeline writes to complete.
Follow creates a FollowEdge. The system then backfills the new followee's recent tweets into the follower's timeline (so they immediately see content from the newly followed account, not just future posts). Unfollow marks the edge as inactive and triggers background cleanup of the unfollowed account's tweets from the follower's timeline.
The core read endpoint. Returns a page of tweets from the user's home timeline, ordered by scoreTs (timestamp or ranking score).
cursor is an opaque token encoding the last-seen scoreTs — enabling efficient pagination without offset-based queries. Offset-based pagination (e.g., "skip 200, take 50") requires the system to read and discard 200 entries on every page — increasingly expensive as the user scrolls deeper. Cursor-based pagination reads directly from the cursor position: ZREVRANGEBYSCORE timeline:{userId} cursor-1 -inf LIMIT 0 50 in Redis. Constant cost regardless of page depth.
limit is bounded to 50 (configurable) to protect tail latency. Unbounded limits would allow a single request to read thousands of entries, spiking memory and serialization time.
With the API defined, we can build the architecture.
Let us start with the simplest correct implementation: no precomputation. When a user requests their home timeline, the feed service fetches their follow list, queries the tweet store for each followee's recent tweets, merge-sorts by time, and returns the top N.
┌──────────┐ ┌──────────────┐ ┌─────────────┐
│ User │────▶│ Feed Service │────▶│ Follow Graph│
│ │◀────│ │ │ (Postgres) │
└──────────┘ │ │────▶│ │
│ │ └─────────────┘
│ │ ┌─────────────┐
│ │────▶│ Tweet Store │
│ │ │ (Cassandra) │
└──────────────┘ └─────────────┘
Read: fetch followees → fetch recent tweets per followee → merge-sort → return top 50Tech decision: Postgres for the follow graph.
The follow graph needs two efficient queries: "who does user X follow?" (for timeline assembly) and "who follows user X?" (for fanout). These are relational edge queries — bidirectional lookups on a two-column table. Postgres handles this naturally with two indexes: (followerId, followeeId) and (followeeId, followerId).
Why not a graph database (Neo4j, etc.)? Our queries are simple edge lookups, not multi-hop graph traversals. A graph database adds operational complexity (cluster management, query language) for a capability we do not need. Postgres with proper indexing answers "who does X follow?" in under 1 ms for typical users with hundreds of followees.
Why not Cassandra for the follow graph? Cassandra excels at append-heavy workloads, but follow/unfollow requires atomic updates (mark edge inactive, enforce no-duplicate-follow constraint). These are relational guarantees that Postgres provides naturally and Cassandra requires application-level workarounds.
Tech decision: Cassandra for the tweet store.
Tweets are append-only, high-volume (46,000/s peak), and read by author+time range. This is the exact access pattern Cassandra is optimized for: partition by authorId, cluster by createdAt descending. "Get the latest 20 tweets from author X" is a single-partition scan.
Why not Postgres for tweets? At 46,000 writes/s peak, Postgres could handle it, but the tweet table will grow to 500M rows/day — 180 billion rows/year. Cassandra's horizontal scaling (add nodes, rebalance automatically) handles this growth more naturally than Postgres's sharding story.
What this step gives us: a correct timeline that always reflects current follow relationships and the latest tweets. But the read path is expensive: for a user following 300 accounts, every GET /v1/home requires 300 tweet store queries + a merge-sort. At 1.85M reads/s peak, that is 555 million tweet store queries per second — clearly unsustainable. We must precompute.
When a user posts a tweet, fanout workers push the tweet entry into each follower's precomputed timeline in Redis. The read path becomes a single Redis sorted-set query: ZREVRANGEBYSCORE timeline:{userId} +inf -inf LIMIT 0 50 — return the 50 most recent entries. No merge-sort, no follow graph lookup, no tweet store queries on the hot read path.
WRITE PATH:
┌──────────┐ ┌──────────────┐ ┌─────────────┐
│ Author │────▶│ Tweet Service │────▶│ Cassandra │ (persist tweet)
│ │ │ │────▶│ (tweets) │
└──────────┘ └──────┬───────┘ └─────────────┘
│ publish tweet event
▼
┌──────────────┐ ┌─────────────┐ ┌─────────────┐
│ Kafka │────▶│ Fanout │────▶│ Redis │
│ (tweet │ │ Workers │ │ (timelines) │
│ events) │ │ (~60 fleet) │ │ (128 shards)│
└──────────────┘ └─────────────┘ └─────────────┘
│ For each follower:
│ ZADD timeline:{followerId} scoreTs tweetId
READ PATH:
┌──────────┐ ┌──────────────┐ ┌─────────────┐
│ Reader │────▶│ Feed Service │────▶│ Redis │
│ │◀────│ (150 fleet) │◀────│ (timelines) │
└──────────┘ └──────────────┘ └─────────────┘
ZREVRANGEBYSCORE timeline:{userId} → top 50 tweetIds
→ batch fetch tweet content from Cassandra/cacheTech decision: Kafka for tweet event distribution.
When a tweet is persisted, the tweet service publishes an event to Kafka. Fanout workers consume these events and write timeline entries. Why Kafka? At 46,000 tweets/s peak, each triggering fanout to hundreds of followers, the total timeline write volume is in the millions per second. Kafka provides durable buffering between the tweet write and the fanout — if fanout workers fall behind during a spike, events queue in Kafka rather than being dropped. Kafka also enables replay: if a fanout worker crashes mid-batch, it resumes from the last committed offset.
Tech decision: Redis sorted sets for timeline storage.
Each user's timeline is a Redis sorted set where the score is the tweet's timestamp and the member is the tweetId. ZREVRANGEBYSCORE returns the most recent entries in O(log N + M) time (N = total entries, M = returned entries). For a timeline of 800 entries returning 50, this completes in microseconds.
Why Redis and not Memcached? Timeline operations require sorted insertion (ZADD), range queries (ZREVRANGEBYSCORE), and bounded trimming (ZREMRANGEBYRANK to cap at 800 entries). Memcached supports only get/set — it cannot maintain a sorted collection. We would need to read the entire timeline into the application, modify it, and write it back — a read-modify-write cycle that is both slower and prone to race conditions when multiple fanout workers write to the same user's timeline concurrently.
Fanout worker sizing: with the hybrid strategy (Step C), only non-celebrity tweets are fanned out. If we define the celebrity threshold at 100,000 followers, and 99.5% of tweets come from accounts below this threshold (the vast majority of the 500M tweets/day), effective fanout volume is approximately: 45,770 tweets/s peak × 50 median followers = ~2.3 million timeline writes/s. At ~40,000 ZADD operations per worker per second (accounting for batching and Redis pipeline latency), we need ~60 fanout workers at peak.
Timeline entry TTL and trimming: when a fanout worker pushes an entry, it also trims the sorted set to 800 entries (ZREMRANGEBYRANK timeline:{userId} 0 -801). This caps Redis memory and keeps the working set bounded. For users who scroll past 800 entries (rare), the feed service falls back to the tweet store for older content.
What this step gives us: blazing-fast reads. GET /v1/home is a single Redis sorted-set query plus a batch content fetch. At 1.85M reads/s peak, the Redis sharded cluster handles the load comfortably. But we have a problem: what happens when a celebrity posts?
A user with 20 million followers posts a tweet. If we fan out to all followers, that is 20 million ZADD operations in Redis — generating as much write traffic as 400,000 normal tweets. The fanout workers backlog. Normal users' tweets are delayed in the Kafka queue behind the celebrity fanout. Feed freshness degrades for everyone.
The solution: do not fan out celebrity tweets at all. Store them in the tweet store like any tweet, and merge them into the timeline at read time for users who follow that celebrity.
The celebrity threshold: we classify an account as "high-fanout" when their follower count exceeds 100,000. This threshold captures approximately the top 0.5% of accounts by follower count — but these accounts produce a disproportionate share of total fanout volume because their tweets each generate 100,000+ writes. By excluding them from fanout-on-write, we reduce peak write amplification from 13.8M/s to ~2.3M/s — an 83% reduction.
Why 100,000 and not 1 million? At 1 million, only a few hundred accounts are excluded, and accounts with 500,000 followers still generate enormous fanout spikes. At 100,000, we capture the heavy tail while keeping the vast majority of accounts (99.5%) on the fast fanout-on-write path. The threshold can be dynamic — monitored and adjusted based on observed fanout queue depth.
How the read path merges pushed and pulled tweets:
When a user requests their home timeline, the feed service executes two operations in parallel:
ZREVRANGEBYSCORE timeline:{userId} +inf cursor LIMIT 0 50 from Redis. These are precomputed entries from non-celebrity followees. Latency: ~1 ms.scoreTs, return the top 50. The merge is an in-memory operation on ~50 + 8×5 = ~90 entries — negligible CPU.READ PATH (hybrid):
┌──────────────┐
│ Feed Service │
│ │──1──▶ Redis: get pushed entries (50 most recent)
│ │──2──▶ Cassandra: get celebrity tweets (parallel, 8 queries)
│ │──3──▶ Merge-sort pushed + pulled entries
│ │──4──▶ (optional) Ranking service enrichment
│ │──5──▶ Return top 50 to client
└──────────────┘Total read latency: Redis (1 ms) + parallel Cassandra queries (5-10 ms, dominated by the slowest) + merge (0.1 ms) = ~7-12 ms for the data fetch. Well within the 200 ms p95 budget, leaving room for optional ranking enrichment.
What if a user follows 50 celebrities? The parallel Cassandra queries scale linearly — 50 queries at ~2 ms each, parallelized with a concurrency limit of 10, complete in ~10-15 ms. Still within budget, but approaching the comfort zone. For users following an extreme number of celebrities (>100), we cap the pull count at the most-recently-active 20 celebrities and rely on pushed entries for the rest. In practice, very few users follow more than 20-30 celebrities, so this cap rarely activates.
What this step gives us: a timeline that is fast for reads (most entries are precomputed in Redis), manageable for writes (celebrity tweets are not fanned out), and correct (pulled entries are always fresh from the source). But we have not addressed ranking, caching, or failure modes. That is next.
The timeline from Step C is sorted by recency — newest first. This is the "Following" feed: a chronological stream of tweets from accounts you follow. Modern Twitter/X also has a "For You" feed: an ML-ranked feed that includes tweets from accounts you don't follow, surfaced based on engagement signals and inferred interests. Our architecture must support both, because they have fundamentally different data sources and serving characteristics.
The "Following" feed is what we have built so far — hybrid fanout + merge, sorted by time. It only contains tweets from followed accounts. This is the simpler product: the candidate set is defined by the follow graph, and the ordering is chronological with optional engagement-weighted reranking.
The "For You" feed is architecturally different. Its candidate set extends beyond the follow graph: tweets liked by people you follow, tweets trending in your interest graph, tweets from accounts similar to ones you follow, and promoted content. This requires a candidate generation layer that sources tweets from outside the follow graph — topic-based indexes, engagement-propagation signals ("Alice, who you follow, liked this tweet"), and trending/viral content pools. The candidate generator produces ~500-1,000 candidate tweets, which the ranking model scores and the top 50 are returned.
Designing the full "For You" candidate generation pipeline is a substantial system on its own (interest graph modeling, embedding-based similarity, trending detection). For this blog, we focus on the "Following" feed architecture with a ranking layer that also serves as the scoring component for "For You" candidates. The key point: the ranking service is shared between both feeds, but the candidate sources differ.
How the ranking model works:
The ranking service scores each candidate tweet using a combination of features, producing a relevance score that replaces pure chronological ordering:
Engagement signals are the strongest features. A tweet with 50,000 likes and 10,000 retweets is almost certainly more interesting than a tweet with 3 likes, even if the 3-like tweet is newer. The TweetEngagement entity (likes, retweets, replies) provides these counts, updated via a streaming pipeline with ~30-second freshness. For brand-new tweets with zero engagement (the cold-start problem), the model falls back to author-level priors: if the author's recent tweets average 5,000 likes, the new tweet inherits a prior engagement estimate until real signals accumulate.
Social graph proximity measures how closely connected the reader is to the tweet's author. A tweet from someone you interact with frequently (reply to, like, retweet) ranks higher than a tweet from someone you follow but never engage with. This feature is precomputed: a background job calculates pairwise interaction scores between each user and their followees, stored in a lightweight feature table and refreshed daily. For 100M users with ~300 followees each, this is ~30 billion feature rows — but only the active subset (users who opened the app today) needs to be warm in the feature store.
Content type and recency decay ensure that diverse content surfaces (not just the most-liked text tweets) and that older tweets decay in relevance. A tweet from 2 hours ago with 10,000 likes might rank similarly to a tweet from 5 minutes ago with 500 likes — the recency decay function balances freshness against proven engagement.
User-specific preferences adjust the model per reader: if a user historically engages more with image tweets, image tweets get a boost. If they prefer political content or sports content, topic signals adjust scoring. These are stored in UserFeedConfig and loaded per-request from a cached feature store.
The ranking call path: the feed service assembles ~100-200 candidate tweets (50 pushed + pulled celebrity tweets), sends them to the ranking service with the user's feature context, and receives scored results within the 50 ms budget. The ranking service loads precomputed features from a Redis-backed feature store (engagement counts, interaction scores, user preferences) and evaluates a lightweight model (typically a gradient-boosted tree or a small neural network). At 1.85M feed reads/s, the ranking service fleet must handle the same QPS — approximately 200 ranking instances at peak, each scoring ~100-200 tweets per request in under 50 ms.
Cold-start for new tweets: when a tweet is just posted, it has zero engagement data. The ranking model cannot rely on likes/retweets. The solution is a two-phase approach: for the first 30-60 minutes, new tweets receive a "novelty boost" in the ranking score, ensuring they are shown to a sample of followers. As engagement signals accumulate (the streaming pipeline updates TweetEngagement within ~30 seconds of each like/retweet), the model transitions from the novelty prior to real engagement features. This ensures new tweets are not buried behind older high-engagement tweets, while still allowing the model to demote tweets that receive no engagement after exposure.
Latency budget: ranking must complete within 50 ms (leaving 150 ms of the 200 ms budget for data fetch + serialization + network). If the ranking service exceeds its budget, the circuit breaker opens and the feed service returns recency-ordered results.
Degradation tiers:
Content cache: after the feed service fetches tweetId references from Redis, it needs the actual tweet content (text, author name, media URLs). Rather than hitting Cassandra for every tweet, a content cache (Redis or Memcached, separate from the timeline cache) stores recently-fetched tweet objects. At a 95% cache hit rate, the feed service makes ~2.5 Cassandra reads per request instead of 50.
FINAL ARCHITECTURE:
WRITE PATH:
Author ──▶ Tweet Service ──▶ Cassandra (persist)
│
▼ (Kafka)
Fanout Workers (60 fleet)
│
▼ (for non-celebrity tweets only)
Redis Timelines (128 shards, 6.4 TB)
READ PATH ("Following" feed):
Reader ──▶ Feed Service (150 fleet)
├──▶ Redis: pushed entries
├──▶ Cassandra: celebrity tweets (parallel)
├──▶ Content Cache: tweet objects
└──▶ Ranking Service (200 fleet, 50ms budget)
├──▶ Feature Store (Redis): engagement, interaction scores
└──▶ Model: score + sort top 50
│
▼
Merged, ranked, paginated response
READ PATH ("For You" feed — candidate generation out of scope):
Reader ──▶ Candidate Generator
├──▶ Interest graph: tweets liked by connections
├──▶ Trending pool: viral content
├──▶ Topic index: interest-matched tweets
└──▶ Ranking Service (same as above, shared)Let us trace two tweets: one from a normal user, one from a celebrity.
Normal user (Alice, 200 followers) posts "Great morning!":
POST /v1/tweets. Tweet service persists to Cassandra with tweetId: t-001, authorId: alice. Returns immediately.{tweetId: t-001, authorId: alice} to Kafka.ZADD timeline:{followerId} 1710412345 t-001. Also trims each timeline to 800 entries. 200 Redis writes.GET /v1/home?limit=50. Feed service reads ZREVRANGEBYSCORE timeline:bob +inf -inf LIMIT 0 50 from Redis. Alice's tweet is in the result. Feed service batch-fetches tweet content from the content cache (or Cassandra on miss). Optional ranking enrichment. Returns 50 tweets in ~15 ms.Celebrity (Taylor, 20 million followers) posts "New album out Friday!":
POST /v1/tweets. Tweet service persists to Cassandra with tweetId: t-002, authorId: taylor. Returns immediately.GET /v1/home?limit=50. Feed service reads pushed entries from Redis (includes Alice's tweet). Separately, fetches Taylor's latest tweets from Cassandra (Bob follows Taylor, and Taylor is flagged as a celebrity). Merges the two sources, sorts by time. Both Alice's and Taylor's tweets appear in Bob's feed. Response in ~12 ms.No 20 million writes. No fanout queue backlog. Normal users' tweets are not delayed. The system stays healthy.
Now the data model needs to reflect the hybrid strategy.
Tweet(tweetId, authorId, createdAt, text, mediaRef?, replyToId?)
FollowEdge(followerId, followeeId, createdAt, isActive)
HomeTimeline(userId, tweetId, authorId, scoreTs, sourceType)
UserFeedConfig(userId, rankingMode, contentPrefs, language)
TweetEngagement(tweetId, likeCount, retweetCount, replyCount, updatedAt)
AuthorProfile(authorId, followerCount, isCelebrity, lastTweetTs)HomeTimeline gained sourceType. Entries are either PUSHED (written by fanout workers for normal accounts) or PULLED (fetched at read time for celebrity accounts). This distinction drives cache invalidation: pushed entries are explicitly managed; pulled entries are ephemeral.
AuthorProfile is new. The feed service needs to quickly determine which of a user's followees are celebrities (to decide what to pull at read time). AuthorProfile caches the follower count and celebrity flag, updated whenever the follower count crosses the threshold.
TweetEngagement is new. The ranking service needs engagement features. Rather than querying the engagement service on every feed request, we materialize engagement counts into a lightweight entity that is refreshed every few minutes.
UserFeedConfig is new. Supports per-user ranking preferences (chronological vs ranked, content filters, language).
A power user follows 500 normal accounts and 30 celebrities. They open the app and request their home timeline. The feed service must merge two data sources: 50 pushed entries from Redis and recent tweets from 30 celebrity accounts in Cassandra. What happens?
The parallel celebrity fetch: the feed service sends 30 Cassandra queries in parallel, capped at a concurrency of 10. Each query fetches the latest 5 tweets from one celebrity author (a single-partition scan — fast). At ~2 ms per query with concurrency 10, the 30 queries complete in 3 batches: ~6 ms total. Combined with the Redis read (1 ms, in parallel), total data fetch: ~7 ms.
The merge: the service now has 50 pushed entries + 150 pulled entries (30 celebrities × 5 tweets). It sorts all 200 entries by scoreTs and returns the top 50. Sorting 200 items is sub-microsecond. Total merge cost: negligible.
But what if celebrities are extremely prolific? If a celebrity posts 50 tweets in one hour, and the user follows 30 such celebrities, the pull phase fetches 1,500 entries. Sorting 1,550 entries is still fast (microseconds), but the Cassandra reads now return more data per query. We cap the pull to 5 tweets per celebrity per read request to bound this: the feed is "recent tweets," not "complete history." If the user scrolls past the first page, subsequent cursor-based reads fetch older entries.
Edge case: user follows only celebrities. Their Redis timeline is empty (no fanout-on-write entries). The feed is assembled entirely from Cassandra pulls. If they follow 100 celebrities, that is 100 parallel queries — still completable in ~20 ms with concurrency limiting. The feed works correctly, just slightly slower than a user with a mix of pushed and pulled entries.
A user posts a tweet, it fans out to 200 followers' timelines, and then the user deletes it. Those 200 Redis entries now reference a tweet that no longer exists. If the feed service fetches these entries, it returns 404s from the content cache.
The naive approach: let deleted tweets appear until they naturally age out of the timeline (800-entry cap). Users see "This tweet has been deleted" placeholders for hours. Bad user experience, and potentially violating content moderation requirements.
The good approach: lazy deletion on read. The feed service, when batch-fetching tweet content, detects that a tweet is deleted (the content cache returns a tombstone). It filters the deleted tweet from the response and asynchronously removes the entry from the user's Redis timeline. Over time, all 200 entries are cleaned up by their respective followers' reads. Latency impact: one extra Redis ZREM per deleted tweet per read — negligible.
The great approach: eager deletion via event-driven cleanup. When a tweet is deleted, the tweet service publishes a delete event to Kafka. Deletion workers consume the event, look up the author's follower list, and issue ZREM timeline:{followerId} tweetId for each follower. This proactively removes the deleted tweet from all timelines within seconds.
For celebrity deletions (no fanout-on-write happened), no cleanup is needed — the deleted tweet simply will not appear in Cassandra queries at read time (assuming a tombstone or soft-delete flag).
The tradeoff: eager deletion adds write volume (200 ZREM operations per delete for a normal user). At ~50,000 tweet deletions per day (estimated 10% of tweets), this adds ~15 million Redis operations/day — a rounding error compared to the billions of timeline reads.
A user follows @sports_news at 10:00 AM. At 10:15 AM, they unfollow @sports_news. At 10:20 AM, they open their feed. Should they see @sports_news tweets from 10:05 AM? No — they unfollowed at 10:15.
The problem with precomputed timelines: the 10:05 AM tweet was already fanned out to the user's Redis timeline at 10:05. The unfollow at 10:15 marked the FollowEdge as inactive, but the fanout entry was already written. Without cleanup, the user sees tweets from an account they intentionally unfollowed.
Layer 1: background cleanup. When an unfollow occurs, an async worker removes all entries from the unfollowed author from the user's Redis timeline: ZRANGEBYSCORE to find entries by that authorId, then ZREM each one. This typically completes within seconds.
Layer 2: read-time safety filter. Even before the background cleanup runs, the feed service applies a safety check: for each tweet in the response, verify that the user still follows the author (using the cached follow graph). If the follow edge is inactive, filter the tweet from the response. This adds a lightweight check (a set lookup against the user's cached follow list — typically ~300 entries, cached in-process) to each read request. Cost: sub-millisecond.
The two-layer approach provides both speed (background cleanup prevents stale data from accumulating) and correctness (safety filter guarantees zero stale entries even before cleanup completes).
A new user signs up and follows 50 accounts. Their Redis timeline is empty — no fanout has happened because these follows just occurred. The user opens the app and sees a blank feed. Terrible first impression.
The backfill mechanism: when a follow is created, the system triggers a backfill for the new followee: fetch their latest 20 tweets from Cassandra and insert them into the follower's Redis timeline. For 50 new follows, this is 50 Cassandra queries (parallelized) + 1,000 Redis ZADD operations. At ~10 ms for the Cassandra reads and ~5 ms for the Redis writes, the backfill completes in ~15 ms — fast enough to happen synchronously before the first feed load, or asynchronously with the first feed load using pulled entries as fallback.
For celebrity follows (no fanout), the backfill is not needed — the read path will pull their tweets directly from Cassandra.
The ranking service's feature store has a latency spike — p95 goes from 20 ms to 500 ms during a deployment. If the feed service waits for ranking on every request, feed latency jumps from 15 ms to 500+ ms. Users see a spinner. Engagement drops.
The 50 ms circuit breaker: the feed service sets a hard timeout of 50 ms on the ranking call. If the ranking service exceeds this, the request falls back to recency ordering. The user gets a chronological feed — not as engaging, but it loads in under 15 ms.
Degradation tiers in action:
The user experience degrades from "personalized" to "chronological" — not from "working" to "broken." This is the key design principle: ranking is enrichment, not a dependency.
We started with a read path that made 300 database queries per feed request. Let us look at what we built and why:
| Step | Problem Solved | Components Added | Number That Forced It |
|---|---|---|---|
| A | Correct baseline | Tweet Store (Cassandra) + Follow Graph (Postgres) + Feed Service | 1.85M reads/s × 300 followees = 555M queries/s (unsustainable) |
| B | Fast reads via precomputation | Kafka + Fanout Workers (60) + Redis Timelines (128 shards) | 1.85M reads/s needs O(1) read path, not O(followees) |
| C | Celebrity write explosion | Hybrid fanout (skip >100K followers) + read-time merge | 20M writes per celebrity tweet; 83% write reduction with hybrid |
| D | Ranking + "Following"/"For You" split + resilience | Ranking Service (200 fleet) + Feature Store + Content Cache + Circuit Breaker | 200ms p95 SLA; cold-start for new tweets; ranking as enrichment not dependency |
| Entity | In v1? | In v2? | What Triggered the Change |
|---|---|---|---|
| HomeTimeline | Yes | Yes (+ sourceType) | Hybrid fanout needs pushed/pulled distinction |
| AuthorProfile | No | Yes | Feed service needs celebrity flag for read-time pull decisions |
| TweetEngagement | No | Yes | Ranking service needs engagement features |
| UserFeedConfig | No | Yes | Per-user ranking preferences |
Picking only one fanout model. Full fanout-on-write breaks on celebrities. Full fanout-on-read breaks on read volume. The hybrid is not optional at this scale — it is the only strategy that handles both extremes.
No celebrity threshold definition. "We handle celebrities differently" is a hand-wave. "Accounts with >100,000 followers are excluded from fanout-on-write, reducing peak write amplification from 13.8M/s to 2.3M/s" is a design decision.
No cursor-based pagination. Offset pagination degrades linearly with depth. At 1.85M reads/s, even a small percentage of deep-scroll requests creates significant load.
Coupling ranking to feed availability. If your ranking service going down means the feed goes down, you have made the system less reliable than having no ranking at all.
No deletion or unfollow propagation. Precomputed timelines are stale by nature. Without explicit cleanup and read-time safety filters, users see tweets from unfollowed accounts and deleted content.
Did you quantify the read/write asymmetry? 40:1 read-to-write ratio → precomputation is mandatory. This single number should drive your entire architecture.
Did you recognize and solve celebrity write amplification? The hybrid fanout strategy, with a specific threshold and the read-time merge algorithm, is the core intellectual contribution of this design.
Did you explain the merge algorithm? Pushed entries from Redis + pulled celebrity tweets from Cassandra, merged in memory, with latency math showing it fits within 200 ms. The mechanism, not just the concept.
Did you design a fallback path? Ranking is enrichment, not a dependency. The circuit breaker with degradation tiers shows mature reliability thinking.
A home timeline is the first thing a user sees when they open the app. It must load fast, show fresh content, and never display tweets from accounts the user unfollowed. Behind that simple experience is a hybrid fanout system that writes timeline entries for 99.5% of tweets and merges on read for the other 0.5% — and that 0.5% accounts for the most followed accounts on the platform.
We started with a brute-force read path making 300 queries per request. We ended with a precomputed serving layer, a hybrid write strategy that reduces write amplification by 83%, a parallel merge algorithm that assembles celebrity tweets in 7 milliseconds, and a ranking enrichment layer that degrades gracefully. Not because we drew the final architecture first, but because each problem — read volume, write amplification, celebrity skew, ranking fragility — demanded exactly one targeted solution.
The same discipline, one more time: start simple, let numbers force decisions, and always know what breaks first.
Social Media
PRODesign a Tinder-like Dating Platform
It is 8:47 PM on a Friday night. Alex in San Francisco opens the app and starts swiping. At 8:47:32.445 PM, Alex swipes right on Jordan. At that exact same millisecond — 8:47:32.445 PM — Jordan in ...
Social Media
PRODesign a Strava-like Fitness Platform
It is 6:47 AM. Sarah just finished her morning run through downtown San Francisco. Her Garmin syncs to the app. The screen shows: 10.2 km, 52:18 moving time, 312 meters elevation gain. She checks t...
Social Media
PRODesign a Google News Feed Platform
It is 3:47 PM on a Tuesday. A major tech company announces unexpected layoffs. Within 90 seconds, 2.3 million users open Google News expecting to see the story. The wire services published 47 versi...