Evaluate impact of short videos in feed
Company: Meta
Role: Data Engineer
Category: System Design
Difficulty: hard
Interview Round: Onsite
## Scenario
You work on a social app’s main **News Feed**. The team wants to introduce a **short-form video module ("Reels")** into the feed.
## Prompt
1. How would you determine whether adding Reels to the News Feed is **successful**?
2. What **metrics** would you define (primary + secondary/guardrails)?
3. How would you design an **experiment / launch plan** to measure impact?
## Notes / Assumptions
- The feed currently contains mixed content (text, images, long videos, links).
- Reels may affect both **consumption** (viewing) and **distribution** (other content getting less exposure).
- You have standard event logging for impressions, clicks, watch time, likes, comments, shares, hides, follows, etc.
Overview: This question evaluates competency in metrics definition, causal experimentation, product analytics, and feed system behavior within the System Design and Data Engineering domain.
Read the full Meta Data Engineer interview experience this question came from
Community answers
Answer by boz23004
I would first clarify the goal: adding Reels should make the News Feed more valuable and engaging, encouraging users to return. High Reels usage alone would not establish success, because it could simply replace engagement with existing content.
For the primary metric, I would use seven-day retention among eligible News Feed users: the percentage who return to the feed seven days after their initial experiment exposure. This measures whether the experience creates lasting value.
For secondary metrics, I would track:
Total meaningful engagement per eligible user across the entire feed, including shares, comments, and likes.
Total feed time per user, with Reels watch time reported separately.
Reels adoption, repeat viewing, completion rate, and quick-skip rate.
My guardrails would include hides, reports, and ‘not interested’ actions per impression, plus feed latency and playback failures. I would also monitor impressions and engagement for existing content types, and how exposure is distributed across creators. Some substitution is expected, but I would investigate whether Reels displaces valuable interactions or substantially concentrates distribution.
To measure impact, I would run a randomized A/B test at the user level, with persistent assignment. The control group would keep the existing feed, while treatment would receive a clearly defined Reels module with fixed placement and frequency. I would analyze all assigned eligible users, including those who never watch Reels, t