Meta Feed/Reels Ranking Ecosystem Metrics

What's being tested
Ability to define, prioritize, and defend ranking metrics for a feed/reels recommender: choosing an OEC, balancing short- vs long-term objectives, handling bias and guardrails, and designing valid experiments.

Core knowledge

• Primary OEC candidates: watch time, completed-view rate, session length, DAU/MAU retention, creator engagement, and ad revenue uplift.
• Guardrail metrics: user reports, abusive content flags, content diversity, creator churn, advertiser KPI degradation.
• Position/exposure bias: correct with inverse propensity scoring (IPS) or randomized exposure; account for non-random impressions.
• Offline proxies: NDCG, AUC, predicted watch probability useful but must be validated against online uplift.
• Experiment design: user-level randomization, power calculations, holdouts, parallel experiments, and correction for cross-experiment interference.
• Long-term tradeoffs: optimize for retention and lifetime value, not only instantaneous watch-time; include delayed-treatment outcomes.
• Slice and heterogeneity analysis: evaluate by cohort, content type, device, region, and creator to detect distributional harms.

Worked example — "Design metrics to evaluate a new Reels ranking algorithm"
Start by naming stakeholders (end users, creators, advertisers, safety/ops) and propose a single OEC—e.g., per-user weekly watch-time uplift weighted by retention. List guardrails: safety (reports), creator distribution, ad revenue. Specify measurement: user-level randomized A/B test, at least X days for retention signal, pre-computed power to detect minimal detectable uplift. Finally, plan slice analyses (new users, heavy watchers, regions) and offline proxies to pre-filter models before rollout.

A common pitfall
The tempting approach is to optimize only for aggregate watch time because it’s easy to measure and moves fast. That often amplifies sensational content and hurts retention, creator diversity, and ad metrics. Ignoring position bias, failing to randomize exposures, or skipping guardrails produces short-term gains that reverse once feedback loops and creator behavior adapt.

Further reading

• Swaminathan, A. & Joachims, T., "Counterfactual Risk Minimization: Learning from Logged Bandit Feedback" (useful for IPS and offline evaluation).
• Li, L. et al., "A Contextual-Bandit Approach to Personalized News Article Recommendation" (practical bandit/online learning foundations).