Evaluate Home-Feed Diversity's Impact on User Engagement Metrics
You run a personalized home feed where each post is tagged with one or more topics, such as animals, recipes, travel, or fashion. Product leadership wants to increase topical diversity without harming relevance.
Constraints & Assumptions
-
Treat diversity as a measurable feed-quality intervention, not as a vague goal.
-
Assume topic labels, ranking scores, impressions, clicks, hides, saves, sessions, retention, and experiment infrastructure are available.
-
A more diverse feed can improve discovery and reduce fatigue, but can also reduce short-term relevance if pushed too far.
-
Include both diversity metrics and user/business outcome metrics.
Clarifying Questions to Ask
-
What level of topic taxonomy is used, and can posts have multiple topics?
-
Is the goal to increase diversity within a session, across days, or across a user's long-term exposure?
-
What is the current ranking objective, and where would diversity be added?
-
Are there important creator, advertiser, or content-quality constraints?
Part 1 - Explain Business Benefits
What business benefits can come from increasing home-feed diversity?
What This Part Should Cover
-
Reduced fatigue, broader discovery, long-term retention, better preference learning, healthier creator/topic ecosystems, and monetization resilience.
-
The trade-off between novelty and relevance.
-
The possibility that benefits appear over longer horizons rather than immediate clicks.
Part 2 - Define Diversity Metrics
Propose several quantitative metrics to measure diversity of a user's feed. For each metric, define it and discuss possible biases or limitations.
What This Part Should Cover
-
Metrics such as unique topics per session, entropy, Herfindahl-Hirschman concentration, same-topic run length, topic coverage, creator diversity, and distance between adjacent items.
-
Biases from topic taxonomy granularity, multi-label posts, impression position, user interest breadth, and supply availability.
-
Segment-level interpretation so niche-interest users are not penalized for having concentrated preferences.
Part 3 - Analyze a Specific Metric
Consider the metric "percentage of posts from the same topic." What bias does it carry?
What This Part Should Cover
-
Dependence on the denominator and the chosen topic taxonomy.
-
Penalizing users with genuine narrow interests or markets with limited supply.
-
Sensitivity to multi-topic posts, ranking position, and whether repeated posts are clustered or spread out.
-
Why a concentration metric should be paired with relevance and satisfaction outcomes.
Part 4 - Design an Experiment
If you modify the ranking algorithm to improve diversity, how would you test the change and define launch criteria?
What This Part Should Cover
-
A/B test design with treatment affecting ranking diversity and a clear exposure definition.
-
Primary metrics for engagement and long-term user value, plus diversity metrics that confirm the intervention worked.
-
Guardrails for hides, skips, session abandonment, creator fairness, ad revenue, latency, and satisfaction.
-
Segment analysis and long-run holdouts to detect novelty or retention effects.
What a Strong Answer Covers
A strong answer defines diversity rigorously, explains metric biases, and designs an experiment that measures both the intended diversity change and the user/business outcomes it is supposed to improve.
Follow-up Questions
-
How would you choose the strength of the diversity constraint in ranking?
-
What would you do if diversity improves retention but lowers same-day clicks?
-
How would you avoid hurting users with narrow but legitimate interests?