Evaluate Home-Feed Diversity's Impact on User Engagement Metrics
Quick Overview
Evaluates how to measure and experiment on home-feed diversity in a personalized ranking system. Strong answers define diversity metrics, explain their biases, connect diversity to business benefits, and design an A/B test that balances discovery, relevance, retention, and guardrails.
Evaluate Home-Feed Diversity's Impact on User Engagement Metrics
Company: TikTok
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: hard
Interview Round: Onsite
##### Scenario
Pinterest home-feed diversity initiative: posts can belong to topics like animals, recipes, etc.
##### Question
What business benefits can come from increasing home-feed diversity? Propose several quantitative metrics to measure diversity of a user’s feed. For each metric, discuss possible biases and limitations. Consider the metric ‘percentage of posts from the same topic’—what bias does it carry? If you modify the ranking algorithm to improve diversity, design an experiment to test the change and define launch criteria.
##### Hints
Think entropy, Gini, coverage; discuss user satisfaction, engagement, exploration; outline A/B-test design and guardrail metrics.
Quick Answer: Evaluates how to measure and experiment on home-feed diversity in a personalized ranking system. Strong answers define diversity metrics, explain their biases, connect diversity to business benefits, and design an A/B test that balances discovery, relevance, retention, and guardrails.
Evaluate Home-Feed Diversity's Impact on User Engagement Metrics
You run a personalized home feed where each post is tagged with one or more topics, such as animals, recipes, travel, or fashion. Product leadership wants to increase topical diversity without harming relevance.
Constraints & Assumptions
Treat diversity as a measurable feed-quality intervention, not as a vague goal.
Assume topic labels, ranking scores, impressions, clicks, hides, saves, sessions, retention, and experiment infrastructure are available.
A more diverse feed can improve discovery and reduce fatigue, but can also reduce short-term relevance if pushed too far.
Include both diversity metrics and user/business outcome metrics.
Clarifying Questions to Ask Guidance
What level of topic taxonomy is used, and can posts have multiple topics?
Is the goal to increase diversity within a session, across days, or across a user's long-term exposure?
What is the current ranking objective, and where would diversity be added?
Are there important creator, advertiser, or content-quality constraints?
Part 1 - Explain Business Benefits
What business benefits can come from increasing home-feed diversity?
The possibility that benefits appear over longer horizons rather than immediate clicks.
Part 2 - Define Diversity Metrics
Propose several quantitative metrics to measure diversity of a user's feed. For each metric, define it and discuss possible biases or limitations.
What This Part Should Cover Guidance
Metrics such as unique topics per session, entropy, Herfindahl-Hirschman concentration, same-topic run length, topic coverage, creator diversity, and distance between adjacent items.
Biases from topic taxonomy granularity, multi-label posts, impression position, user interest breadth, and supply availability.
Segment-level interpretation so niche-interest users are not penalized for having concentrated preferences.
Part 3 - Analyze a Specific Metric
Consider the metric "percentage of posts from the same topic." What bias does it carry?
What This Part Should Cover Guidance
Dependence on the denominator and the chosen topic taxonomy.
Penalizing users with genuine narrow interests or markets with limited supply.
Sensitivity to multi-topic posts, ranking position, and whether repeated posts are clustered or spread out.
Why a concentration metric should be paired with relevance and satisfaction outcomes.
Part 4 - Design an Experiment
If you modify the ranking algorithm to improve diversity, how would you test the change and define launch criteria?
What This Part Should Cover Guidance
A/B test design with treatment affecting ranking diversity and a clear exposure definition.
Primary metrics for engagement and long-term user value, plus diversity metrics that confirm the intervention worked.
Guardrails for hides, skips, session abandonment, creator fairness, ad revenue, latency, and satisfaction.
Segment analysis and long-run holdouts to detect novelty or retention effects.
What a Strong Answer Covers Guidance
A strong answer defines diversity rigorously, explains metric biases, and designs an experiment that measures both the intended diversity change and the user/business outcomes it is supposed to improve.
Follow-up Questions Guidance
How would you choose the strength of the diversity constraint in ranking?
What would you do if diversity improves retention but lowers same-day clicks?
How would you avoid hurting users with narrow but legitimate interests?