Designing Metrics and Experiments to Evaluate a TikTok For You Ranking Change

TikTok changes the recommendation algorithm behind its feed frequently. How would you measure the impact of an algorithm update on user engagement? Explain which metrics you would use, how you would separate the algorithm's effect from everything else that moves engagement, and how you would interpret what you find. ```hint Engagement is not one number More watching is not automatically a better feed. Decide which behaviors show that recommendations improved, which could rise for the wrong reasons, and what the steady stream of other algorithm changes does to your ability to see any single one. ``` ### Clarifying Questions - Are we evaluating one specific change before it launches, or designing how TikTok measures the steady stream of algorithm updates? - What was the update meant to improve: relevance, content variety, new-user retention, creator reach, or something else? - Can we randomize a share of users into the new algorithm, or has the change already reached everyone? - Does the change affect only the For You feed, or also other surfaces such as Following, search or LIVE? ### What a Strong Answer Covers - A metric hierarchy tied to the update's purpose: a primary engagement metric, supporting signals, and guardrails for satisfaction, wellbeing, integrity and creators - A sound experiment: randomization unit, sample size and duration, and validity checks - Effects peculiar to ranking changes: novelty and learning periods, feedback loops through training data, and spillover onto creators and users outside the test - How to measure when many updates overlap, including the cumulative effect of a series of launches - A decision rule for metric trade-offs and for segments that move in different directions ### Follow-up Questions - Watch time rises, but "not interested" taps rise and next-day retention among new users falls. Do you ship? - The update wins a two-week test. How would you check that the gain lasts? - How would you measure the update's effect on creators rather than viewers?