Deciding what to do after an experiment shows an engagement decline

You are a product manager at Meta. You ran an A/B test on a product change, and the results show that the treatment group's engagement is lower than the control group's. What would you do about it? ```hint A drop is not automatically a failure Before deciding anything, check whether the drop is real, find out which part of engagement fell and for whom, and compare the loss with what the change was meant to achieve. ``` ### Clarifying Questions - What was the change meant to improve, and was engagement its primary success metric or a guardrail? - Which engagement measure fell (time spent, sessions, likes and comments, posts), by how much, and is the drop statistically significant? - How long did the test run, and does the gap between treatment and control shrink, hold steady or grow over time? - Were users randomized individually, and could people in treatment and control see or react to each other's content? ### What a Strong Answer Covers - Tying the decision to the goal of the change and to decision criteria agreed before the test. - Validating the result: sample ratio, instrumentation, performance, statistical and practical significance, and test duration. - Diagnosing the drop by metric, surface, segment and over time, including change aversion and spillover between friends in different test groups. - Weighing the engagement loss against the intended gains and long-term health, then making a clear ship, iterate or roll back decision. - Communicating the decision and the follow-up plan, such as a long-term holdout. ### Follow-up Questions - The change was meant to reduce misleading posts in Feed; time spent fell slightly while reports of false news dropped. Would you ship it, and what would you watch afterwards? - The drop is concentrated among the most active users and absent among new users. What does that suggest, and what would you do next? - How would you redesign the test if the change affects how friends interact with each other?