Interpret an A/B Test p-Value Against Alpha and Explain It to a Product Manager

An A/B test has finished, and you are given its significance level $\alpha$ and the p-value of its primary metric. Explain what the result means, then explain it to a product manager or business stakeholder who does not work with statistics, and say what you would do next. The report gives neither the experiment nor the two numbers, so this version fixes only the setting: a user-randomized test of one product change against control, with a primary metric chosen before launch. Be ready for either outcome the interviewer might hand you: a p-value at or below $\alpha$, or one above it. ### Clarifying Questions - What are the observed difference between the arms and its confidence interval, not only the p-value? - Were $\alpha$, the primary metric, the sample size and the duration fixed before the test started, and what minimum detectable effect was the test powered for? - Is the test one-sided or two-sided? - How many metrics, variants or segments were compared, and was the result checked repeatedly while the test was running? - Which launch decision does this result feed, and how large an effect would justify the cost of the change? ### Part 1 — Interpret the result State what the p-value and $\alpha$ mean, and what you can and cannot conclude when $p \le \alpha$ and when $p > \alpha$. ```hint What the p-value assumes The p-value is computed as if one particular statement were true. Name that statement before saying what a small or a large value implies. ``` ```hint A non-significant result When $p > \alpha$, ask whether this test could have detected an effect of the size the business cares about. ``` #### What This Part Should Cover - The definitions of the p-value and $\alpha$, and the decision rule that links them - What a significant result establishes and what it leaves open - What a non-significant result establishes and what it leaves open, including the role of power ### Part 2 — Explain it to a product manager How would you explain each outcome to a product manager or business stakeholder? Give the words you would use, and say which statements you would avoid. ```hint Start from their decision The stakeholder will act on what you say. Ask which quantity, in the metric's own units, matters more to that decision than the p-value does. ``` #### What This Part Should Cover - A plain-language statement of the result and its uncertainty, in business terms - A recommendation tied to the launch decision - Common mistranslations of a statistical result and how to avoid them ### Part 3 — Next steps What would you do next after each outcome, both before and after the launch decision? ```hint Trust before you act Before deciding what the p-value implies for launch, ask what could make the p-value itself wrong. ``` #### What This Part Should Cover - Checks on the validity of the test before its result is used - How guardrail metrics, practical significance and cost enter the decision - Follow-up analyses or experiments for an inconclusive or surprising result ### What a Strong Answer Covers - A correct interpretation of the p-value, free of the common misreadings - A clear separation between statistical significance and practical or business significance - A stakeholder explanation that conveys effect size and uncertainty honestly and ends in a recommendation - Next steps that check the test's validity first and connect the result to a launch decision - Awareness of how peeking, multiple metrics and post-hoc segment cuts push the false-positive rate above $\alpha$ ### Follow-up Questions - The p-value is just above $\alpha$, and the product manager asks whether the result can be called "almost significant" and launched. What do you say? - The team watched the dashboard daily and wants to stop the test on the first day the p-value drops below $\alpha$. What is wrong with that, and what would you propose instead? - The primary metric improved significantly, but a guardrail metric moved in the wrong direction. How do you present the trade-off to the stakeholder? - After a non-significant overall result, the product manager asks you to cut the data by platform, country and plan type to find where the change worked. How do you respond?