An A/B test estimates a 2.0% relative improvement in a primary metric, with a two-sided 95% confidence interval from -0.5% to 4.5% and (p=0.12). Guardrail metrics show no material harm. A stakeholder believes the change is still valuable. Explain statistical significance and make a recommendation. Address practical significance, uncertainty, power, the original decision rule, and what additional evidence would justify shipping, iterating, or stopping. ### Constraints & Assumptions - The experiment and primary metric were specified before results were examined. - There was no sample-ratio mismatch or known instrumentation failure. - The confidence interval is for the same relative effect reported in the estimate. - Do not reinterpret “not significant” as proof of no effect. ### Clarifying Questions to Ask - What minimum effect would make the change worthwhile after implementation and maintenance costs? - Was the experiment powered for that effect, and did it reach its planned sample size? - Is the decision reversible, and what is the cost of a false positive or false negative? - Were multiple variants, metrics, or interim looks analyzed? ```hint Compare the interval with the decision threshold The interval contains zero, but it may also contain effects that are meaningfully positive or too small to matter. ``` ### What a Strong Answer Covers - A correct interpretation of the p-value and confidence interval. - Separation of evidence strength from business value. - Recognition that the result is inconclusive, not necessarily negative. - A recommendation conditioned on the minimum worthwhile effect and risk. - A principled next step rather than post hoc metric shopping. ### Follow-up Questions - How would your advice change if the entire interval were below the minimum worthwhile effect but above zero? - When is a non-inferiority design more appropriate? - How can a Bayesian decision analysis incorporate implementation cost and reversibility?