Explain p‑values to a product manager
Overview
This question evaluates a candidate's understanding of hypothesis testing and p-value interpretation, statistical inference concepts (including null hypothesis framing, test statistics, sampling distributions, and the relationships among p-values, confidence intervals, alpha, and power), awareness of experimental-design pitfalls such as optional stopping and multiple looks, and the ability to communicate these ideas to a non-technical product manager in a Data Scientist role within the Statistics & Math domain. It is commonly asked because interviewers need evidence that a candidate can avoid and correct common misinterpretations (for example conflating a p-value with Pr(H0|data) or equating a small p with a large effect), and its level of abstraction is primarily conceptual understanding with applied communication and interpretation aspects rather than implementation-level calculation.
Interpreting a Two-Sided A/B Test p-value of 0.03
You ran a two-sided A/B test on conversion rate. Results:
- Baseline (control) conversion: 5.0%
- Observed difference (treatment − control): +0.4 percentage points (pp)
- Pooled standard error (SE) of the difference: 0.18 pp
- Two-sided p-value: 0.03
Explain to a product manager what a p-value of 0.03 actually means and, critically, what it does NOT mean. Use the concrete numbers above and cover:
- Null hypothesis, test statistic, sampling distribution, and the “extremeness under the null” interpretation.
- Why p ≠ Pr(H0|data) and why p is not the false positive rate for this single result.
- The relationship between p-values, confidence intervals, alpha, and statistical power (and why a small p doesn’t imply a large effect).
- How optional stopping/peeking and multiple looks inflate type I error and how to communicate this risk.
- A non-technical 2–3 sentence explanation you would actually say to the PM, and a contrasting Bayesian framing for the same result.
Note: “pp” = percentage points (e.g., 5.0% to 5.4% is +0.4 pp).
Solution
Login required