Hypothesis Testing And Confidence Intervals

What's being tested
Your ability to translate a business question into formal hypotheses, pick the correct test and assumptions, compute/interpret p-values and confidence intervals, and judge practical versus statistical significance.

Core knowledge

Worked example — "Comparing conversion rates between two groups (A/B test)"
First define the metric (conversion probability) and estimand (p_A − p_B). State H0: p_A = p_B and HA appropriately (two-sided or directional). Check design: independent random assignment and sufficient counts for normal approximation (np, n(1−p) ≥ ~5). Choose two-proportion z-test (or exact test if small counts), decide pooled versus unpooled variance based on H0, and plan to report both p-value and a 95% CI for the difference. Finally, discuss practical significance (minimum detectable uplift) and any multiple-testing or sequential monitoring corrections.

A common pitfall
Interpreting a non-significant result as evidence of no effect—without considering power—is common. Similarly, treating p-values as the probability the null is true leads to incorrect conclusions. Another frequent error is ignoring dependence (e.g., repeated users) and using independent-sample formulas, which underestimates variance and inflates false positives.

Further reading