Choose the right test for proportions
Company: Intuit
Role: Data Scientist
Category: Statistics & Math
Difficulty: hard
Interview Round: HR Screen
You run an A/B test on delivery completion rate (delivered_bool). Results:
- Control: 1,000 orders, 920 delivered, 80 not delivered.
- Variant: 1,000 orders, 880 delivered, 120 not delivered.
1) Select the most appropriate significance test and justify why it’s preferred over a t-test on proportions. Derive the test statistic formula starting from Bernoulli → Binomial → Normal approximations and state the conditions under which the approximation is valid (expected cell counts, continuity corrections).
2) Compute the p-value to 3 significant digits and a 95% confidence interval for the difference in proportions; show intermediate steps.
3) Now suppose a small-market rollout yields: Control 20 delivered / 25 total; Variant 12 delivered / 18 total. Re-evaluate test choice (Chi-square vs. Fisher’s exact vs. z-test). Compute the exact p-value or explain precisely how you would obtain it.
4) Explain when a two-sample t-test gives approximately the same result as a z-test for proportions, and when it fails. Include at least two concrete failure modes (e.g., low counts, unpooled variance misuse).
Quick Answer: This question evaluates understanding of hypothesis testing for binary outcomes, including selection among z-test, chi-square, and Fisher’s exact test, derivation of the test statistic from Bernoulli→Binomial→Normal approximations, and computation of p-values and confidence intervals.