Choose the right test for proportions
Overview
This question evaluates understanding of hypothesis testing for binary outcomes, including selection among z-test, chi-square, and Fisher’s exact test, derivation of the test statistic from Bernoulli→Binomial→Normal approximations, and computation of p-values and confidence intervals.
Intuit
Oct 13, 2025, 9:49 PM
Data Scientist
HR Screen
Statistics & Math
0
0
A/B Test on Delivery Completion Rate (Binary Outcome)
Context: You are evaluating an A/B test on a binary metric (delivered_bool). Treat each order as an independent Bernoulli trial. Assume two-sided hypotheses at alpha = 0.05 unless stated otherwise.
Data (Phase 1):
- Control: n1 = 1,000 orders, x1 = 920 delivered, 80 not delivered
- Variant: n2 = 1,000 orders, x2 = 880 delivered, 120 not delivered
Tasks
- Choose the most appropriate significance test to compare completion rates and justify why it is preferred over a t-test on proportions. Derive the test statistic starting from Bernoulli → Binomial → Normal approximations, and state conditions for validity (expected cell counts; continuity corrections).
- Compute the two-sided p-value (3 significant digits) and a 95% confidence interval for the difference in proportions. Show intermediate steps.
-
Small-market rollout (Phase 2) yields:
- Control: 20 delivered / 25 total
- Variant: 12 delivered / 18 total Re-evaluate the test choice (Chi-square vs. Fisher’s exact vs. z-test). Compute the exact p-value or explain precisely how to get it.
- Explain when a two-sample t-test gives approximately the same result as a z-test for proportions, and when it fails. Include at least two concrete failure modes.
Solution
Login required
Show