Design an A/B test; choose Z vs T
Overview
This question evaluates a candidate's competency in experimental design and statistical inference, covering sample-size and power calculations, the choice between Z and T tests, sequential testing/alpha-spending strategies, and causal-inference planning for non-randomized rollouts.
A/B Test on a Signup Funnel: Sample Size, Test Choice, Sequential Design, and Causal Plan
Context
You are planning a two-variant A/B test on a signup funnel with the following parameters:
- Baseline conversion p0 = 0.050
- Target minimum detectable effect (MDE) = +10% relative → p1 = 0.055 (Δ = 0.005 absolute)
- Two-sided alpha = 0.05; power = 0.80
- Allocation = 1:1
- Daily site traffic = 50,000 visitors; only 60% meet targeting criteria; 90% of those are successfully bucketed
- Assume independence across users
Tasks
(a) Compute the required sample size per variant and the expected calendar duration (in days) to reach it given the traffic constraints above. State your formula and any continuity or pooled-variance assumptions.
(b) Explain whether you would use a Z-test or a T-test for the primary proportion metric and why. Under what conditions do their results materially differ? Include how estimating variance from data, small-sample corrections, and unequal sample sizes/variances affect your choice.
(c) If product wants the test to stop “as soon as it looks good,” propose a sequential testing or alpha-spending approach (e.g., group-sequential boundaries) that controls Type I error. Specify stopping rules and how they change the nominal sample size and expected duration.
(d) Suppose randomization is not possible for a related rollout. Sketch a causal inference plan: draw a DAG to identify confounders, propose an identification strategy (e.g., difference-in-differences with parallel trends checks or an instrumental variable) and list key assumptions you would need to defend. How would you validate those assumptions in practice?
Solution
Login required