Plan a two-sided A/B test for a binary conversion metric. Control conversion is 10%, the smallest effect worth detecting is an increase to 11%, significance level is 5%, desired power is 80%, allocation is 50/50, and 20,000 eligible users enter the experiment per day. Estimate the required sample size and nominal test duration. Then explain which operational inputs can make the actual duration longer and what must be decided before performing power analysis. ### Constraints & Assumptions - Each user is assigned once and contributes one binary outcome. - Use a standard normal approximation for two independent proportions. - The 1 percentage point change is an absolute minimum detectable effect. - Ignore finite-population correction. ### Clarifying Questions to Ask - Is the hypothesis one-sided or two-sided? - Is 80% power required at exactly the minimum effect or across several scenarios? - What fraction of eligible users actually trigger the experiment and produce a mature outcome? - Are there clustering, repeated exposure, multiple variants, or planned interim looks? ```hint Convert sample size to calendar time last First compute observations per arm from alpha, power, baseline variance, and the minimum detectable effect; only then divide by effective daily traffic. ``` ### What a Strong Answer Covers - Inputs to power analysis and the distinction between absolute and relative effects. - An approximate two-proportion sample-size calculation. - Per-arm versus total sample size and traffic allocation. - Trigger rate, attrition, delayed outcomes, design effect, and full-cycle constraints. - A duration recommendation that does not mechanically stop at the first significant result. ### Follow-up Questions - How does a 20% trigger rate change the duration? - What adjustment is needed when users are randomized by household? - Why might the test run for full weekly cycles even after reaching sample size?