Sizing an A/B test from baseline, minimum detectable effect, significance and power

As a product manager, explain how you would decide how many users an A/B test needs before you launch it. ```hint What makes a change hard to see List the quantities that make a real change easier or harder to tell apart from noise, then reason about which way each one pushes the number of users. ``` ### Clarifying Questions - Is the primary metric a rate, such as conversion, or a continuous value, such as revenue or trips per user? - What is the smallest change in that metric that would be worth acting on? - Is the unit of randomization an individual user, or something coarser such as a city or a time window? - How many variants will be compared with control, and will traffic be split evenly? ### What a Strong Answer Covers - The inputs: baseline rate or variance, minimum detectable effect, significance level, power and the allocation split - The calculation, and how each input moves the required sample - A worked example with correct arithmetic, converted into a duration using eligible traffic and full weekly cycles - Adjustments for multiple variants, uneven splits, clustered randomization, heavy-tailed metrics and variance reduction - What to do when the required sample is out of reach ### Follow-up Questions - If collecting the required sample would take several months, what options do you have? - How does the calculation change if you randomize by city instead of by individual rider? - Why is it a mistake to stop the test the first time the p-value falls below 0.05?