Walk Through an A/B Test You Designed: Hypothesis, Setup, Result and Ship Decision

Walk me through an A/B experiment you designed: the decision it was meant to inform, how you set it up, what the results showed and what happened next. ```hint Start from the decision Before you describe variants or metrics, name the choice the team was stuck on and what result would have changed it. ``` ### Clarifying Questions - Does it have to be a test I designed alone, or can it be one I designed with a data scientist or analyst, as long as I am clear about which parts were mine? - How deep should I go on the statistics, such as power, sample size and significance, compared with the product reasoning and the decision? - Is an experiment that came back flat or negative a good example, or would you rather hear about one that shipped? ### What a Strong Answer Covers - A testable hypothesis tied to a real product decision, with the primary metric and the decision rule fixed before launch - Explicit design choices: randomization unit, eligible audience, guardrail metrics, and how sample size and run length were set - A complication the candidate handled, such as an uneven traffic split, a novelty effect, metrics that disagreed or pressure to stop early - An honest reading of the result and the decision it led to, including what a null or negative outcome taught the team - The candidate's own role, and what they would design differently today ### Follow-up Questions - Your primary metric improved significantly, but a guardrail metric got worse. What did you ship, and who made that call? - A stakeholder wanted to stop the test after a few days because it looked like a win. How did you respond? - Tell me about a decision where you chose not to run an A/B test. Why not?