Walk through experiments you owned: hypothesis, test design, results and decision

In a product manager interview, you are asked to share a few examples of A/B tests that you planned and carried out. Choose your examples and walk through each one: what prompted the test, how you designed it, what the results showed, and what you decided because of them. ```hint Show range, not repetition Pick examples that each reveal something different about how you run experiments, and make sure every one ends with a decision that the result changed. ``` ### Clarifying Questions - Should the examples be tests I owned from design through decision, or may they include tests where data science or engineering partners ran the analysis? - Do you want one test in depth and the others briefly, or roughly equal time on each? - Are tests with inconclusive or negative results welcome as examples? - How much statistical detail do you want, such as how sample size and significance thresholds were chosen? ### What a Strong Answer Covers - A clear hypothesis rooted in a product problem, with the primary metric and guardrails fixed before launch. - Sound design: randomization unit, control group, sample size and duration, and how bias or interference between groups was handled. - Honest interpretation of results, including statistical significance, the practical size of the effect and any segment differences. - The decision taken, its business impact, and a clear account of what the candidate personally did versus the team. - Range across the examples and reflection on what the candidate would do differently. ### Follow-up Questions - In one of your tests the primary metric improved but a guardrail got worse. Walk me through how you decided what to ship. - Leadership wanted to stop a test early because the first few days looked great. What did you do, or what would you do? - Tell me about a decision where you chose not to run an A/B test. Why?