Running a feature A/B test end to end: metrics, randomization, sizing and the launch call

You are a product manager about to launch a new feature. Explain how you would A/B test it, from setting up the experiment to deciding whether to ship. ```hint Decide before you look Settle what success means, who gets randomized and how long the test runs before any data arrives, then ask what could make the readout misleading. ``` ### Clarifying Questions - Is there a specific feature and goal in mind, or may I choose an illustrative one? - Do users affect each other through this feature (sharing, a marketplace, shared accounts), so that treated users could change what control users experience? - What is the smallest improvement that would justify the feature's build and maintenance cost? - Are there users who must not be exposed, or legal or contractual limits on who can be randomized? ### What a Strong Answer Covers - A falsifiable hypothesis with a primary metric, supporting metrics and guardrails fixed before launch - Randomization unit, eligibility and exposure, and how sample size and duration follow from the baseline, minimum detectable effect, significance and power - Pre-launch and in-flight checks: instrumentation, A/A tests, sample ratio mismatch, a safe ramp and stop rules - Analysis discipline: no peeking, multiple comparisons, segment cuts, novelty and change-aversion effects - A launch rule that weighs practical as well as statistical significance and resolves conflicting metrics ### Follow-up Questions - The primary metric improves significantly, but a guardrail metric gets slightly worse. How do you make the call? - How would you test a sharing feature where users in the control group see content created by treated users? - The result looks significant after a few days. Why might you keep the test running anyway?