Ship or Kill: Turning A/B Test Results into a Launch Decision

An A/B test of a new feature has finished and the readout is in front of you. Walk through how you decide whether the feature should launch. ```hint Trust before you interpret Before looking at whether the main metric moved, ask what could make the readout wrong, and what bar a result has to clear for a launch to be worth it. ``` ### Clarifying Questions - Will you share a specific readout (metric changes, confidence intervals, sample sizes), or should I lay out the decision process for any result? - Did the team agree on a primary metric, guardrail metrics and a launch threshold before the test started? - What does the feature cost to keep running: engineering upkeep, infrastructure, or added complexity for users? - How easy is it to undo the launch if problems appear later? ### What a Strong Answer Covers - Checks that the result can be trusted before it is interpreted: assignment balance, logging, run length, novelty effects and interference between users. - Statistical versus practical significance, judged with confidence intervals against the smallest effect worth shipping. - Weighing the primary metric against guardrails and secondary metrics, including how trade-offs between them are priced. - Disciplined use of segment results that avoids cherry-picking and multiple-comparison traps. - A clear recommendation (ship, ship to a segment, iterate, extend or stop) that accounts for cost, reversibility and long-term effects. ### Follow-up Questions - The primary metric rose significantly, but a guardrail fell slightly and the drop is not significant. Do you launch? - The overall result is flat, but one user segment shows a large, significant lift. What do you do? - A senior leader wants to launch despite a flat result because the feature is strategic. How do you respond?