Feature Launch Measurement: Deciding What Counts as Success

When you launch a new feature, how do you decide which metrics will tell you whether it succeeded? ```hint Start from the change you expect Before naming any metric, state what users should do differently if the feature works, and what could get worse as a side effect. ``` ### Clarifying Questions - Should I lay out a general method, or apply it to a specific feature, either one you name or one I pick? - Will the launch include a randomized control group, or does the feature go to every user at once? - Is success judged in the first weeks after launch, or on longer-term outcomes such as retention and repeat purchases? ### What a Strong Answer Covers - Starting from the feature's purpose and the user behavior it should change - One primary metric chosen for sensitivity, attribution and resistance to gaming, plus adoption and diagnostic metrics - Guardrails for side effects such as cannibalization, performance and core business health - Baselines, targets, a time horizon and a ship, iterate or roll back rule agreed before launch - A measurement design that attributes the change to the feature, accounting for novelty effects and spillover ### Follow-up Questions - The primary metric improved but a guardrail dipped slightly. How do you decide whether to ship? - Only a small share of users will ever see the feature. How do you still get a readable result? - How would you judge success for a feature that must reach every user at once, with no control group?