Success Metrics for a Feature at Launch: North Star, Guardrails and Experiment Plan

Pick any product you know well and choose one of its features. Assume the feature is launching right now. Explain how you would judge whether it succeeds: what the feature is supposed to achieve, which metrics you would track, and what results would convince you that the launch worked. ```hint Start from the feature's job Before naming any metric, state in one sentence what the feature should change for its users and for the product. Because the launch is happening now, separate what you can learn in the first weeks from what only shows up after months. ``` ### Clarifying Questions - May I pick a product from any company, or should it be one this company builds? - Is the feature launching to all users at once, or can it go to a randomly chosen group first so its effect can be compared against a control group? - Over what window should success be judged: the first few weeks after launch, or a longer period? - Is the feature expected to earn money directly, or to improve engagement, retention or user satisfaction? ### What a Strong Answer Covers - A feature choice explained through its purpose, its users and how it fits the product's goals. - A user journey for the feature that turns into metrics at each step, from first exposure through repeat use to the value delivered. - One primary success metric tied to the value users get, with supporting metrics and guardrails that catch harm to the rest of the product. - A plan to measure the feature's causal impact, such as a randomized rollout or holdout, accounting for novelty effects. - Targets or decision rules set before launch, separating early signals from long-term outcomes. ### Follow-up Questions - Usage of the feature is high, but overall use of the product is flat. What could explain that, and is the launch a success? - Two weeks after launch, the primary metric is up but one guardrail has worsened. How do you decide whether to keep rolling out? - If a randomized experiment is impossible, how would you estimate the feature's effect?