Forecasting a New Feature's Adoption Lift With Pre-Launch Research and Tests

You are a product manager with a new feature on the roadmap. Before it launches, how would you work out the effect it is likely to have on adoption of your product? Describe the methods you would use as the feature moves from an idea to a working version, what each method can and cannot tell you, and how you would turn the evidence into a launch decision. ```hint Evidence changes as the feature takes shape Ask what evidence you could gather at each stage, from idea to working version, and what could make each kind of evidence look better than reality. ``` ### Clarifying Questions - What does "adoption" mean here: how many users try the feature, how many keep using it, or whether it brings in new users and keeps existing users active? - Does "before launch" rule out releasing a working version to a small random group of real users, or only a release to everyone? - Is the feature already built, or is the estimate meant to decide whether to build it? - Does the feature's value depend on other users adopting it too (sharing, collaboration, a marketplace), which would make a user-level split unreliable? ### What a Strong Answer Covers - A precise definition of the adoption outcome, separating feature usage from product-level adoption and retention. - A staged set of pre-launch methods, with what each can and cannot show and the biases in each. - A controlled experiment design: randomization unit, primary and guardrail metrics, sample size and duration, and novelty effects. - Alternatives when a clean user-level test is impossible. - A decision rule agreed before the results arrive. ### Follow-up Questions - Early test results show heavy use of the feature in the first week but no change in retention. How do you interpret that, and what do you do? - The team points out that users who adopted the feature in beta retain much better than everyone else. Is that evidence that the feature improves retention? - Leadership wants a single forecast of the adoption lift before anything is built. What do you give them, and how do you express the uncertainty?