De-Risking a New Product Design: What Evidence to Gather and When

Imagine you and your team have settled on a design for a new product or feature. Before you commit engineering time and a launch to it, how would you confirm that the design is the right one: that it actually solves the problem it was meant to solve, and that the people it is for can use it and will? Explain which methods you would use, at which stage of development each one fits, and how the results would lead you to build the design, change it or drop it. Ground your answer in an example design of your choice. ```hint List the ways it could be wrong Before naming any method, write down the separate doubts you have about the design. Different doubts call for different evidence, and some of that evidence is far cheaper to collect early than late. ``` ### Clarifying Questions - How far along is the design: a concept sketch, a clickable prototype, or a working build that could be shown to some real users? - Is this a change to a product that already has users and traffic, so live experiments are possible, or a new product with no audience yet? - Which worry is biggest for the team right now: whether customers want this at all, whether they can use it, or whether it can be built and run at a reasonable cost? - How much time is there before the build or launch decision has to be made? ### What a Strong Answer Covers - Going back to the original problem and turning the design's key beliefs into testable hypotheses, with success thresholds agreed before any test runs - Separating the kinds of risk (value, usability, feasibility, business fit) and matching a method to each - Ordering methods from quick qualitative checks on prototypes to controlled experiments with real users, and knowing when each one fits - Sound experiment practice: sample size, duration, guardrail metrics, novelty effects, and how to read results that disagree with user research - Explicit rules for proceeding, iterating, pivoting or stopping, and continuing to validate after launch ### Follow-up Questions - Usability sessions went well, but the A/B test shows no lift in your primary metric. What do you do next? - How would you validate a design for a brand-new product that has no users or traffic to experiment on? - Leadership wants to skip validation to hit a launch date. How do you respond, and what is the minimum evidence you would insist on?