What to Do When User Research and Product Metrics Disagree

Your user research and your product data disagree: what users tell you points one way, and the numbers point the other. How would you work through the conflict and reach a decision? ```hint Check whether they measure the same thing Before picking a winner, ask whether the two sources describe the same users, the same behavior and the same period of time. A contradiction is often a clue that each source is answering a different question. ``` ### Clarifying Questions - Is a specific product decision riding on this, such as shipping or removing a feature, or should I walk through a general approach with an example? - What kind of qualitative evidence is involved: a handful of interviews, usability sessions, support tickets, or open-ended survey responses? - Does the quantitative data come from a controlled experiment, from observational product analytics, or from a large survey? - How reversible is the decision, and how soon must it be made? ### What a Strong Answer Covers - Checking each source's validity: who was sampled, what was measured, and the biases each method carries - Common reasons the two can both be true, such as different segments, stated versus revealed behavior, or a metric that misses what users value - Concrete ways to resolve the conflict: segmenting the data, quantifying the qualitative theme, targeted follow-up research or an experiment - A decision proportional to the stakes and reversibility, and how it is communicated to the team - A worked example that makes the approach tangible ### Follow-up Questions - An A/B test shows a redesign lifts engagement, but usability sessions show people find it confusing. Would you launch it? - How do you avoid cherry-picking whichever source supports what the team already wanted to build? - When would you act on qualitative insight alone while the quantitative data still points the other way?