Weighing Usage Metrics Against User Research When Making Product Decisions

In a product manager interview, you are asked how you balance quantitative data (usage metrics, funnels, experiment results) against qualitative data (user interviews, usability sessions, open-ended feedback) when you make product decisions. Explain how you decide which kind of evidence should carry more weight, how you use the two together over the course of a decision, and what you do when they point in different directions. Support the approach with an example. ```hint Match the data to the question Think about which questions each kind of data can answer well and which it cannot, and how the balance should shift between finding a problem, choosing a solution and confirming that it worked. ``` ### Clarifying Questions - Would you like my general approach, a specific decision where I combined the two, or both? - Should I assume a product with plenty of usage data and the ability to run A/B tests, or a new product that few people use yet? - Does qualitative data include sources such as support tickets and the judgment of domain experts inside the company, or only formal user research? ### What a Strong Answer Covers - What each kind of data is good and bad at, tied to the questions it answers (how many and how much versus why) - A deliberate order for using them across a decision, rather than a statement that both matter - A method for resolving conflicts between them that checks data quality and segments before choosing a side - The factors that shift the balance, such as how reversible the decision is, how much traffic exists, and whether the outcome is hard to measure - A concrete example that shows which evidence led, what was decided and what happened ### Follow-up Questions - Your A/B test shows a clear engagement win, but the users you interviewed dislike the change. What do you ship, and why? - How many user interviews are enough before you act, and how do you know you heard a representative group? - When would you make a product decision with no quantitative data at all?