Product Case: Define Success Metrics for a Product, Using Enterprise Search as Example

In a hiring-manager interview for a data scientist role, one question was a product case: how would you measure the success of a product? The report does not say which product. For practice, assume the interviewer meant the company's own product: an enterprise search tool that employees of customer companies use to find documents, people and answers across their workplace apps. In a real interview, confirm the product before answering. ```hint Two kinds of customer In a business-to-business product, the person who uses it and the organization that pays for it are different. Decide what success means for each. ``` ```hint Guard the headline metric For each metric you propose, ask how it could go up while the product actually gets worse. ``` ### Clarifying Questions - Is the goal to define success for the whole product, or for one feature or launch? - What decision will the metrics inform: roadmap prioritization, a launch decision, or reporting to customers? - Which data is available: search logs with clicks, per-customer seat counts, renewals, user surveys? ### What a Strong Answer Covers - A goal hierarchy: the product goal, a north-star metric, the input metrics that drive it, and guardrails - Metrics at both the user level and the customer-account level - Search-quality measures that separate a successful search from mere activity - How to attribute changes causally, through experiments, and what to do when randomization is hard - Segmentation, and the pitfalls of averages dominated by a few large customers ### Follow-up Questions - Queries per user rose after a release. Is that good news? - How would you run an A/B test when users in the same company share documents and influence each other? - Which single metric would you put on the executive dashboard, and why?