Responding When a Product Experiment Contradicts Your Hypothesis

Behavioral question for a product manager role at Amazon: tell me about an experiment you ran whose results were not what you expected. How did you respond once you saw them? ```hint Make the surprise concrete Pick an experiment where you had written down what you expected before it ran. Then you can show exactly how the result differed, what you checked before believing it, and what you decided because of it. ``` ### Clarifying Questions - Does it have to be a controlled A/B test, or does a pilot, a beta, a pricing test or a prototype study with customers count? - Should the story center on how I investigated the result, or on the product decision and its impact afterwards? - Is a story acceptable where the surprise turned out to be a logging or setup error rather than real customer behavior? - Should it be an experiment I owned end to end, or can it be one I designed while another team ran it? ### What a Strong Answer Covers - A specific experiment with the hypothesis, the primary metric and the expected result stated up front, so the direction and size of the surprise are clear - Checking that the result was trustworthy (instrumentation, randomization, sample size, novelty effects) before acting on it - Investigating why it happened through segment cuts and customer evidence, rather than explaining it away - The decision taken, how it was communicated to stakeholders who expected a different outcome, and the measurable result - What the candidate changed about how they plan or read experiments afterwards ### Follow-up Questions - Walk me through exactly which data you examined to convince yourself the result was real and not a bug or noise. What would have changed your mind? - A senior stakeholder had already committed to launching the feature. How did you present the result, and what did you recommend? - Looking back, was there an early signal that should have told you the outcome might differ from your hypothesis?