Describe an experiment you designed or analyzed. Explain the decision it was meant to inform, how you chose the experimental unit and metrics, what the results showed, and what you recommended. Include a complication or trade-off you had to manage. ### Constraints & Assumptions - Do not disclose confidential experiment names, exact traffic levels, or unreleased results. - Clearly distinguish pre-launch design choices from post-launch analysis. - Report uncertainty rather than treating every observed difference as a causal effect. ### Clarifying Questions to Ask - Would you like the emphasis on experiment design, statistical analysis, or stakeholder decision-making? - Should I use an example with a clean result or one with conflicting metrics? ### What a Strong Answer Covers - The hypothesis, decision, randomization unit, eligibility rule, and treatment exposure. - A primary metric, guardrails, expected direction, and a justified analysis window. - Checks for sample-ratio mismatch, instrumentation, novelty, interference, and important segments. - Effect size and uncertainty, followed by a recommendation that accounts for trade-offs. ### Follow-up Questions 1. What would you do if the primary metric improved but a guardrail worsened? 2. How would you handle users exposed on multiple devices? 3. Which result would make you extend rather than end the experiment?