# Present and Defend a Product Experiment You Designed Describe a real experiment you designed or analyzed. The discussion will probe the decision context, hypothesis, metric definitions, randomization, sample size, execution, result, and what you would change. Keep the answer technical enough to show experimental judgment rather than only project storytelling. ### Constraints & Assumptions - Use one concrete experiment and distinguish what you personally owned from the broader team's work. - Do not disclose confidential company names, exact proprietary thresholds, or participant data. - State deviations from the original analysis plan and their consequences. - A null or unexpected result is acceptable when the reasoning is rigorous. ### Clarifying Questions to Ask - What decision and user population did the experiment target? - What was randomized, and could treatment spill across units? - Which outcome was primary and when was it considered mature? ### Part 1 — Design Explain the causal hypothesis, randomization unit, eligibility, primary metric, guardrails, power inputs, and prelaunch validation. #### What This Part Should Cover - A mechanism connecting treatment to outcome - Denominators and observation windows - Interference, novelty, and instrumentation risks ### Part 2 — Execution and analysis Describe monitoring, data-quality checks, statistical analysis, heterogeneity, and any deviations without turning exploratory findings into confirmed results. #### What This Part Should Cover - Intent-to-treat and uncertainty - Predeclared versus exploratory work - Practical significance and guardrail interpretation ### Part 3 — Decision and reflection State the decision, how you communicated it, what changed afterward, and what you would redesign. #### What This Part Should Cover - A decision linked to evidence - Stakeholder trade-offs and ownership - Specific lessons and a better next design ### What a Strong Answer Covers - Clear personal ownership - Causal and statistical precision - Honest treatment of uncertainty and limitations ```hint Lead with the decision Frame every design choice around the product decision the experiment needed to inform. This keeps the story from becoming a list of statistical techniques. ``` ### Follow-up Questions - What would invalidate your conclusion? - How would you proceed if the required sample were unattainable?