A content platform wants to increase creation of video Pins. Design an experiment, then interpret the illustrative result table below and make a launch recommendation. The values are synthetic practice data, not reported company results. ### Constraints & Assumptions - Randomization is at the eligible creator level, with one assigned experience per creator. - The analysis window is seven days after first eligible exposure. - The primary metric is video Pin creation rate; content reports are a lower-is-better guardrail. - Confidence intervals below are 95% intervals for the treatment-minus-control difference. ### Clarifying Questions to Ask - What product change is being tested, and which creators are eligible to receive it? - Does “creation” mean starting, completing, or publishing a video Pin? - Is the goal more creators, more total video Pins, or retained creation behavior? - Could treatment affect viewers or untreated creators through marketplace spillovers? ### Part 1: Design the Experiment Define the hypothesis, experimental unit, eligibility and exposure rules, primary metric, guardrails, analysis population, duration, and pre-launch checks. Explain how you would handle creators with multiple devices and repeat exposures. #### What This Part Should Cover - Alignment between the decision, causal mechanism, unit of randomization, and metric denominator. - A precise creation metric plus retained behavior and quality guardrails. - Power and duration reasoning, instrumentation validation, and sample-ratio checks. - Risks from interference, novelty, missing outcomes, and repeated user activity. ### Part 2: Interpret the Results | Metric | Control | Treatment | Difference | 95% CI | |---|---:|---:|---:|---:| | Assigned creators | 100,000 | 99,800 | — | — | | Video Pin creation rate | 8.0% | 8.5% | +0.5 pp | [+0.15, +0.85] pp | | Seven-day creator retention | 31.0% | 31.1% | +0.1 pp | [-0.4, +0.6] pp | | Viewer session minutes | 18.0 | 18.6 | +0.6 | [-0.1, +1.3] | | Content reports per 10,000 video impressions | 12 | 15 | +3 | [+0.8, +5.2] | What can and cannot be concluded? Would you launch, stop, or continue with a modified test? #### What This Part Should Cover - Absolute and relative effect sizes, uncertainty, and the direction of every metric. - Recognition that creator retention and viewer session time are inconclusive in this table. - Serious treatment of the statistically and practically worse report-rate guardrail. - A recommendation tied to risk tolerance, diagnosis, and a pre-specified next decision rule. ### What a Strong Answer Covers - A coherent experiment plan before reading the outcomes. - Correct interpretation without equating “not significant” with “no effect.” - Checks for sample-ratio mismatch, data quality, multiple metrics, and heterogeneous effects. - A decision that balances creator growth with viewer and content-quality consequences. ### Follow-up Questions 1. How would you investigate why reports increased? 2. What if the creation lift came entirely from previously inactive creators? 3. How would interference change your randomization strategy?