This question evaluates expertise in experimentation design and causal inference—covering unit of randomization, interference and spillovers, exposure capping and ramping, precise primary and guardrail metric definitions, power/MDE planning, A/A checks, and quasi-experimental alternatives—within the Analytics & Experimentation domain for a Data Scientist role, requiring a mix of practical application and conceptual reasoning. It is commonly asked to assess the ability to interpret treatment/control readouts, balance engagement lifts against negative guardrails and multiple-testing concerns, and identify necessary follow-up analyses and diagnostics for robust product decisions.
You plan to increase the proportion of video pins surfaced in the home feed. Design a rigorous evaluation and then interpret provided results. A) Experiment design
B) Interpret this 14-day readout (N ≈ 2.0M users; user-level randomization; robust SEs) metric | control_mean | treatment_mean | lift_% | p_value CTR | 3.00% | 3.60% | +20.0 | 0.010 avg_session_sec| 310 | 340 | +9.7 | 0.040 7d_retention | 28.0% | 27.0% | -3.6 | 0.070 complaint_rate | 0.50% | 0.65% | +30.0 | 0.030