Design and interpret video-pins experiment results
Company: Pinterest
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Technical Screen
Pinterest wants to increase the share of video pins in Home Feed by +10 percentage points to boost engagement. (1) State a clear hypothesis and at least two alternative hypotheses (e.g., substitution effects, novelty effects). (2) Choose a primary success metric and guardrails; justify each (consider CTR, saves/impression, session length, creator follows, crash rate, content complaints). (3) Using the summary below, interpret the outcome: statistical significance, directionality, and practical significance; call out any red flags (SRM, multiple testing, heterogeneous effects). Finally, recommend whether to ship, ramp, or stop, and propose next steps.
Experiment summary (14 days):
Metric | Control | Treatment | Lift vs Ctrl | p-value | 95% CI
CTR (clicks/impressions) | 4.70% | 4.85% | +0.15 pp | 0.040 | [+0.01, +0.29] pp
Saves per impression | 0.92% | 0.87% | -0.05 pp | 0.090 | [-0.11, +0.01] pp
Avg session time | 12.0 m | 12.1 m | +0.8% | 0.200 | [-0.3%, +1.9%]
Session crash rate | 1.20% | 1.32% | +0.12 pp | 0.010 | [+0.03, +0.21] pp
Exposed users | 200,300 | 199,700 | — | SRM p=0.62 | —
Address: whether the CTR win offsets the crash-rate increase; if not, what mitigations or follow-ups would you require before full rollout?
Quick Answer: This question evaluates experiment design, causal inference, and metrics-driven decision-making skills within Analytics & Experimentation for Data Scientist roles, covering hypothesis formulation, selection of a primary success metric and guardrails, and the use of manipulation checks such as video share of impressions.