Design and interpret video-pins experiment results
Overview
This question evaluates experiment design, causal inference, and metrics-driven decision-making skills within Analytics & Experimentation for Data Scientist roles, covering hypothesis formulation, selection of a primary success metric and guardrails, and the use of manipulation checks such as video share of impressions.
Pinterest
Oct 13, 2025, 9:49 PM
Data Scientist
Technical Screen
Analytics & Experimentation
0
0
A/B Test: Increasing Video Pins in Home Feed by +10 pp
Context: You ran a 14-day A/B test that increases the share of video pins in Home Feed by +10 percentage points to boost engagement. Evaluate hypotheses, define success and guardrails, interpret results, and make a ship/ramp/stop decision.
Tasks
-
Hypotheses
- State one clear primary hypothesis and at least two alternative hypotheses (e.g., substitution, novelty).
-
Metrics
- Select one primary success metric and appropriate guardrails. Justify each choice. Consider: CTR, saves/impression, session length, creator follows, crash rate, content complaints.
-
Results Interpretation
- Using the summary below, assess: statistical significance, directionality, and practical significance.
- Call out red flags: SRM, multiple testing, heterogeneous effects, and any missing manipulation checks.
- Specifically address whether the CTR win offsets the crash-rate increase; if not, outline required mitigations or follow-ups before full rollout.
-
Decision and Next Steps
- Recommend whether to ship, ramp, or stop, and propose concrete next steps.
Experiment Summary (14 days)
| Metric | Control | Treatment | Lift vs Ctrl | p-value | 95% CI |
|---|---|---|---|---|---|
| CTR (clicks/impressions) | 4.70% | 4.85% | +0.15 pp | 0.040 | [+0.01, +0.29] pp |
| Saves per impression | 0.92% | 0.87% | -0.05 pp | 0.090 | [-0.11, +0.01] pp |
| Avg session time | 12.0 m | 12.1 m | +0.8% | 0.200 | [-0.3%, +1.9%] |
| Session crash rate | 1.20% | 1.32% | +0.12 pp | 0.010 | [+0.03, +0.21] pp |
| Exposed users | 200,300 | 199,700 | — | SRM p=0.62 | — |
Note: The goal was to increase the video share in Home Feed by +10 pp. Include a manipulation check for “video share of impressions” in analysis.
Solution
Login required
Show