Design and assess a video-pin increase experiment
Company: Pinterest
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Technical Screen
##### Question
Pinterest wants to increase the share of video pins surfaced in the Home Feed (e.g., raising video share from a ~30% baseline toward a 45% target, an increase of roughly +10 to +15 percentage points) to boost engagement. Design a rigorous evaluation, then interpret provided results and make a recommendation.
**A) Experiment design**
1. **Unit of randomization & exposure control.** Specify the unit of randomization (user-level vs. session-level) and justify it given network/content-supply interference and feed-ranking spillovers. State how you would cap per-user exposure to the new mix (e.g., from a 30% baseline video share to a 45% target) and how you would ramp.
2. **Metrics.** State a clear primary hypothesis plus at least two alternative hypotheses (e.g., substitution effects, novelty effects, reliability cost). Define a primary success metric and guardrails with exact formulas (e.g., `saves_per_impression`, `clicks_per_impression`, `time_spent_per_user_day`, `complaint_rate = complaints / impressions`, `session_end_rate`, creator churn/follows, crash rate, bandwidth cost per user). Justify each and state win/loss directions.
3. **Power, duration & sequential analysis.** Outline your power / MDE and duration assumptions (alpha, two-sided test, allocation, variance source), and how you will handle sequential looks or peeking (e.g., group sequential boundaries or CUPED for variance reduction). Include an A/A check, a manipulation check (did video share actually move?), and a novelty/fatigue plan (minimum run and long-term holdout).
4. **Quasi-experiment.** If an RCT is infeasible, propose a credible quasi-experiment (e.g., staggered-rollout difference-in-differences with user fixed effects + inverse-propensity weighting, or synthetic control). List identifying assumptions, diagnostics, and sensitivity checks.
**B) Interpret the readout(s)** — assess statistical significance, directionality, and practical significance; call out red flags (SRM, multiple testing, heterogeneous effects, cross-unit metrics). Then recommend whether to **ship, ramp/iterate, or stop**, justify using the metrics, multiple-testing/guardrail considerations, and potential mitigations (e.g., cap video share for sensitive cohorts, rank-quality filters), and state what additional data or follow-up analysis you would run before a full rollout.
**Readout 1 — 14 days, user-level randomization, robust SEs (N ≈ 2.0M users)**
| metric | control | treatment | lift_% | p_value |
|---|---|---|---|---|
| CTR | 3.00% | 3.60% | +20.0 | 0.010 |
| avg_session_sec | 310 | 340 | +9.7 | 0.040 |
| 7d_retention | 28.0% | 27.0% | -3.6 | 0.070 |
| complaint_rate | 0.50% | 0.65% | +30.0 | 0.030 |
**Readout 2 — 14 days (alternate run, with absolute lifts and CIs)**
| metric | control | treatment | lift vs ctrl | p-value | 95% CI |
|---|---|---|---|---|---|
| CTR (clicks/impressions) | 4.70% | 4.85% | +0.15 pp | 0.040 | [+0.01, +0.29] pp |
| Saves per impression | 0.92% | 0.87% | -0.05 pp | 0.090 | [-0.11, +0.01] pp |
| Avg session time | 12.0 m | 12.1 m | +0.8% | 0.200 | [-0.3%, +1.9%] |
| Session crash rate | 1.20% | 1.32% | +0.12 pp | 0.010 | [+0.03, +0.21] pp |
| Exposed users | 200,300 | 199,700 | — | SRM p=0.62 | — |
In both cases: does the CTR win offset the negative guardrail (retention/complaints in Readout 1, crash rate in Readout 2)? If not, what mitigations or follow-ups would you require before full rollout?
Overview: A Pinterest Data Scientist technical-screen question on Analytics & Experimentation: design a rigorous A/B test for increasing video-pin share in the Home Feed — unit of randomization, primary metric and guardrails, power/MDE, sequential analysis, and a quasi-experiment fallback — then interpret two 14-day readouts and recommend ship, iterate, or stop. It rewards balancing engagement lifts (CTR) against guardrail regressions (retention, complaints, crash rate), multiple-testing robustness, and cross-unit (per-impression vs per-session) trade-offs.