Experiment Design: Increasing Video Pins in Pinterest Home Feed

Context

Pinterest wants to increase the proportion of video pins in the Home Feed to boost user engagement. You are asked to design an end-to-end A/B test to evaluate the impact rigorously.

Tasks

1. Hypotheses
   • State precise hypotheses (H0 and H1) for a single primary metric, including sidedness.
   • List at least two additional hypotheses that capture potential trade-offs (e.g., engagement up but quality down), and specify whether they are one- or two-sided.
2. Experiment Design
   • Define experiment unit, randomization, exposure, and target population (e.g., all users vs. a specific cohort such as US new users defined by signup within 30 days of action date).
   • Address user-to-user interference and content supply constraints introduced by changing the video mix.
3. Metrics
   • Choose a single primary metric.
   • Choose 2–4 secondary metrics.
   • Define guardrail metrics (e.g., hide rate, complaint rate) and justify metric choices for sensitivity and directionality.
4. Segmentation and Heterogeneous Effects
   • Specify segmentation cuts (e.g., pin_format affinity, device, new vs. existing users).
   • Explain how you will handle heterogeneous effects without p-hacking.
5. Rollout, Data Quality, and Stopping Rules
   • Propose a rollout plan (holdout size, ramp schedule, duration) and a power/MDE strategy.
   • Detail data quality checks (e.g., event logging completeness, attribution).
   • Define stopping rules (early stop for harm, futility, or success).