Interpret A/B results for video-pin increase
Company: Pinterest
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Technical Screen
Pinterest is testing increasing the share of video pins in the home feed for NEW users to boost engagement. 50% of eligible new users are assigned to Treatment, 50% to Control. Primary metric: 7-day time spent per user (minutes), one-sided test (Treatment > Control) with MDE = +2% relative to Control. Secondary metrics: overall CTR (total clicks / total impressions) and D7 retention (users retained on day 7 / assigned users).
Observed results after 14 days:
Assignments (users): Control = 98,750; Treatment = 101,250
Primary metric (per-user): Control mean = 12.00, sd = 8.00; Treatment mean = 12.36, sd = 8.00
CTR (aggregate): Control = 150,000 clicks / 5,000,000 impressions; Treatment = 171,600 clicks / 5,200,000 impressions
D7 retention: Control = 21,725 retained / 98,750 users; Treatment = 22,680 retained / 101,250 users
Answer the following:
1) State H0 and H1 for the primary metric precisely, including direction and the MDE.
2) Check for sample ratio mismatch (SRM) using a chi-square test at α = 0.001. Should SRM be suspected given the observed assignments for an intended 50/50 split?
3) For the primary metric, compute the absolute lift, relative lift (%), a 95% CI for the difference in means, and the one-sided p-value. Is the result significant at α = 0.05?
4) For CTR and D7 retention, run appropriate two-proportion tests and adjust for multiple comparisons across these two secondary metrics using Holm–Bonferroni at familywise α = 0.05. Which, if any, remain significant?
5) Provide a ship/no-ship recommendation. If you detect SRM or other validity threats (e.g., novelty effects, outliers, country mix shifts), discuss how they impact your decision and what additional diagnostics or guardrail checks you would run before launching.
Quick Answer: This question evaluates statistical inference and experimentation skills, including A/B test hypothesis formulation with a predefined MDE, estimation of absolute and relative lift with confidence intervals, proportion testing for CTR and retention, sample ratio mismatch detection, and multiple-comparison adjustments.