Interpret A/B results for video-pin increase

Quick Overview

This question evaluates statistical inference and experimentation skills, including A/B test hypothesis formulation with a predefined MDE, estimation of absolute and relative lift with confidence intervals, proportion testing for CTR and retention, sample ratio mismatch detection, and multiple-comparison adjustments.

Interpret A/B results for video-pin increase

Company: Pinterest

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: medium

Interview Round: Technical Screen

Pinterest is testing increasing the share of video pins in the home feed for NEW users to boost engagement. 50% of eligible new users are assigned to Treatment, 50% to Control. Primary metric: 7-day time spent per user (minutes), one-sided test (Treatment > Control) with MDE = +2% relative to Control. Secondary metrics: overall CTR (total clicks / total impressions) and D7 retention (users retained on day 7 / assigned users). Observed results after 14 days: Assignments (users): Control = 98,750; Treatment = 101,250 Primary metric (per-user): Control mean = 12.00, sd = 8.00; Treatment mean = 12.36, sd = 8.00 CTR (aggregate): Control = 150,000 clicks / 5,000,000 impressions; Treatment = 171,600 clicks / 5,200,000 impressions D7 retention: Control = 21,725 retained / 98,750 users; Treatment = 22,680 retained / 101,250 users Answer the following: 1) State H0 and H1 for the primary metric precisely, including direction and the MDE. 2) Check for sample ratio mismatch (SRM) using a chi-square test at α = 0.001. Should SRM be suspected given the observed assignments for an intended 50/50 split? 3) For the primary metric, compute the absolute lift, relative lift (%), a 95% CI for the difference in means, and the one-sided p-value. Is the result significant at α = 0.05? 4) For CTR and D7 retention, run appropriate two-proportion tests and adjust for multiple comparisons across these two secondary metrics using Holm–Bonferroni at familywise α = 0.05. Which, if any, remain significant? 5) Provide a ship/no-ship recommendation. If you detect SRM or other validity threats (e.g., novelty effects, outliers, country mix shifts), discuss how they impact your decision and what additional diagnostics or guardrail checks you would run before launching.

Quick Answer: This question evaluates statistical inference and experimentation skills, including A/B test hypothesis formulation with a predefined MDE, estimation of absolute and relative lift with confidence intervals, proportion testing for CTR and retention, sample ratio mismatch detection, and multiple-comparison adjustments.

|Home/Analytics & Experimentation/Pinterest
Pinterest logo
Pinterest
Oct 13, 2025, 9:49 PM
mediumData ScientistTechnical ScreenAnalytics & Experimentation
16
0

A/B Test: Increasing Video Pins for New Users

Context

Pinterest ran an online controlled experiment on new users to increase the share of video pins in the home feed. Users were randomized 50/50 into Treatment and Control. The evaluation window is 14 days. The primary metric is 7-day time spent per user (minutes), evaluated with a one-sided test (Treatment > Control) and a planning MDE of +2% relative to Control. Secondary metrics include CTR (clicks/impressions) and D7 retention (users retained on day 7 / assigned users).

Given Data

  • Assignments (users): Control = 98,750; Treatment = 101,250
  • Primary metric (per-user): Control mean = 12.00, sd = 8.00; Treatment mean = 12.36, sd = 8.00
  • CTR (aggregate): Control = 150,000 clicks / 5,000,000 impressions; Treatment = 171,600 clicks / 5,200,000 impressions
  • D7 retention: Control = 21,725 / 98,750; Treatment = 22,680 / 101,250

Tasks

  1. State H0 and H1 for the primary metric precisely, including direction and the MDE.
  2. Check for sample ratio mismatch (SRM) using a chi-square test at α = 0.001. Should SRM be suspected given the observed assignments for an intended 50/50 split?
  3. For the primary metric, compute the absolute lift, relative lift (%), a 95% CI for the difference in means, and the one-sided p-value. Is the result significant at α = 0.05?
  4. For CTR and D7 retention, run appropriate two-proportion tests and adjust for multiple comparisons across these two secondary metrics using Holm–Bonferroni at familywise α = 0.05. Which, if any, remain significant?
  5. Provide a ship/no-ship recommendation. If you detect SRM or other validity threats, discuss their impact and list additional diagnostics/guardrails you would run before launching.
Loading comments...