PracHub
QuestionsPremiumLearningGuidesCheatsheetNEWCoaches
|Home/Analytics & Experimentation/Pinterest

Interpret A/B results for video-pin increase

Last updated: Mar 29, 2026

Quick Overview

This question evaluates statistical inference and experimentation skills, including A/B test hypothesis formulation with a predefined MDE, estimation of absolute and relative lift with confidence intervals, proportion testing for CTR and retention, sample ratio mismatch detection, and multiple-comparison adjustments.

  • medium
  • Pinterest
  • Analytics & Experimentation
  • Data Scientist

Interpret A/B results for video-pin increase

Company: Pinterest

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: medium

Interview Round: Technical Screen

Pinterest is testing increasing the share of video pins in the home feed for NEW users to boost engagement. 50% of eligible new users are assigned to Treatment, 50% to Control. Primary metric: 7-day time spent per user (minutes), one-sided test (Treatment > Control) with MDE = +2% relative to Control. Secondary metrics: overall CTR (total clicks / total impressions) and D7 retention (users retained on day 7 / assigned users). Observed results after 14 days: Assignments (users): Control = 98,750; Treatment = 101,250 Primary metric (per-user): Control mean = 12.00, sd = 8.00; Treatment mean = 12.36, sd = 8.00 CTR (aggregate): Control = 150,000 clicks / 5,000,000 impressions; Treatment = 171,600 clicks / 5,200,000 impressions D7 retention: Control = 21,725 retained / 98,750 users; Treatment = 22,680 retained / 101,250 users Answer the following: 1) State H0 and H1 for the primary metric precisely, including direction and the MDE. 2) Check for sample ratio mismatch (SRM) using a chi-square test at α = 0.001. Should SRM be suspected given the observed assignments for an intended 50/50 split? 3) For the primary metric, compute the absolute lift, relative lift (%), a 95% CI for the difference in means, and the one-sided p-value. Is the result significant at α = 0.05? 4) For CTR and D7 retention, run appropriate two-proportion tests and adjust for multiple comparisons across these two secondary metrics using Holm–Bonferroni at familywise α = 0.05. Which, if any, remain significant? 5) Provide a ship/no-ship recommendation. If you detect SRM or other validity threats (e.g., novelty effects, outliers, country mix shifts), discuss how they impact your decision and what additional diagnostics or guardrail checks you would run before launching.

Quick Answer: This question evaluates statistical inference and experimentation skills, including A/B test hypothesis formulation with a predefined MDE, estimation of absolute and relative lift with confidence intervals, proportion testing for CTR and retention, sample ratio mismatch detection, and multiple-comparison adjustments.

Related Interview Questions

  • How would you evaluate a carousel launch? - Pinterest (medium)
  • How to evaluate a new Carousel feature - Pinterest (easy)
  • Evaluate Fresh Content and Video Experiments - Pinterest (medium)
  • Design and Evaluate a Home Carousel - Pinterest (medium)
  • Evaluate Carousel and Billboard Lift - Pinterest (medium)
Pinterest logo
Pinterest
Oct 13, 2025, 9:49 PM
Data Scientist
Technical Screen
Analytics & Experimentation
7
0

A/B Test: Increasing Video Pins for New Users

Context

Pinterest ran an online controlled experiment on new users to increase the share of video pins in the home feed. Users were randomized 50/50 into Treatment and Control. The evaluation window is 14 days. The primary metric is 7-day time spent per user (minutes), evaluated with a one-sided test (Treatment > Control) and a planning MDE of +2% relative to Control. Secondary metrics include CTR (clicks/impressions) and D7 retention (users retained on day 7 / assigned users).

Given Data

  • Assignments (users): Control = 98,750; Treatment = 101,250
  • Primary metric (per-user): Control mean = 12.00, sd = 8.00; Treatment mean = 12.36, sd = 8.00
  • CTR (aggregate): Control = 150,000 clicks / 5,000,000 impressions; Treatment = 171,600 clicks / 5,200,000 impressions
  • D7 retention: Control = 21,725 / 98,750; Treatment = 22,680 / 101,250

Tasks

  1. State H0 and H1 for the primary metric precisely, including direction and the MDE.
  2. Check for sample ratio mismatch (SRM) using a chi-square test at α = 0.001. Should SRM be suspected given the observed assignments for an intended 50/50 split?
  3. For the primary metric, compute the absolute lift, relative lift (%), a 95% CI for the difference in means, and the one-sided p-value. Is the result significant at α = 0.05?
  4. For CTR and D7 retention, run appropriate two-proportion tests and adjust for multiple comparisons across these two secondary metrics using Holm–Bonferroni at familywise α = 0.05. Which, if any, remain significant?
  5. Provide a ship/no-ship recommendation. If you detect SRM or other validity threats, discuss their impact and list additional diagnostics/guardrails you would run before launching.

Solution

Show

Comments (0)

Sign in to leave a comment

Loading comments...

Browse More Questions

More Analytics & Experimentation•More Pinterest•More Data Scientist•Pinterest Data Scientist•Pinterest Analytics & Experimentation•Data Scientist Analytics & Experimentation
PracHub

Master your tech interviews with 7,500+ real questions from top companies.

Product

  • Questions
  • Learning Tracks
  • Interview Guides
  • Resources
  • Premium
  • For Universities
  • Student Access

Browse

  • By Company
  • By Role
  • By Category
  • Topic Hubs
  • SQL Questions
  • Compare Platforms
  • Discord Community

Support

  • support@prachub.com
  • (916) 541-4762

Legal

  • Privacy Policy
  • Terms of Service
  • About Us

© 2026 PracHub. All rights reserved.