Formulate hypotheses and metrics for video-pin ramp

Quick Overview

This question evaluates experiment-design and data-science competencies including hypothesis specification, metric choice and guardrails, segmentation, interference handling, rollout planning, power/MDE considerations, and data-quality monitoring, and it falls under the Analytics & Experimentation domain.

Formulate hypotheses and metrics for video-pin ramp

Company: Upstart

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: hard

Interview Round: HR Screen

Pinterest plans to increase the proportion of video pins in the home feed to boost engagement. Design the A/B test end-to-end. Tasks: - State precise hypotheses: H0 and H1 for the primary metric; list at least two alternative hypotheses capturing possible trade-offs (e.g., engagement up but quality down), and specify whether they are one- or two-sided. - Define the experiment unit, randomization, exposure, and target population (e.g., all users vs. US new users as defined by signup within 30 days of action date). Address user-to-user interference and content supply constraints. - Choose a single primary metric and 2–4 secondary metrics plus guardrails (e.g., hide rate, complaint rate). Justify each metric’s sensitivity and directionality. - Specify segmentation cuts (e.g., pin_format, device, new vs. existing users) and how you will handle heterogeneous effects without p-hacking. - Detail the rollout plan (holdout size, ramp schedule, duration), data quality checks (event logging completeness, attribution), and stopping rules (early stop for harm, futility, or success).

Quick Answer: This question evaluates experiment-design and data-science competencies including hypothesis specification, metric choice and guardrails, segmentation, interference handling, rollout planning, power/MDE considerations, and data-quality monitoring, and it falls under the Analytics & Experimentation domain.

|Home/Analytics & Experimentation/Upstart
Upstart logo
Upstart
Oct 13, 2025, 9:49 PM
hardData ScientistHR ScreenAnalytics & Experimentation
7
0

Experiment Design: Increasing Video Pins in Pinterest Home Feed

Context

Pinterest wants to increase the proportion of video pins in the Home Feed to boost user engagement. You are asked to design an end-to-end A/B test to evaluate the impact rigorously.

Tasks

  1. Hypotheses
    • State precise hypotheses (H0 and H1) for a single primary metric, including sidedness.
    • List at least two additional hypotheses that capture potential trade-offs (e.g., engagement up but quality down), and specify whether they are one- or two-sided.
  2. Experiment Design
    • Define experiment unit, randomization, exposure, and target population (e.g., all users vs. a specific cohort such as US new users defined by signup within 30 days of action date).
    • Address user-to-user interference and content supply constraints introduced by changing the video mix.
  3. Metrics
    • Choose a single primary metric.
    • Choose 2–4 secondary metrics.
    • Define guardrail metrics (e.g., hide rate, complaint rate) and justify metric choices for sensitivity and directionality.
  4. Segmentation and Heterogeneous Effects
    • Specify segmentation cuts (e.g., pin_format affinity, device, new vs. existing users).
    • Explain how you will handle heterogeneous effects without p-hacking.
  5. Rollout, Data Quality, and Stopping Rules
    • Propose a rollout plan (holdout size, ramp schedule, duration) and a power/MDE strategy.
    • Detail data quality checks (e.g., event logging completeness, attribution).
    • Define stopping rules (early stop for harm, futility, or success).
Loading comments...