You plan to increase the proportion of video pins surfaced in the home feed. Design a rigorous evaluation and then interpret provided results. A) Experiment design 1) Specify the unit of randomization (user-level vs. session-level) and justify considering network/content-supply interference and feed-ranking spillovers. State how to cap per-user exposure to the new mix (e.g., from 30% baseline video share to 45% target) and how to ramp. 2) Define primary success metrics with exact formulas (e.g., saves_per_user_day, clicks_per_impression, time_spent_per_user_day) and guardrails (e.g., complaint_rate = complaints/impressions, session_end_rate, creator churn, bandwidth cost per user). State win/loss directions. 3) Outline power/MDE and duration assumptions (alpha, two-sided test, allocation, variance source), and how you will handle sequential looks or peeking (e.g., group sequential or CUPED). Include an A/A check and novelty/fatigue plan (minimum run and long-term holdout). 4) If an RCT is infeasible, propose a credible quasi-experiment (e.g., staggered rollout DiD with user fixed effects + inverse-propensity weighting, or synthetic control). List identifying assumptions, diagnostics, and sensitivity checks. B) Interpret this 14-day readout (N ≈ 2.0M users; user-level randomization; robust SEs) metric | control_mean | treatment_mean | lift_% | p_value CTR | 3.00% | 3.60% | +20.0 | 0.010 avg_session_sec| 310 | 340 | +9.7 | 0.040 7d_retention | 28.0% | 27.0% | -3.6 | 0.070 complaint_rate | 0.50% | 0.65% | +30.0 | 0.030 - Provide a clear recommendation to the PM: roll out, iterate, or stop? Justify using the metrics above, multiple-testing/guardrail considerations, and potential mitigations (e.g., cap video share for sensitive cohorts, rank-quality filters). Also state what additional data or follow-up analysis you would run before a full rollout.

This question evaluates expertise in experimentation design and causal inference—covering unit of randomization, interference and spillovers, exposure capping and ramping, precise primary and guardrail metric definitions, power/MDE planning, A/A checks, and quasi-experimental alternatives—within the Analytics & Experimentation domain for a Data Scientist role, requiring a mix of practical application and conceptual reasoning. It is commonly asked to assess the ability to interpret treatment/control readouts, balance engagement lifts against negative guardrails and multiple-testing concerns, and identify necessary follow-up analyses and diagnostics for robust product decisions.

How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a Medium difficulty Analytics & Experimentation question, commonly asked during Technical Screen rounds at Pinterest.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Pinterest during technical interviews.

Design and assess video-pin increase experiment

You plan to increase the proportion of video pins surfaced in the home feed. Design a rigorous evaluation and then interpret provided results. A) Experiment design

Specify the unit of randomization (user-level vs. session-level) and justify considering network/content-supply interference and feed-ranking spillovers. State how to cap per-user exposure to the new mix (e.g., from 30% baseline video share to 45% target) and how to ramp.
Define primary success metrics with exact formulas (e.g., saves_per_user_day, clicks_per_impression, time_spent_per_user_day) and guardrails (e.g., complaint_rate = complaints/impressions, session_end_rate, creator churn, bandwidth cost per user). State win/loss directions.
Outline power/MDE and duration assumptions (alpha, two-sided test, allocation, variance source), and how you will handle sequential looks or peeking (e.g., group sequential or CUPED). Include an A/A check and novelty/fatigue plan (minimum run and long-term holdout).
If an RCT is infeasible, propose a credible quasi-experiment (e.g., staggered rollout DiD with user fixed effects + inverse-propensity weighting, or synthetic control). List identifying assumptions, diagnostics, and sensitivity checks.

B) Interpret this 14-day readout (N ≈ 2.0M users; user-level randomization; robust SEs) metric | control_mean | treatment_mean | lift_% | p_value CTR | 3.00% | 3.60% | +20.0 | 0.010 avg_session_sec| 310 | 340 | +9.7 | 0.040 7d_retention | 28.0% | 27.0% | -3.6 | 0.070 complaint_rate | 0.50% | 0.65% | +30.0 | 0.030

Provide a clear recommendation to the PM: roll out, iterate, or stop? Justify using the metrics above, multiple-testing/guardrail considerations, and potential mitigations (e.g., cap video share for sensitive cohorts, rank-quality filters). Also state what additional data or follow-up analysis you would run before a full rollout.

You plan to increase the proportion of video pins surfaced in the home feed. Design a rigorous evaluation and then interpret provided results. A) Experiment design

Specify the unit of randomization (user-level vs. session-level) and justify considering network/content-supply interference and feed-ranking spillovers. State how to cap per-user exposure to the new mix (e.g., from 30% baseline video share to 45% target) and how to ramp.
Define primary success metrics with exact formulas (e.g., saves_per_user_day, clicks_per_impression, time_spent_per_user_day) and guardrails (e.g., complaint_rate = complaints/impressions, session_end_rate, creator churn, bandwidth cost per user). State win/loss directions.
Outline power/MDE and duration assumptions (alpha, two-sided test, allocation, variance source), and how you will handle sequential looks or peeking (e.g., group sequential or CUPED). Include an A/A check and novelty/fatigue plan (minimum run and long-term holdout).
If an RCT is infeasible, propose a credible quasi-experiment (e.g., staggered rollout DiD with user fixed effects + inverse-propensity weighting, or synthetic control). List identifying assumptions, diagnostics, and sensitivity checks.

Provide a clear recommendation to the PM: roll out, iterate, or stop? Justify using the metrics above, multiple-testing/guardrail considerations, and potential mitigations (e.g., cap video share for sensitive cohorts, rank-quality filters). Also state what additional data or follow-up analysis you would run before a full rollout.

Design and assess video-pin increase experiment

Quick Overview

Comments (0)

Design and assess video-pin increase experiment

Quick Overview

Comments (0)