Design and interpret video-pins experiment results

Q: Design and interpret video-pins experiment results

This question evaluates experiment design, causal inference, and metrics-driven decision-making skills within Analytics & Experimentation for Data Scientist roles, covering hypothesis formulation, selection of a primary success metric and guardrails, and the use of manipulation checks such as video share of impressions.

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Q: What difficulty level is this interview question?

This is a medium difficulty Analytics & Experimentation question, commonly asked during Technical Screen rounds at Pinterest.

Q: What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Pinterest during technical interviews.

Question

A/B Test: Increasing Video Pins in Home Feed by +10 pp

Context: You ran a 14-day A/B test that increases the share of video pins in Home Feed by +10 percentage points to boost engagement. Evaluate hypotheses, define success and guardrails, interpret results, and make a ship/ramp/stop decision.

Tasks

Hypotheses
- State one clear primary hypothesis and at least two alternative hypotheses (e.g., substitution, novelty).
Metrics
- Select one primary success metric and appropriate guardrails. Justify each choice. Consider: CTR, saves/impression, session length, creator follows, crash rate, content complaints.
Results Interpretation
- Using the summary below, assess: statistical significance, directionality, and practical significance.
- Call out red flags: SRM, multiple testing, heterogeneous effects, and any missing manipulation checks.
- Specifically address whether the CTR win offsets the crash-rate increase; if not, outline required mitigations or follow-ups before full rollout.
Decision and Next Steps
- Recommend whether to ship, ramp, or stop, and propose concrete next steps.

Experiment Summary (14 days)

Metric	Control	Treatment	Lift vs Ctrl	p-value	95% CI
CTR (clicks/impressions)	4.70%	4.85%	+0.15 pp	0.040	[+0.01, +0.29] pp
Saves per impression	0.92%	0.87%	-0.05 pp	0.090	[-0.11, +0.01] pp
Avg session time	12.0 m	12.1 m	+0.8%	0.200	[-0.3%, +1.9%]
Session crash rate	1.20%	1.32%	+0.12 pp	0.010	[+0.03, +0.21] pp
Exposed users	200,300	199,700	—	SRM p=0.62	—

Note: The goal was to increase the video share in Home Feed by +10 pp. Include a manipulation check for “video share of impressions” in analysis.

Design and interpret video-pins experiment results

Quick Overview

A/B Test: Increasing Video Pins in Home Feed by +10 pp

Tasks

Experiment Summary (14 days)

Solution

Comments (0)