Design A/B Test to Evaluate New Video-Feed Feature
Overview
This question evaluates a candidate's skills in experimental design, statistical inference, metric definition, and product analytics for measuring a new short-video feed feature, with attention to randomization, cross-device consistency, sample size, guardrail metrics, and interference risks.
Meta
Aug 4, 2025, 10:55 AM
Data Scientist
Onsite
Analytics & Experimentation
0
0
Scenario
A consumer social-media app is launching a short‑video feed (TikTok-style). A newly added feed feature (e.g., UI change, ranking tweak, or interaction control) may alter how users consume and engage with videos.
Task
Design an end-to-end A/B test to measure the feature's impact. Address:
- Experiment unit and eligibility criteria.
- Randomization and bucketing (including cross-device consistency).
- Primary and secondary success metrics with definitions and rationale.
- Guardrail/health metrics and thresholds.
- Sample-size and test-duration needs; state assumptions and how you would compute MDE/sample size.
- Handling novelty effects and seasonality.
- Monitoring, analysis plan, and decision criteria.
- Risks (e.g., interference/network effects) and mitigations.
Hints: Consider activation, retention, engagement, and negative outcomes. Outline the experiment design and what would make you ship, iterate, or roll back.
Solution
Login required
Show