Design and analyze end-to-end A/B test
Company: Meta
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: hard
Interview Round: Technical Screen
You are changing the Friend Recommendation algorithm to prioritize higher-quality connections. Design and analyze an end-to-end A/B test: 1) define primary success metrics and at most two counter-metrics, plus guardrails (latency, crash, abuse), and a cannibalization read on Stories; 2) pre-register MDEs and the minimum launch bar; 3) choose randomization unit and bucketing consistency for long-lived effects; 4) plan duration and ramp strategy with a holdback for long-term read; 5) specify how you’ll handle multiple looks and stopping rules to avoid alpha inflation; 6) interpret this realistic outcome: MAU +5% (p=0.03), per-user comments −3% (p=0.07), time spent +1% (p=0.20), Story creation −2% (p=0.01). Would you ship, partially ship, iterate, or stop? Justify to product, growth, and integrity stakeholders in one paragraph each.
Quick Answer: This question evaluates experimentation design and analysis competencies for A/B testing, including metric selection, statistical interpretation, integrity and performance guardrails, and cross-functional stakeholder communication.