Design A/B Test for New Recommendation Algorithm Launch
Company: Snapchat
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Onsite
##### Scenario
Product team wants to launch a new recommendation algorithm and asks for a rigorous A/B test plan.
##### Question
Design an A/B test to measure the uplift of the new recommender on GMV. Which primary and guardrail metrics will you track and why? Compute required sample size assuming 3% baseline conversion, 7% relative lift, α=0.05, power=0.8. How would you address novelty effect and uneven seasonality across groups? Explain how you would interpret results if the primary metric is flat but secondary engagement metrics improve.
##### Hints
Talk randomization, CUPED, sequential testing, and post-test segmentation.
Quick Answer: This question evaluates experimental design and causal inference skills, including randomization strategy, metric definition and guardrails, sample size and power calculations, variance reduction techniques, and interpretation of primary versus secondary metrics.