Design A/B Test for New Recommendation Algorithm Launch
Overview
This question evaluates experimental design and causal inference skills, including randomization strategy, metric definition and guardrails, sample size and power calculations, variance reduction techniques, and interpretation of primary versus secondary metrics.
Snapchat
Aug 4, 2025, 10:55 AM
Data Scientist
Onsite
Analytics & Experimentation
65
0
A/B Test Design: New Recommendation Algorithm
Objective
Design a rigorous A/B test to estimate the incremental impact of a new recommendation algorithm on gross merchandise value (GMV).
Tasks
- Define the experimental design and randomization strategy.
- Specify the primary metric and guardrail metrics, and justify each choice.
-
Compute the per-variant sample size given:
- Baseline conversion rate: 3%
- Expected relative lift: 7%
- Significance level: α = 0.05 (two-sided)
- Power: 0.8
- Explain how to handle novelty effects and uneven seasonality across groups.
- Describe how you would interpret results if the primary GMV metric is flat but secondary engagement metrics improve.
Hints
- Discuss randomization strategy, CUPED or other variance reduction, sequential testing, and post-test segmentation.
Solution
Login required
Show