Design A/B Test for New Recommendation Algorithm Launch
A/B Test Design: New Recommendation Algorithm
Objective
Design a rigorous A/B test to estimate the incremental impact of a new recommendation algorithm on gross merchandise value (GMV).
Tasks
-
Define the experimental design and randomization strategy.
-
Specify the primary metric and guardrail metrics, and justify each choice.
-
Compute the per-variant sample size given:
-
Baseline conversion rate: 3%
-
Expected relative lift: 7%
-
Significance level: α = 0.05 (two-sided)
-
Power: 0.8
-
Explain how to handle novelty effects and uneven seasonality across groups.
-
Describe how you would interpret results if the primary GMV metric is flat but secondary engagement metrics improve.
Hints
-
Discuss randomization strategy, CUPED or other variance reduction, sequential testing, and post-test segmentation.
Constraints & Assumptions
-
Preserve the scope, facts, inputs, and requested outputs from the prompt above.
-
If the prompt leaves a detail unspecified, state a reasonable assumption before relying on it.
-
Keep the answer interview-ready: concise enough to present, but concrete enough to implement or evaluate.
Clarifying Questions to Ask
-
Clarify the business objective, unit of analysis, time window, exposure definition, and primary metric.
-
State assumptions about instrumentation, randomization, sample size, and data quality.
-
Separate descriptive analysis from causal claims.
What a Strong Answer Covers
-
A metric framework with primary, guardrail, and diagnostic metrics.
-
A credible analysis or experiment design with clear assumptions and bias checks.
-
SQL/statistical logic for segmentation, variance, confidence, and data validation where relevant.
-
An actionable recommendation that explains trade-offs and next steps.
Follow-up Questions
-
What sanity checks would you run before trusting the result?
-
How would you handle novelty effects, seasonality, or selection bias?
-
What decision would you make if metrics disagree?