Design A/B Test for New Recommendation Algorithm Launch

Q: Design A/B Test for New Recommendation Algorithm Launch

This question evaluates experimental design and causal inference skills, including randomization strategy, metric definition and guardrails, sample size and power calculations, variance reduction techniques, and interpretation of primary versus secondary metrics.

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Question

A/B Test Design: New Recommendation Algorithm

Objective

Design a rigorous A/B test to estimate the incremental impact of a new recommendation algorithm on gross merchandise value (GMV).

Tasks

Define the experimental design and randomization strategy.
Specify the primary metric and guardrail metrics, and justify each choice.
Compute the per-variant sample size given:
- Baseline conversion rate: 3%
- Expected relative lift: 7%
- Significance level: α = 0.05 (two-sided)
- Power: 0.8
Explain how to handle novelty effects and uneven seasonality across groups.
Describe how you would interpret results if the primary GMV metric is flat but secondary engagement metrics improve.

Hints

Discuss randomization strategy, CUPED or other variance reduction, sequential testing, and post-test segmentation.

Design A/B Test for New Recommendation Algorithm Launch

A/B Test Design: New Recommendation Algorithm

Objective

Tasks

Hints

Solution

Comments (0)