Design A/B Test to Evaluate Algorithm's Revenue Impact
Overview
This question evaluates experimental design, statistical hypothesis testing, power analysis, outcome metric selection, and causal inference competencies for quantifying an algorithm's causal impact on revenue.
DoorDash
Jul 12, 2025, 6:59 PM
Data Scientist
Technical Screen
Statistics & Math
50
0
A/B Test Design for a New Recommendation Algorithm
Context
You are evaluating a new recommendation algorithm in a consumer marketplace app (e.g., browsing menus and item recommendations). The goal is to measure its causal impact on revenue while safeguarding user experience and marketplace health.
Task
Design an online A/B test to measure the algorithm’s impact on revenue. Specifically:
- Define clear hypotheses and the primary outcome.
- Choose the unit of randomization and describe the assignment plan.
- Compute the required sample size (state assumptions, show formula, and a small numeric example).
- Define success metrics: primary metric, secondary metrics, and guardrails.
-
After running the experiment, you obtain p = 0.08 for revenue lift:
- Interpret this result and recommend whether to ship.
-
If randomization were not possible, explain how you would estimate causal impact and compare methods:
- Difference-in-differences
- Propensity score matching/weighting
- Instrumental variables
Hint: Demonstrate knowledge of hypothesis testing, power analysis, Type I/II errors, and causal inference techniques.
Solution
Login required
Show