Design A/B Test to Evaluate Algorithm's Revenue Impact

Q: Design A/B Test to Evaluate Algorithm's Revenue Impact

This question evaluates experimental design, statistical hypothesis testing, power analysis, outcome metric selection, and causal inference competencies for quantifying an algorithm's causal impact on revenue.

Q: How do I approach Statistics & Math interview questions?

Statistics & Math questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master statistics & math interviews.

Question

A/B Test Design for a New Recommendation Algorithm

Context

You are evaluating a new recommendation algorithm in a consumer marketplace app (e.g., browsing menus and item recommendations). The goal is to measure its causal impact on revenue while safeguarding user experience and marketplace health.

Task

Design an online A/B test to measure the algorithm’s impact on revenue. Specifically:

Define clear hypotheses and the primary outcome.
Choose the unit of randomization and describe the assignment plan.
Compute the required sample size (state assumptions, show formula, and a small numeric example).
Define success metrics: primary metric, secondary metrics, and guardrails.
After running the experiment, you obtain p = 0.08 for revenue lift:
- Interpret this result and recommend whether to ship.
If randomization were not possible, explain how you would estimate causal impact and compare methods:
- Difference-in-differences
- Propensity score matching/weighting
- Instrumental variables

Hint: Demonstrate knowledge of hypothesis testing, power analysis, Type I/II errors, and causal inference techniques.

Design A/B Test to Evaluate Algorithm's Revenue Impact

A/B Test Design for a New Recommendation Algorithm

Context

Task

Solution

Comments (0)