##### Scenario You are responsible for evaluating the lift of a new recommendation algorithm via online experiments. ##### Question Design an A/B test to measure the algorithm’s impact on revenue: define hypotheses, choose unit of randomization, compute required sample size, and detail success metrics. After the experiment you obtain p = 0.08 for revenue lift; interpret this result and recommend whether to ship. Explain how you would estimate causal impact if randomization were not possible; compare methods such as difference-in-differences, propensity score matching, and instrumental variables. ##### Hints Demonstrate knowledge of hypothesis testing, power analysis, Type I/II errors, and causal inference techniques.

This question evaluates experimental design, statistical hypothesis testing, power analysis, outcome metric selection, and causal inference competencies for quantifying an algorithm's causal impact on revenue.

How do I approach Statistics & Math interview questions?

Statistics & Math questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master statistics & math interviews.

What difficulty level is this interview question?

This is a hard difficulty Statistics & Math question, commonly asked during Technical Screen rounds at DoorDash.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at DoorDash during technical interviews.

Design A/B Test to Evaluate Algorithm's Revenue Impact

A/B Test Design for a New Recommendation Algorithm

Context

You are evaluating a new recommendation algorithm in a consumer marketplace app (e.g., browsing menus and item recommendations). The goal is to measure its causal impact on revenue while safeguarding user experience and marketplace health.

Task

Design an online A/B test to measure the algorithm’s impact on revenue. Specifically:

Define clear hypotheses and the primary outcome.
Choose the unit of randomization and describe the assignment plan.
Compute the required sample size (state assumptions, show formula, and a small numeric example).
Define success metrics: primary metric, secondary metrics, and guardrails.
After running the experiment, you obtain p = 0.08 for revenue lift:
- Interpret this result and recommend whether to ship.
If randomization were not possible, explain how you would estimate causal impact and compare methods:
- Difference-in-differences
- Propensity score matching/weighting
- Instrumental variables

Hint: Demonstrate knowledge of hypothesis testing, power analysis, Type I/II errors, and causal inference techniques.

Task

Design an online A/B test to measure the algorithm’s impact on revenue. Specifically:

Define clear hypotheses and the primary outcome.

Choose the unit of randomization and describe the assignment plan.

Compute the required sample size (state assumptions, show formula, and a small numeric example).

Define success metrics: primary metric, secondary metrics, and guardrails.

After running the experiment, you obtain p = 0.08 for revenue lift:

Interpret this result and recommend whether to ship.

If randomization were not possible, explain how you would estimate causal impact and compare methods:

Difference-in-differences
Propensity score matching/weighting
Instrumental variables

Hint: Demonstrate knowledge of hypothesis testing, power analysis, Type I/II errors, and causal inference techniques.

Design A/B Test to Evaluate Algorithm's Revenue Impact

Quick Overview

A/B Test Design for a New Recommendation Algorithm

Context

Task

Solution

Submit Your Answer to Earn 20XP

Design A/B Test to Evaluate Algorithm's Revenue Impact

Quick Overview

A/B Test Design for a New Recommendation Algorithm

Context

Task

Solution

Submit Your Answer to Earn 20XP