Design A/B Test to Evaluate Algorithm's Revenue Impact

Quick Overview

DoorDash statistics and experimentation prompt on A/B testing a recommendation algorithm's revenue impact, covering hypotheses, randomization, sample size, heavy-tailed revenue, p=0.08 interpretation, and causal alternatives.

Design A/B Test to Evaluate Algorithm's Revenue Impact

Company: DoorDash

Role: Data Scientist

Category: Statistics & Math

Difficulty: hard

Interview Round: Technical Screen

##### Scenario You are responsible for evaluating the lift of a new recommendation algorithm via online experiments. ##### Question Design an A/B test to measure the algorithm’s impact on revenue: define hypotheses, choose unit of randomization, compute required sample size, and detail success metrics. After the experiment you obtain p = 0.08 for revenue lift; interpret this result and recommend whether to ship. Explain how you would estimate causal impact if randomization were not possible; compare methods such as difference-in-differences, propensity score matching, and instrumental variables. ##### Hints Demonstrate knowledge of hypothesis testing, power analysis, Type I/II errors, and causal inference techniques.

Quick Answer: DoorDash statistics and experimentation prompt on A/B testing a recommendation algorithm's revenue impact, covering hypotheses, randomization, sample size, heavy-tailed revenue, p=0.08 interpretation, and causal alternatives.

|Home/Statistics & Math/DoorDash
DoorDash logo
DoorDash
Jul 12, 2025, 6:59 PM
hardData ScientistTechnical ScreenStatistics & Math
58
0

A/B Test a Recommendation Algorithm's Revenue Impact

You are evaluating a new recommendation algorithm in a consumer marketplace app. The goal is to measure its causal impact on revenue while protecting user experience and marketplace health.

Constraints & Assumptions

  • Use an online randomized experiment when possible.
  • Revenue is likely heavy-tailed and zero-inflated.
  • Pre-register hypotheses, metrics, and decision rules.
  • Interpret an observed p-value of 0.08 for revenue lift.
  • If randomization is impossible, compare causal inference alternatives.

Clarifying Questions to Ask Guidance

  • What revenue metric matters: revenue per user, contribution margin, GMV, or long-term value?
  • What is the randomization unit: user, session, market, or merchant?
  • Can the recommendation algorithm learn during the test?
  • What MDE and Type I/II error trade-off is acceptable?

What a Strong Answer Covers Guidance

  • Hypotheses for revenue lift and a primary metric such as revenue per user over a fixed window.
  • Randomization unit and assignment plan, with attention to persistent exposure and interference.
  • Sample-size calculation using baseline variance, MDE, alpha, and power, plus variance reduction such as CUPED.
  • Secondary metrics and guardrails: conversion, order frequency, AOV, retention, latency, cancellation, support, diversity, fairness, and marketplace balance.
  • Interpretation of p = 0.08 : not statistically significant at alpha 0.05 unless a different threshold was pre-registered; consider effect size, confidence interval, power, guardrails, and business risk before shipping.
  • Decision options: do not ship broadly, extend test, run targeted rollout, or ship only if low risk and supported by pre-registered decision rules.
  • Non-randomized alternatives: diff-in-diff, propensity score matching/weighting, instrumental variables, synthetic control, and their assumptions and limitations.

Follow-up Questions Guidance

  • What if p = 0.08 but the confidence interval excludes large harm?
  • How would you handle revenue outliers?
  • Why is p-value alone insufficient for launch decisions?
  • When would instrumental variables be credible?
Loading comments...