Design an RCT for app-open discount

Quick Overview

This question evaluates a data scientist's competency in experimental design and causal inference, including metric definition, statistical power/sample-size reasoning, instrumentation, and managing interference and spillovers in two-sided marketplaces.

Design an RCT for app-open discount

Company: Uber

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: hard

Interview Round: Technical Screen

Your company plans a 'X dollars off on app open' promotion in a two‑sided marketplace. Design a randomized controlled experiment to test success: (1) pick one primary success metric aligned to long‑term profit (e.g., 28‑day incremental gross profit per user) and list guardrails (refund rate, support tickets, seller cancellations, cannibalization of full‑price orders); (2) choose and justify the unit of randomization (user vs device vs account vs session) considering repeated sessions per user, cross‑device logins, households sharing devices, and seller‑side spillovers; (3) specify exposure/eligibility rules, sticky assignment, holdout enforcement, and contamination controls; (4) describe how you will detect/mitigate network effects (geo or cluster randomization, graph clustering, exposure mappings) and how you’d quantify spillovers; (5) provide sample size and test duration with assumptions and MDE, ramp and throttle plan, and stopping rules; (6) list instrumentation/logging needed to compute ITT and TOT (e.g., assignment id, exposure timestamp, eligibility flags, device id, session id).

Quick Answer: This question evaluates a data scientist's competency in experimental design and causal inference, including metric definition, statistical power/sample-size reasoning, instrumentation, and managing interference and spillovers in two-sided marketplaces.

|Home/Analytics & Experimentation/Uber
Uber logo
Uber
Oct 13, 2025, 9:49 PM
hardData ScientistTechnical ScreenAnalytics & Experimentation
7
0

Design an RCT for an "X dollars off on app open" promotion in a two‑sided marketplace

Context

You operate a two‑sided marketplace mobile app (e.g., riders/eaters on the demand side; drivers/couriers/merchants on the supply side). Product wants to show a one‑time, immediate “X dollars off” offer when a user opens the app. You need to design a randomized controlled experiment to evaluate whether the promotion increases long‑term profit without harming marketplace health.

Assume:

  • Users may have repeated sessions and can log in across multiple devices.
  • Households may share devices.
  • Supply is shared across nearby users, so network effects and spillovers are plausible (e.g., surge, wait times, cancellations).

Tasks

  1. Pick one primary success metric aligned to long‑term profit (e.g., 28‑day incremental gross profit per user) and list key guardrails (e.g., refund/chargeback rate, support tickets, seller cancellations, cannibalization of full‑price orders).
  2. Choose and justify the unit of randomization (user vs device vs account vs session), considering repeated sessions per user, cross‑device logins, households that share devices, and seller‑side spillovers.
  3. Specify exposure/eligibility rules, sticky assignment, holdout enforcement, and contamination controls.
  4. Describe how you will detect/mitigate network effects (e.g., geo or cluster randomization, graph clustering, exposure mappings) and how you’d quantify spillovers.
  5. Provide sample size and test duration with assumptions and MDE, a ramp and throttle plan, and stopping rules.
  6. List instrumentation/logging needed to compute ITT and TOT (e.g., assignment id, exposure timestamp, eligibility flags, device id, session id).
Loading comments...