Design and power an incentive experiment

Quick Overview

This question evaluates experimental design, causal inference, A/B testing and statistical power calculation skills, including defining eligibility, unit of randomization, treatment arms, stratification, metric selection with guardrails, handling noncompliance and interference, and interpreting heterogeneous treatment effects.

Design and power an incentive experiment

Company: Uber

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: hard

Interview Round: Onsite

Offer timing and efficacy of benefits: Design an experiment to test whether a bonus (e.g., cash or queue priority) increases the share of paperwork-complete candidates who complete their first action within 14 days. Specify eligibility/exclusions; unit of randomization; treatment arms (control, bonus-now, bonus-after-first-action, staged bonus); stratification to separate bonus-dependent vs. intrinsically motivated users; primary/secondary metrics and guardrails (quality, fraud, downstream retention); handling noncompliance, spillovers, and interference; stopping rules and decision criteria. Compute the minimum per-arm sample size to detect a +3 percentage-point lift from a 20% baseline at 90% power, α=0.05 using a two-proportion z-test—show formula and a numeric result. Explain when to roll out benefits broadly vs. target narrowly based on heterogeneous treatment effects.

Quick Answer: This question evaluates experimental design, causal inference, A/B testing and statistical power calculation skills, including defining eligibility, unit of randomization, treatment arms, stratification, metric selection with guardrails, handling noncompliance and interference, and interpreting heterogeneous treatment effects.

|Home/Analytics & Experimentation/Uber
Uber logo
Uber
Oct 13, 2025, 9:49 PM
hardData ScientistOnsiteAnalytics & Experimentation
12
0

Experiment: Timing and Efficacy of Onboarding Benefits

Context

You operate a two-sided marketplace with supply-side candidates who often complete required paperwork but stall before their first action. You want to test whether offering a benefit (e.g., cash bonus or queue priority) changes the share of paperwork-complete candidates who complete their first action within 14 days of becoming paperwork-complete.

Task

Design an experiment that addresses all of the following:

  1. Eligibility and exclusions for the test population.
  2. Unit of randomization.
  3. Treatment arms:
    • (a) Control
    • (b) Bonus-now
    • (c) Bonus-after-first-action
    • (d) Staged bonus
  4. Stratification to differentiate bonus-dependent vs. intrinsically motivated candidates.
  5. Primary and secondary metrics, plus guardrails (quality, fraud, downstream retention).
  6. Handling noncompliance, spillovers, and interference.
  7. Stopping rules and decision criteria.
  8. Power and sample size: Compute the minimum per-arm sample size to detect a +3 percentage-point lift from a 20% baseline at 90% power, α = 0.05 using a two-proportion z-test. Show the formula and provide a numeric result.
  9. Explain when to roll out benefits broadly vs. target narrowly based on heterogeneous treatment effects.
Loading comments...