Design and power an incentive experiment
Overview
This question evaluates experimental design, causal inference, A/B testing and statistical power calculation skills, including defining eligibility, unit of randomization, treatment arms, stratification, metric selection with guardrails, handling noncompliance and interference, and interpreting heterogeneous treatment effects.
Uber
Oct 13, 2025, 9:49 PM
Data Scientist
Onsite
Analytics & Experimentation
0
0
Experiment: Timing and Efficacy of Onboarding Benefits
Context
You operate a two-sided marketplace with supply-side candidates who often complete required paperwork but stall before their first action. You want to test whether offering a benefit (e.g., cash bonus or queue priority) changes the share of paperwork-complete candidates who complete their first action within 14 days of becoming paperwork-complete.
Task
Design an experiment that addresses all of the following:
- Eligibility and exclusions for the test population.
- Unit of randomization.
-
Treatment arms:
- (a) Control
- (b) Bonus-now
- (c) Bonus-after-first-action
- (d) Staged bonus
- Stratification to differentiate bonus-dependent vs. intrinsically motivated candidates.
- Primary and secondary metrics, plus guardrails (quality, fraud, downstream retention).
- Handling noncompliance, spillovers, and interference.
- Stopping rules and decision criteria.
- Power and sample size: Compute the minimum per-arm sample size to detect a +3 percentage-point lift from a 20% baseline at 90% power, α = 0.05 using a two-proportion z-test. Show the formula and provide a numeric result.
- Explain when to roll out benefits broadly vs. target narrowly based on heterogeneous treatment effects.
Solution
Login required
Show