Design an RCT for an "X dollars off on app open" promotion in a two‑sided marketplace
Context
You operate a two‑sided marketplace mobile app (e.g., riders/eaters on the demand side; drivers/couriers/merchants on the supply side). Product wants to show a one‑time, immediate “X dollars off” offer when a user opens the app. You need to design a randomized controlled experiment to evaluate whether the promotion increases long‑term profit without harming marketplace health.
Assume:
-
Users may have repeated sessions and can log in across multiple devices.
-
Households may share devices.
-
Supply is shared across nearby users, so network effects and spillovers are plausible (e.g., surge, wait times, cancellations).
Tasks
-
Pick one primary success metric aligned to long‑term profit (e.g., 28‑day incremental gross profit per user) and list key guardrails (e.g., refund/chargeback rate, support tickets, seller cancellations, cannibalization of full‑price orders).
-
Choose and justify the unit of randomization (user vs device vs account vs session), considering repeated sessions per user, cross‑device logins, households that share devices, and seller‑side spillovers.
-
Specify exposure/eligibility rules, sticky assignment, holdout enforcement, and contamination controls.
-
Describe how you will detect/mitigate network effects (e.g., geo or cluster randomization, graph clustering, exposure mappings) and how you’d quantify spillovers.
-
Provide sample size and test duration with assumptions and MDE, a ramp and throttle plan, and stopping rules.
-
List instrumentation/logging needed to compute ITT and TOT (e.g., assignment id, exposure timestamp, eligibility flags, device id, session id).