Experiment Design: Increasing Order‑Related Push Notifications

Context

You are asked to design, measure, and make decisions about increasing order‑related push notifications in a consumer mobile app to drive more sessions and conversions. Assume a large active user base across multiple time zones and multiple messaging channels (push, email, SMS). Order‑related pushes include reminders, cart/checkout nudges, reorder prompts, and relevant deal notifications tied to ordering behavior.

Tasks

1. Experiment design:
   • Propose an RCT to measure the incremental impact of higher notification frequency and/or smarter timing.
   • Specify the randomization unit, exposure caps, time-of-day stratification, and controls for cross‑channel effects (email/SMS).
   • Consider multi‑armed variants and when (not) to use bandits.
2. Metrics:
   • Define primary success metrics (incremental orders, revenue, profit) and guardrails (opt‑out rate, uninstall rate, complaint rate, session quality).
   • Include per‑user and per‑notification level metrics and describe aggregation.
3. Long‑term effects:
   • Design a holdout or staggered rollout with a 4–8 week follow‑up.
   • Address novelty effects, fatigue, and decay, and describe how to estimate persistent lift (e.g., switchbacks, long‑lived holdouts, synthetic controls).
4. Interference and repeated exposure:
   • Propose analysis to account for saturation and diminishing returns (e.g., dose‑response curves, exposure‑weighted treatment, instrumental variables via randomized send/no‑send at trigger level).
5. Power/MDE, stopping rules, and decisions:
   • Define power/MDE assumptions, stopping rules, and a decision framework balancing short‑term lift vs long‑term retention risk.
   • Include a concrete rollback criterion.