Mobile App Ads Marketplace: Experiment Plan to Lift Revenue by 10% in 60 Days

Context

You run a mobile app ads marketplace. Target is to increase ad revenue by 10% within 60 days while preserving user experience:

• Guardrails: 7-day retention no worse than −0.5 percentage points; average session length no worse than −2%.
• Baselines: DAU = 1,000,000; avg sessions/user/day = 3; avg impressions/session = 2; CTR = 2%; CPM = $5; fill rate = 90%; per-user daily revenue variance ≈ 0.04 (USD²).
• Two levers to test: (A) +10% ad load, (B) +$0.10 floor price.

Assume mixed demand (CPM and CPC) but price realization is well-described by effective CPM and fill rate.

Tasks

1. Choose unit of randomization (user vs session vs geo) and justify interference risk (auction dynamics, supply constraints). How will you mitigate cross-treatment spillover? Would you prefer cluster randomization, ghost ads, or geo holdouts?
2. Define primary KPI(s) for decisioning (e.g., revenue/user/day) and guardrails (7D retention, session length, crash rate). Specify exact success and stop-loss thresholds.
3. Power analysis: compute sample size, traffic split, and duration for 90% power, α = 0.05 to detect a 10% uplift in revenue/user/day. State assumptions (variance, ICC if clustering) and show formulas.
4. Bias control: outline use of CUPED or pre-experiment covariates, seasonality controls, and sequential vs fixed-horizon testing. Provide your decision rule and how you’ll adjust for multiple variants.
5. Advertiser effects: how will you detect/avoid cannibalization between campaigns and shifts in clearing prices? What diagnostics would you run (e.g., win rate, eCPM distribution, supply-demand curves)?