Design ad revenue A/B with guardrails
Company: Roblox
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: hard
Interview Round: Technical Screen
You own a mobile app ads marketplace. Goal: increase ad revenue by 10% in 60 days without hurting 7-day retention by more than −0.5pp or average session length by more than −2%. Baselines: DAU=1,000,000; avg sessions/user/day=3; avg impressions/session=2; CTR=2%; CPM=$5; fill rate=90%; per-user daily revenue variance≈0.04 (USD^2). Propose an experiment plan to test two levers: (A) +10% ad load; (B) +$0.10 floor price. Answer:
1) Choose unit of randomization (user vs session vs geo) and justify interference risk (auction dynamics, supply constraints). How will you mitigate cross-treatment spillover? Would you prefer cluster randomization, ghost ads, or geo holdouts?
2) Define primary KPI(s) for decisioning (e.g., revenue/user/day) and guardrails (7D retention, session length, crash rate). Specify exact success and stop-loss thresholds.
3) Power analysis: compute sample size, traffic split, and duration for 90% power, α=0.05 to detect a 10% uplift in revenue/user/day. State your assumptions (variance, ICC if clustering) and show formulas.
4) Bias control: outline use of CUPED or pre-experiment covariates, seasonality controls, and sequential vs fixed-horizon testing. Provide your decision rule and how you’ll adjust for multiple variants.
5) Advertiser effects: how will you detect/avoid cannibalization between campaigns and shifts in clearing prices? What diagnostics would you run (e.g., win rate, eCPM distribution, supply-demand curves)?
Quick Answer: This question evaluates a data scientist's competency in experimental design, causal inference, and statistical power analysis for ad monetization, including choice of randomization unit, interference mitigation in auctioned inventory, guardrail definition, bias controls, and diagnostics for advertiser effects.