Experiment Design: Replacing a Homepage Tab and Measuring User + Ecosystem Impact

Assume today is 2025-09-01. Roblox plans to replace an existing homepage tab with a new tab. Design a rigorous experiment to estimate the causal impact at both the user level and the creator/ecosystem level.

Specify the following:

1. Experimental unit and randomization
   • Choose the experimental unit (e.g., user-level vs. device-level) and define a randomization scheme that avoids cross-device contamination.
2. Metrics and guardrails
   • Primary success metrics (e.g., D1/D7 retention, homepage click-through, session length, conversion to play, creator impressions/engagement, revenue).
   • Guardrails (e.g., crash rate, latency, abuse, fairness of content distribution).
3. Novelty and navigation-friction mitigation
   • How you will mitigate and measure novelty and navigation-friction effects when a tab is removed (e.g., ramp plan, cooldowns, holdout re-exposure).
4. Ecosystem interference/network effects
   • How to handle ecosystem interference (e.g., switchback or geo experiments, creator-level holdouts) and measure supply-side externalities.
5. Metric definitions and covariates
   • Define metrics at a user-day grain. State whether you will use CUPED or pre-exposure covariates and how.
6. Power and duration
   • Sample size/power calculations for detecting a 1% relative lift in D7 retention with 80% power and α = 0.05. Include assumptions and minimum test duration accounting for weekly seasonality.
7. Decisions and rollback
   • Decision thresholds for ship/no-ship and rollback criteria.
8. Diagnostics for counterintuitive results
   • What diagnostics you would run if homepage click-through rises but D7 retention falls.
9. Follow-up design for cannibalization vs. long-term retention
   • A follow-up analysis plan if the treatment cannibalizes time from other tabs but increases long-term retention (e.g., 2×2 or crossover, or diff-in-diff with staggered rollout).