Experiment Design: Replacing a Homepage Tab and Measuring User + Ecosystem Impact
Assume today is 2025-09-01. Roblox plans to replace an existing homepage tab with a new tab. Design a rigorous experiment to estimate the causal impact at both the user level and the creator/ecosystem level.
Specify the following:
-
Experimental unit and randomization
-
Choose the experimental unit (e.g., user-level vs. device-level) and define a randomization scheme that avoids cross-device contamination.
-
Metrics and guardrails
-
Primary success metrics (e.g., D1/D7 retention, homepage click-through, session length, conversion to play, creator impressions/engagement, revenue).
-
Guardrails (e.g., crash rate, latency, abuse, fairness of content distribution).
-
Novelty and navigation-friction mitigation
-
How you will mitigate and measure novelty and navigation-friction effects when a tab is removed (e.g., ramp plan, cooldowns, holdout re-exposure).
-
Ecosystem interference/network effects
-
How to handle ecosystem interference (e.g., switchback or geo experiments, creator-level holdouts) and measure supply-side externalities.
-
Metric definitions and covariates
-
Define metrics at a user-day grain. State whether you will use CUPED or pre-exposure covariates and how.
-
Power and duration
-
Sample size/power calculations for detecting a 1% relative lift in D7 retention with 80% power and α = 0.05. Include assumptions and minimum test duration accounting for weekly seasonality.
-
Decisions and rollback
-
Decision thresholds for ship/no-ship and rollback criteria.
-
Diagnostics for counterintuitive results
-
What diagnostics you would run if homepage click-through rises but D7 retention falls.
-
Follow-up design for cannibalization vs. long-term retention
-
A follow-up analysis plan if the treatment cannibalizes time from other tabs but increases long-term retention (e.g., 2×2 or crossover, or diff-in-diff with staggered rollout).