Design experiment for homepage tab replacement

Roblox plans to replace an existing homepage tab with a new tab. Design a rigorous experiment to evaluate the causal impact of this replacement at the user and ecosystem levels. Assume today is 2025-09-01. Specify: (1) the experimental unit (e.g., user-level vs. device-level) and randomization scheme that avoids cross-device contamination; (2) primary success metrics (e.g., D1/D7 retention, homepage click-through, session length, conversion to play, creator impressions/engagement, revenue) and guardrails (crash rate, latency, abuse, fairness of content distribution); (3) how you will mitigate and measure novelty and navigation-friction effects when a tab is removed (e.g., ramp plan, cooldowns, or holdout re-exposure); (4) how to handle ecosystem interference/network effects (e.g., switchback or geo experiments, creator-level holdouts); (5) the metric definitions at a user-day grain and whether to use CUPED or pre-exposure covariates; (6) sample size/power calculations for a 1% relative lift in D7 retention with 80% power and α=0.05, including assumptions and minimum test duration under weekly seasonality; (7) decision thresholds and rollback criteria; (8) what diagnostics you would run if click-through rises but D7 retention falls; (9) a follow-up analysis plan if the treatment cannibalizes time from other tabs but increases long-term retention (e.g., 2x2 or crossover design, or diff-in-diff with staggered rollout).