Design an A/B test for ML model launch

You are replacing the current ranker with a new model in a feed. Baseline CTR is 2.0%. You expect a +5% relative lift on CTR and want 90% power at α=0.05. Daily eligible traffic is 1,000,000 users; assignment is user-level 50/50, stable over time. A) Compute the required sample size and minimum test duration using a two-proportion power analysis; state all assumptions (variance, independence, no interference). Show your formulas and reasoning; adjust for a 5% expected bot/invalid traffic rate and a 1% potential sample ratio mismatch. B) Plan guardrail metrics (bounce rate, crashes, latency p95, revenue per user) and decision thresholds. Describe sequential monitoring with α-spending (e.g., Pocock or O’Brien–Fleming) to allow early stop without inflating Type I error. C) Address novelty and day-of-week effects: propose CUPED (covariate adjustment) using pre-experiment user CTR; specify the exact covariate and how you would validate variance reduction without bias. D) Mitigate interference and contamination: ensure user-level bucketing, prevent cross-arm content spillover, and plan a geo or holdout if network effects are suspected. E) After the test, outline checks for power actually achieved, heterogeneity of treatment effects across cohorts, and how you’d decide to ramp to 100%.