Design an A/B test for comments UI

You will A/B test a new threaded comments UI aimed at increasing weekly unique commenters by 5% without reducing median session length by more than 1%. (a) Define precise hypotheses (primary and guardrail metrics), including measurement windows and denominators. Specify the unit of randomization (user vs. session), handling cross-device users and logged-out traffic, and how to prevent contamination. (b) Given a baseline weekly commenter rate of 8% per user, target relative lift of 5%, two-sided α=0.05, power=0.8, and expected weekly user sample of 2M, outline the sample size formula/inputs you’d use (no need to compute exact N) and whether you’d use CUPED or pre-experiment covariates to reduce variance. Include which covariates and why. (c) Propose a ramp plan with sequential monitoring that controls Type I error (e.g., group sequential or alpha spending). Describe stopping rules for efficacy/futility and how you’d handle novelty effects and learning effects over time. (d) List at least three guardrails (e.g., abuse reports, latency p95, session length) and define actionable thresholds. Explain what you’d do if the primary metric improves but a guardrail degrades. (e) If network effects (users replying across treatments) create interference, describe a cluster- or geo-based randomization design and its trade-offs versus individual-level randomization.