Design and analyze an ads ranking experiment

You are proposing a new ads ranking model that may increase revenue but could harm UX. Design an experiment and analysis plan. Assumptions: Baseline user-day RPM = $1.80, SD = $0.90 at the user-day aggregation; average 2 sessions/user/day and 5 ad requests/session; advertiser budgets pace hourly; the auction is second-price with per-campaign frequency caps. Tasks: A) Define a North Star metric and at least three guardrail metrics, with precise formulas and acceptable movement ranges. Explain trade-offs. B) Choose the randomization unit (request-, session-, or user-level) and justify by addressing interference (auction spillovers, budget pacing, frequency caps) and cross-device identity. C) Provide a sample-size calculation to detect a +2% lift in the primary metric with 80% power and 5% two-sided alpha. State all assumptions (variance unit, clustering, expected correlation from CUPED/stratification) and show your calculation; then describe how you would re-estimate during the test. D) Detail variance reduction you would use (e.g., CUPED with pre-period user RPM, stratification by geo/device/ad vertical) and why each is valid. E) Specify a ramp plan (percentages, duration per stage), early-stop criteria, and how you will handle winner’s curse and novelty effects. F) Describe diagnostics you will run for marketplace distortions (budget reallocation, cannibalization across campaigns, price feedback loops) and how you would correct for them before a full rollout.