Ads Ranking Model: Experiment and Analysis Plan

Context

You are evaluating a new ads ranking model expected to increase revenue but potentially harm user experience (UX). The marketplace has hourly budget pacing, a second-price auction, and per-campaign frequency caps. Cross-device identity is imperfect.

Assumptions:

• Baseline user-day RPM (total ad revenue per user per day) = $1.80; standard deviation (SD) at the user-day level =$ 0.90.
• Average 2 sessions per user per day and 5 ad requests per session (≈10 requests/user-day).
• Advertiser budgets pace hourly; auction is second-price; per-campaign frequency caps apply.

Tasks

A) Define a North Star metric and at least three guardrail metrics, with precise formulas and acceptable movement ranges. Explain trade-offs.

B) Choose the randomization unit (request-, session-, or user-level) and justify by addressing interference (auction spillovers, budget pacing, frequency caps) and cross-device identity.

C) Provide a sample-size calculation to detect a +2% lift in the primary metric with 80% power and 5% two-sided alpha. State all assumptions (variance unit, clustering, expected correlation from CUPED/stratification) and show your calculation; then describe how you would re-estimate during the test.

D) Detail variance reduction you would use (e.g., CUPED with pre-period user RPM, stratification by geo/device/ad vertical) and why each is valid.

E) Specify a ramp plan (percentages, duration per stage), early-stop criteria, and how you will handle winner’s curse and novelty effects.

F) Describe diagnostics you will run for marketplace distortions (budget reallocation, cannibalization across campaigns, price feedback loops) and how you would correct for them before a full rollout.