# Evaluate a New Ads-Ranking Algorithm An ads team has developed a new ranking algorithm that chooses which ad to show for each eligible opportunity. Engineers believe it is better than the current ranker. Design an evaluation that supports a launch decision. Address the randomization unit, success and guardrail metrics, advertiser budget and spend behavior, and a result in which conversion improves while click-through rate declines. State assumptions where product details are missing. ### Clarifying Questions to Ask - What objective is the ranker optimizing: conversions, conversion value, long-term value, revenue, or a constrained combination? - What is an eligible ranking opportunity, and can the same user or advertiser appear repeatedly? - Are budgets, bids, pacing rules, and downstream attribution available in the data? - Could treatment change which ads enter later auctions or how quickly advertisers exhaust budget? ### Part 1: Define success Build a metric framework that connects the ranking change to user, advertiser, platform, and marketplace outcomes. #### What This Part Should Cover - A primary metric tied to the stated objective - Funnel and value metrics with stable denominators - User, advertiser, latency, quality, and revenue guardrails - Segment-level diagnostics without opportunistic metric selection ### Part 2: Choose the experimental unit Compare randomization by user, request or auction, advertiser, and geographic or time cluster. Recommend a unit and explain the resulting analysis. #### What This Part Should Cover - Consistency of treatment exposure - Interference through budgets, pacing, competition, and repeated users - Cluster-aware uncertainty and intent-to-treat analysis ### Part 3: Interpret conversion up and CTR down Explain how you would determine whether this trade-off is acceptable rather than declaring a winner from either metric alone. #### What This Part Should Cover - Absolute effects with uncertainty and denominator checks - Changes in post-click quality, ad mix, value, and user experience - Alignment with the predeclared objective and guardrails ### Part 4: Reason about advertiser budgets and spend Describe what observed spend can reveal, what it cannot reveal without an explicit budget field, and how pacing effects should enter the launch decision. #### What This Part Should Cover - Separation of budget, bid, spend, and opportunity - Spend trajectories, exhaustion timing, and constrained advertisers - Avoidance of inferring an exact budget from spend alone ### What a Strong Answer Covers - A coherent causal experiment linked to the ranking objective - Recognition of auction and budget interference - A principled interpretation of competing funnel metrics - Advertiser-level distribution and pacing effects, not only platform totals - Clear launch, hold, and rollback criteria ### Follow-up Questions 1. What would you do if user-level randomization causes advertiser budget interference across arms? 2. How would you detect that the treatment benefits a few large advertisers while hurting the median advertiser? 3. What if conversion value rises but users hide or report more ads? 4. How would you monitor the launch after a successful experiment?