How would you evaluate a new ads ranking algorithm?

## Context You work at a social network company with an ads marketplace. The company has an existing ads ranking algorithm currently used to select and order ads in the feed. A new ranking algorithm is proposed and is believed to be better. ## Task Describe how you would evaluate and decide whether to roll out the new ads ranking algorithm, covering: 1. **Offline evaluation (if any):** - What historical data you would use. - What offline metrics you would compute (e.g., ranking quality / relevance). - Key pitfalls (selection bias from the current serving policy, feedback loops, delayed outcomes). 2. **Online experimentation plan:** - Experiment design (A/B, interleaving, switchback, staged rollout) and the **unit of randomization** (user, session, impression, geo/time bucket). - How you would handle **interference** (auction dynamics, advertiser budget pacing, marketplace effects) and **novelty**. - How long you would run, and what you would monitor during ramp. 3. **Metrics and tradeoffs:** - Propose **one primary metric** for decision-making and several **diagnostic metrics**. - Include **guardrails** for user experience and advertiser health. - Explicitly discuss tradeoffs among **revenue**, **advertiser value**, and **user engagement/satisfaction**. 4. **Interpreting results:** - How you would determine whether the change is a win (statistical significance, practical significance, heterogeneity). - How you would investigate metric movements (e.g., revenue up but engagement down). 5. **Recommendation and communication:** - How you would summarize results and make a go/no-go (or iterate) recommendation to leadership. - What risks, open questions, and follow-ups you would propose before full launch. ### Assumptions you may state You may assume a typical auction-based ads system (bids, predicted CTR/CVR, relevance/quality signals), and that the ranking model can change the ads shown to users, affecting both user behavior and advertiser spend.