How would you evaluate a new ads ranking algorithm?

Context

You work at a social network company with an ads marketplace. The company has an existing ads ranking algorithm currently used to select and order ads in the feed. A new ranking algorithm is proposed and is believed to be better.

Task

Describe how you would evaluate and decide whether to roll out the new ads ranking algorithm, covering:

1. Offline evaluation (if any):
   • What historical data you would use.
   • What offline metrics you would compute (e.g., ranking quality / relevance).
   • Key pitfalls (selection bias from the current serving policy, feedback loops, delayed outcomes).
2. Online experimentation plan:
   • Experiment design (A/B, interleaving, switchback, staged rollout) and the unit of randomization (user, session, impression, geo/time bucket).
   • How you would handle interference (auction dynamics, advertiser budget pacing, marketplace effects) and novelty .
   • How long you would run, and what you would monitor during ramp.
3. Metrics and tradeoffs:
   • Propose one primary metric for decision-making and several diagnostic metrics .
   • Include guardrails for user experience and advertiser health.
   • Explicitly discuss tradeoffs among revenue , advertiser value , and user engagement/satisfaction .
4. Interpreting results:
   • How you would determine whether the change is a win (statistical significance, practical significance, heterogeneity).
   • How you would investigate metric movements (e.g., revenue up but engagement down).
5. Recommendation and communication:
   • How you would summarize results and make a go/no-go (or iterate) recommendation to leadership.
   • What risks, open questions, and follow-ups you would propose before full launch.

Assumptions you may state

You may assume a typical auction-based ads system (bids, predicted CTR/CVR, relevance/quality signals), and that the ranking model can change the ads shown to users, affecting both user behavior and advertiser spend.