Design an ads ranking system with calibration

ML System Design: Ads Ranking (e-commerce)

Design an online ads ranking (ad “re-ranking”) system for an e-commerce app.

The system receives a request when a user opens a page/feed and must select and order a set of candidate ads to show.

Requirements

• Objective: maximize long-term business value (e.g., revenue), while maintaining user experience
• Latency: low-latency online ranking (tens of milliseconds to a few hundred ms, depending on assumptions)
• Scale: many users/requests, many advertisers/items
• Modeling topics to cover:
  • Feature engineering (user, item/ad, context, cross features)
  • Model architecture choices for ranking
  • Calibration of predicted probabilities (e.g., CTR/CVR) and why it matters
• Evaluation: offline metrics + online A/B testing and guardrails

Explain your end-to-end design: candidate generation, ranking/re-ranking, training pipeline, serving, and monitoring.