Design an ads ranking system with calibration
Company: Meta
Role: Machine Learning Engineer
Category: ML System Design
Difficulty: medium
Interview Round: Onsite
## ML System Design: Ads Ranking (e-commerce)
Design an online **ads ranking** (ad “re-ranking”) system for an e-commerce app.
The system receives a request when a user opens a page/feed and must select and order a set of candidate ads to show.
### Requirements
- **Objective:** maximize long-term business value (e.g., revenue), while maintaining user experience
- **Latency:** low-latency online ranking (tens of milliseconds to a few hundred ms, depending on assumptions)
- **Scale:** many users/requests, many advertisers/items
- **Modeling topics to cover:**
- Feature engineering (user, item/ad, context, cross features)
- Model architecture choices for ranking
- **Calibration** of predicted probabilities (e.g., CTR/CVR) and why it matters
- **Evaluation:** offline metrics + online A/B testing and guardrails
Explain your end-to-end design: candidate generation, ranking/re-ranking, training pipeline, serving, and monitoring.
Quick Answer: This question evaluates a candidate's ability to design scalable, low-latency online machine learning systems for ads ranking, covering competencies in feature engineering, ranking model architecture, probability calibration, training and serving pipelines, and monitoring.