Design an ads ranking ML system

Prompt

You are designing an ads ranking system for a large consumer app (feed/search entry point). For each request, the system receives a user context and a set of eligible ads/candidates and must return a ranked list of ads.

Requirements

• Primary business goal: maximize long-term value (e.g., revenue) while maintaining good user experience.
• Must handle multiple optimization targets such as CTR , CVR , and expected revenue .
• Latency budget: tens of milliseconds for the ranking stage (assume candidate generation is separate).
• Strong emphasis on ML aspects: feature design , training data , modeling choices , and multi-task learning .

Questions to answer

1. What features would you build (user/ad/context/cross features) and how would you generate and keep them fresh?
2. How would you define labels and build training data given bias from the existing ranker (position bias / selection bias)?
3. Propose a ranking model and explain how you’d combine multiple objectives (e.g., CTR + CVR + value). If using multi-task learning , describe the architecture and loss.
4. How would you evaluate the model offline and online? What key metrics and guardrails would you use?
5. How would you handle practical issues: cold start, delayed conversions, distribution shift, and feature leakage?