Design an Advertising Ranking Pipeline beyond Calibration
Company: Instacart
Role: Machine Learning Engineer
Category: Machine Learning
Difficulty: medium
Interview Round: Technical Screen
Design the end-to-end machine-learning pipeline for selecting and ranking ads in a grocery product. Explain the components, the models and losses you would use, and how predicted probabilities would be calibrated and evaluated within the larger system.
### Clarifying Questions
Clarify the ad placement, eligible advertisers and products, billing event, budget and pacing requirements, and the product's relevance constraints. These business rules are not specified. Do not assume a particular auction or pricing policy; describe where that policy would enter your design.
### What a Strong Answer Covers
Candidate eligibility and retrieval, labels and feature freshness, response prediction, calibration, ranking-policy integration, budget controls, serving reliability, and offline and online evaluation. Give calibration appropriate attention without treating it as the entire architecture.
### Follow-up Questions
- How would you train and evaluate conversion models when labels arrive late?
- What could go wrong if predicted click probability is well calibrated overall but poor for new advertisers?
- How would the serving system handle an exhausted budget or unavailable product?
Overview: Design an advertising ML pipeline from eligible candidates to response prediction, calibration, and policy-aware ranking. Cover delayed labels, budget controls, feedback bias, and evaluation across advertiser segments.
Read the full Instacart Machine Learning Engineer interview experience this question came from