Design a Real-Time Personalized Ad Selection System
Overview
This question evaluates competency in end-to-end real-time ML system design for personalized ad selection, including model decisioning, feature engineering and stores, exploration–exploitation strategies, offline and online evaluation, and scalability and latency constraints within the Machine Learning domain.
Upstart
Aug 4, 2025, 10:55 AM
Data Scientist
Technical Screen
Machine Learning
0
0
End-to-End ML System Design: Real-Time Ad Selection
Context
You need to design a real-time, data-driven ad selection system that personalizes ads for millions of users. Assume:
- Request volume: millions of users/day; peak QPS in the thousands.
- Inventory: tens to hundreds of thousands of ads.
- Strict latency targets: p95 < 100 ms for ad decisioning (excluding network).
- Business objective: maximize expected value (e.g., clicks or revenue) while respecting advertiser budgets and policies.
Task
Design an end-to-end ML system to select ads at request time. Discuss and justify:
- Data collection and logging
- Labeling and outcome definition
- Feature engineering and a feature store
- Model choice and training objectives
- Exploration–exploitation strategy (e.g., contextual bandits)
- Offline evaluation and counterfactual/replay evaluation
- Online evaluation (A/B tests, guardrails)
- Latency and reliability constraints
- Scalability and architecture (training/serving pipelines)
Hints: CTR prediction with logistic regression or GBDT; counterfactual evaluation (IPS/DR); feature store for train-serve consistency; bandits for exploration.
Solution
Login required
Show