Design a Real-Time Personalized Ad Selection System

Quick Overview

This interview question evaluates core ML concepts, assumptions, math intuition, training/evaluation trade-offs, and practical failure modes in a realistic interview setting. A strong answer for Design a Real-Time Personalized Ad Selection System states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

Design a Real-Time Personalized Ad Selection System

Company: Upstart

Role: Data Scientist

Category: Machine Learning

Difficulty: hard

Interview Round: Technical Screen

##### Scenario You are asked to build a data-driven advertising platform that serves personalized ads to millions of users in real time. ##### Question Design an end-to-end machine-learning system for ad selection: discuss data collection, labeling, feature engineering, model choice, exploration-exploitation strategy, online/offline evaluation, latency constraints, and scalability. ##### Hints CTR prediction with logistic regression/GBDT, counterfactual evaluation, feature store, bandits for exploration.

Quick Answer: This interview question evaluates core ML concepts, assumptions, math intuition, training/evaluation trade-offs, and practical failure modes in a realistic interview setting. A strong answer for Design a Real-Time Personalized Ad Selection System states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

|Home/Machine Learning/Upstart
Upstart logo
Upstart
Aug 4, 2025, 10:55 AM
hardData ScientistTechnical ScreenMachine Learning
7
0

Design a Real-Time Personalized Ad Selection System

End-to-End ML System Design: Real-Time Ad Selection

Context

You need to design a real-time, data-driven ad selection system that personalizes ads for millions of users. Assume:

  • Request volume: millions of users/day; peak QPS in the thousands.
  • Inventory: tens to hundreds of thousands of ads.
  • Strict latency targets: p95 < 100 ms for ad decisioning (excluding network).
  • Business objective: maximize expected value (e.g., clicks or revenue) while respecting advertiser budgets and policies.

Task

Design an end-to-end ML system to select ads at request time. Discuss and justify:

  1. Data collection and logging
  2. Labeling and outcome definition
  3. Feature engineering and a feature store
  4. Model choice and training objectives
  5. Exploration–exploitation strategy (e.g., contextual bandits)
  6. Offline evaluation and counterfactual/replay evaluation
  7. Online evaluation (A/B tests, guardrails)
  8. Latency and reliability constraints
  9. Scalability and architecture (training/serving pipelines)

Hints: CTR prediction with logistic regression or GBDT; counterfactual evaluation (IPS/DR); feature store for train-serve consistency; bandits for exploration.

Clarifying Questions to Ask Guidance

  • Clarify the task, data shape, labels, constraints, and evaluation metric.
  • State assumptions behind the math or modeling technique you choose.
  • Connect theory to practical training, debugging, and deployment implications.

What a Strong Answer Covers Guidance

  • Correct definitions and formulas where the prompt requires them.
  • A practical explanation of how the method behaves on real data.
  • Trade-offs, failure modes, diagnostics, and mitigation strategies.
  • Evaluation choices that match the product or modeling objective.

Follow-up Questions Guidance

  • How would noisy labels, class imbalance, or distribution shift affect the answer?
  • What would you monitor after deployment?
  • Which baseline would you compare against first?
Loading comments...

Browse More Questions

More Machine Learning•More Upstart•More Data Scientist•Upstart Data Scientist•Upstart Machine Learning•Data Scientist Machine Learning