Design an ML system to **increase the click-through rate (CTR)** of ads shown in the personalized feed of an online social media platform. You run the ranking team for feed ads. Advertisers buy placements that are interleaved into an organic content feed, and the business wants more relevant ads so that users click more often — without degrading the organic experience or the platform's long-term engagement. Walk through how you would frame, build, train, serve, and operate this system end to end. The interviewer will push hardest on **training-data construction**, **bias handling**, and **how you build user and ad embeddings** (graph-based and two-tower approaches), so be ready to go deep there. ### Constraints & Assumptions - Feed ads are interleaved into an organic feed; a typical session surfaces a few ads among many organic items. - Treat this as a large-scale system: on the order of hundreds of millions of users and millions of active ad creatives, with billions of impressions logged per day. - The serving path has a tight latency budget (low tens of milliseconds at p99 for the ranking stage) and must scale to high feed-request QPS. - Ads are sold via an auction; the ranking model's predicted click probability feeds into expected-value scoring, so **predicted probabilities must be calibrated**, not just well-ordered. - You may assume access to standard logged signals: impressions, clicks, downstream conversions (with an attribution window), creative content (text/image), advertiser metadata, and user interaction history. ### Clarifying Questions to Ask - What exactly counts as a "click," and is the true business objective raw CTR or something downstream (conversions, revenue/eCPM, advertiser ROAS)? CTR alone is easy to game. - What is the ad-to-organic ratio and the placement policy — are ad slots fixed, or does the system also decide whether to show an ad at all? - What guardrails define "do no harm" to the organic experience (hide/report rate, session length, retention)? - What is the auction and pricing mechanism, and does the model's output feed directly into bid × pCTR scoring? - What logging infrastructure already exists — do we log the full candidate set and serving propensities, or only the shown ad and its outcome? - What latency and freshness budgets do we have for retrieval, ranking, and embedding updates? ### Part 1 — Goal and Metrics Define the objective and the metric suite. What online and offline metrics would you track? How do you guard against regressions in user experience, revenue, and long-term engagement while pushing CTR up? ```hint Where to start Don't optimize a single number. Pair the **primary** metric with **guardrail** metrics and a separate set of **offline** metrics you can compute before launch. ``` ```hint Pitfall to name CTR in isolation rewards clickbait. Tie it to a post-click quality or value signal (dwell, conversion, hide/report) so the model is penalized for low-quality clicks. ``` #### What This Part Should Cover - A clear primary metric and explicit guardrails (UX, revenue, retention) — not CTR alone. - The distinction between **online** business metrics and **offline** model metrics used pre-launch. - Awareness that offline ranking metrics (e.g., AUC) can be misleading under position bias; calibration matters for an auction. ### Part 2 — Overall Architecture Sketch the end-to-end pipeline from candidate ad retrieval through ranking to serving. What stages exist, what does each do, and roughly how many candidates flow between them? ```hint Decomposition Think of ads ranking as a **funnel**: a cheap, high-recall stage narrows millions of ads to a small set, then progressively more expensive models score fewer candidates. ``` #### What This Part Should Cover - A multi-stage funnel: eligibility/filtering → retrieval (candidate generation) → pre-ranking → full ranking → auction/re-ranking → serving + logging. - Why each stage exists (latency vs. accuracy trade-off) and the rough fan-out at each stage. - Where business constraints (targeting, budget pacing, frequency caps, policy) and logging fit in. ### Part 3 — Training Data What logs do you need to train the CTR model? What is the learning target/label, and how do you construct positive and negative examples? ```hint Where to start The label comes from **impressions**, not from a user's click history alone. For every example you need both the shown item and whether it was clicked. ``` ```hint Positives vs negatives Positives = impressed **and** clicked. Negatives = impressed but **not** clicked. Training only on clicked events (no impression negatives) bakes in severe bias. ``` #### Clarifying Questions for this Part - Do we have the full logged candidate set and serving propensities, or only the single impression that was shown? This determines whether counterfactual/debiased training is even possible. - What attribution window defines a "click," and how do we treat delayed downstream conversions? #### What This Part Should Cover - The per-impression log schema: user/ad/context, rank/position, click outcome, and ideally candidate set + propensities. - Correct label definition and **impression-based** positive/negative construction (the candidate's likely failure mode is "use user history logs" without impression negatives). - Why the naive "history-only" approach is insufficient and how it biases the model. ### Part 4 — Embeddings How would you build user and ad embeddings for this system? Discuss both a **two-tower** retrieval approach and a **graph-based** approach (e.g., neighbor sampling / GNN-style methods such as PinSage / PinnerSage), what features feed each tower, and how the embeddings are trained and served. ```hint Two-tower Separate a **user tower** and an **ad tower** trained so that clicked (user, ad) pairs have high similarity; serve retrieval via an ANN index over ad vectors and dot-product / cosine scoring. ``` ```hint Graph-based Model users, ads, advertisers, and content as nodes in a bipartite/heterogeneous graph. Sample a node's interaction neighborhood (weighted by recency/importance) and aggregate neighbor features over a few layers — this captures transitive similarity and helps cold start. ``` ```hint Freshness User and ad embeddings drift at different rates (user behavior changes fast; ad creatives change on edit). Plan for streaming/near-real-time updates on the user side and recompute on creative changes. ``` #### What This Part Should Cover - A two-tower retrieval design: tower inputs, training objective (contrastive / sampled-softmax on clicked vs. non-clicked/sampled negatives), and ANN serving. - A graph-based design: graph schema (nodes/edges), neighbor sampling + aggregation, link-prediction objective, and when it beats two-tower. - The feature content of each embedding (ad: text/image/category/advertiser; user: interaction sequence + long-term profile; context) and freshness/update strategy. ### Part 5 — Practical Concerns Address the operational realities: cold start, bias and label leakage, delayed feedback, calibration, exploration vs. exploitation, monitoring, and iteration cadence. Pick the ones most load-bearing for an ads system and explain how you handle each. ```hint Bias Name the specific biases — **position bias** (higher slots get more clicks) and **selection bias** (you only observe ads the current policy chose) — and pair each with a concrete mitigation (position features, randomized exploration buckets, logged propensities for IPS/DR). ``` ```hint Calibration Because the auction multiplies bid × predicted click probability, the probabilities must be **calibrated** (e.g., isotonic / Platt per slice), and you must monitor calibration drift after each launch. ``` #### What This Part Should Cover - Cold start for new ads (content/advertiser priors) and new users (cohort priors, context, explore). - Concrete handling of position bias, selection bias, leakage, and delayed conversions. - Calibration for the auction and an exploration mechanism (ε-greedy / Thompson sampling) with propensity logging for unbiased evaluation. - A monitoring plan (data quality, prediction/calibration drift, business metrics) and an iteration cadence. ### What a Strong Answer Covers Across all parts, a strong answer threads a single narrative rather than answering five disconnected questions: the metric definition (Part 1) constrains the model objective; the training-data and bias decisions (Part 3) determine whether offline gains survive online; the embedding strategy (Part 4) feeds both retrieval and ranking; and the practical concerns (Part 5) close the loop back to the guardrails from Part 1. The candidate should make the **CTR-alone-is-gameable** trade-off explicit, get the **impression-based positive/negative construction** right, and connect **calibration + exploration + propensity logging** as the machinery that makes the auction and offline evaluation trustworthy. ### Follow-up Questions - Your model shows large offline AUC gains but the A/B test is flat or negative. Walk through the most likely causes (position/selection bias, calibration, train/serve skew) and how you'd diagnose them. - How would you design the system to jointly optimize CTR and post-click value (e.g., conversions) without letting one cannibalize the other? Discuss multi-task modeling and how the auction score changes. - A major advertiser launches a brand-new campaign with zero history. Trace exactly how it gets retrieved, ranked, and given a fair chance to accumulate signal without over-spending budget. - How do delayed conversions (which arrive hours or days after the click) break a naive training pipeline, and what would you change to handle them?