Ads Ranking And Auction-Aware ML
Asked of: Data Scientist
Last updated
-
What's being tested
The interviewer is checking whether you understand how auction mechanics change model objectives, data bias, and evaluation for ad-ranking. They expect you to connect pCTR/pCVR modeling, revenue/eCPM objectives, counterfactual evaluation, and practical mitigation strategies (calibration, reweighting, A/B/auction simulation). -
Core knowledge
- Generalized second-price (GSP) and Vickrey (second‑price) auction payment rules and their effect on advertiser incentives.
- eCPM = predicted value (pCTRpV or pCTRbid) used for rank; optimizing CTR alone can reduce revenue.
- Winner’s curse / selection bias: logged impressions are conditional on past auction outcomes.
- Counterfactual offline evaluation: Inverse Propensity Scoring (IPS) and Doubly Robust estimators.
- Loss reweighting / cost-sensitive objective: weight training loss by bid or expected payment to optimize revenue.
- Calibration techniques: Platt scaling, isotonic regression to align p(predicted) with actual click rates.
- Online considerations: budget pacing, bid shading, exploration (epsilon-greedy/Thompson sampling) and A/B vs auction simulation.
-
Worked example (framing: "Design an auction-aware CTR model that maximizes revenue under a second-price auction")
First define the production objective: maximize platform revenue (sum of payments), not raw CTR. Map model output to the auction score (pCTR * bid or pCTR * pV) and state how that changes loss (cost-sensitive with weight=bid). Call out selection bias: training data only contains shown ads → use IPS/doubly robust or randomized logging to correct. Describe offline evaluation via counterfactual estimators and an auction simulator, then propose an online experiment (randomized bucket or small exploration) to validate. Finally discuss calibration and regularization to avoid overfitting to high-bid noise. -
A common pitfall
A tempting but wrong approach is to train a high-accuracy CTR model and assume revenue will follow. High CTRs on low-bid advertisers often lower eCPM; without bid-aware loss weighting or optimizing predicted value, rank changes can reduce revenue. Likewise, using naive holdout metrics on logged data ignores selection bias and will overstate online gains.