Replace legacy ads model safely
Overview
This question evaluates competence in machine learning systems engineering, including offline validation, counterfactual/off‑policy evaluation, online experimentation design, and production monitoring for ad-ranking models.
Meta
Oct 13, 2025, 9:49 PM
Data Scientist
Onsite
Machine Learning
0
0
Facebook Ads Ranking Replacement: M0 to M1
You are asked to replace a legacy ads ranking model (M0) with a new model (M1) in a large-scale feed ads system. Design an end-to-end replacement plan that covers offline validation, counterfactual evaluation, online experimentation, acceptance criteria when metrics disagree, and productionization/monitoring.
Assume:
- The ranking objective is platform revenue while maintaining user experience and advertiser value. Auctions are second-price-like with pacing controls. Logs contain per-impression data (scores, bids, position, clicks, conversions, revenue), and a small exploration slice is available for propensity logging.
Required:
- Offline checks (last 28 days): define target metrics (e.g., calibrated CTR, AUC/PR, calibration error, lift in expected revenue), leakage tests, covariate/label shift diagnostics, and backtests.
- Counterfactual/off-policy evaluation (OPE): explain how to use IPS/DR estimators with propensity clipping to estimate expected CTR/revenue for M1 on M0 logs; specify assumptions and bias/variance trade-offs.
- Online plan: experiment design (e.g., user-level buckets, geo stratification), guardrails (revenue, spend pacing, latency, integrity), SRM/interference checks, ramp schedule (1%→10%→25%→50%→100%) with explicit stop/rollback rules.
- Acceptance criteria when key metrics disagree (e.g., CTR −1.5% but revenue per mille +3% and advertiser ROAS +2%): define an objective, weights, and minimum guardrail thresholds.
- Productionization risks: model latency/SLA, feature freshness, replay/backfill, and post-launch monitoring.
Solution
Login required
Show