Facebook Ads has a legacy ranking model M0; an engineering team built a new model M1 you’re asked to ship and 'rule out' M0. Design the end‑to‑end replacement plan: (1) Offline checks: define target metrics (e.g., calibrated CTR, AUC/PR, calibration error, lift in expected revenue), leakage tests, covariate/label shift diagnostics, and backtests on the last 28 days. (2) Counterfactual/off‑policy evaluation: describe how you’d use IPS/DR estimators with propensity clipping to estimate expected CTR/revenue for M1 on M0 logs; specify assumptions and bias/variance tradeoffs. (3) Online plan: experiment design (user-level buckets, geo stratification), guardrails (revenue, spend pacing, latency, integrity), SRM/interference checks, ramp schedule (1%→10%→25%→50%→100%) with explicit stop/roll-back rules. (4) Acceptance criteria when key metrics disagree (e.g., CTR −1.5% but revenue per mille +3% and advertiser ROAS +2%): define the objective, weights, and minimum guardrail thresholds. (5) Productionization risks: model latency/SLA, feature freshness, replay/backfill, and monitoring you would implement post‑launch.

This question evaluates competence in machine learning systems engineering, including offline validation, counterfactual/off‑policy evaluation, online experimentation design, and production monitoring for ad-ranking models.

How do I approach Machine Learning interview questions?

Machine Learning questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master machine learning interviews.

What difficulty level is this interview question?

This is a hard difficulty Machine Learning question, commonly asked during Onsite rounds at Meta.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Meta during technical interviews.

Replace legacy ads model safely | Meta Interview Question

Facebook Ads Ranking Replacement: M0 to M1

You are asked to replace a legacy ads ranking model (M0) with a new model (M1) in a large-scale feed ads system. Design an end-to-end replacement plan that covers offline validation, counterfactual evaluation, online experimentation, acceptance criteria when metrics disagree, and productionization/monitoring.

Assume:

The ranking objective is platform revenue while maintaining user experience and advertiser value. Auctions are second-price-like with pacing controls. Logs contain per-impression data (scores, bids, position, clicks, conversions, revenue), and a small exploration slice is available for propensity logging.

Required:

Offline checks (last 28 days): define target metrics (e.g., calibrated CTR, AUC/PR, calibration error, lift in expected revenue), leakage tests, covariate/label shift diagnostics, and backtests.
Counterfactual/off-policy evaluation (OPE): explain how to use IPS/DR estimators with propensity clipping to estimate expected CTR/revenue for M1 on M0 logs; specify assumptions and bias/variance trade-offs.
Online plan: experiment design (e.g., user-level buckets, geo stratification), guardrails (revenue, spend pacing, latency, integrity), SRM/interference checks, ramp schedule (1%→10%→25%→50%→100%) with explicit stop/rollback rules.
Acceptance criteria when key metrics disagree (e.g., CTR −1.5% but revenue per mille +3% and advertiser ROAS +2%): define an objective, weights, and minimum guardrail thresholds.
Productionization risks: model latency/SLA, feature freshness, replay/backfill, and post-launch monitoring.

Facebook Ads Ranking Replacement: M0 to M1

Assume:

The ranking objective is platform revenue while maintaining user experience and advertiser value. Auctions are second-price-like with pacing controls. Logs contain per-impression data (scores, bids, position, clicks, conversions, revenue), and a small exploration slice is available for propensity logging.

Required:

Offline checks (last 28 days): define target metrics (e.g., calibrated CTR, AUC/PR, calibration error, lift in expected revenue), leakage tests, covariate/label shift diagnostics, and backtests.

Counterfactual/off-policy evaluation (OPE): explain how to use IPS/DR estimators with propensity clipping to estimate expected CTR/revenue for M1 on M0 logs; specify assumptions and bias/variance trade-offs.

Online plan: experiment design (e.g., user-level buckets, geo stratification), guardrails (revenue, spend pacing, latency, integrity), SRM/interference checks, ramp schedule (1%→10%→25%→50%→100%) with explicit stop/rollback rules.

Acceptance criteria when key metrics disagree (e.g., CTR −1.5% but revenue per mille +3% and advertiser ROAS +2%): define an objective, weights, and minimum guardrail thresholds.

Productionization risks: model latency/SLA, feature freshness, replay/backfill, and post-launch monitoring.

Replace legacy ads model safely

Quick Overview

Facebook Ads Ranking Replacement: M0 to M1

Solution

Comments (0)

Replace legacy ads model safely

Quick Overview

Facebook Ads Ranking Replacement: M0 to M1

Solution

Comments (0)