Evaluate Factors Before Replacing Recommendation Model
Quick Overview
Evaluates whether an ads recommendation model should replace an existing production model. Strong answers combine offline validation, online experiments, revenue and advertiser outcomes, user guardrails, conflicting metric interpretation, rollout safety, rollback planning, and executive communication.
Evaluate Factors Before Replacing Recommendation Model
Company: Meta
Role: Data Scientist
Category: Machine Learning
Difficulty: hard
Interview Round: Onsite
##### Scenario
Facebook Ads team built a new recommendation model and plans to deprecate the old one.
##### Question
a) What factors must be evaluated before fully replacing the existing model?
b) CTR falls but revenue or other metrics increase—how would you decide which model to ship?
c) Outline the A/B-testing procedure you would follow.
d) How would you visualise and present the final results to the CFO?
##### Hints
Cover offline evaluation, guardrails, rollback plans, stakeholder dashboards, statistical significance.
Quick Answer: Evaluates whether an ads recommendation model should replace an existing production model. Strong answers combine offline validation, online experiments, revenue and advertiser outcomes, user guardrails, conflicting metric interpretation, rollout safety, rollback planning, and executive communication.
Evaluate Factors Before Replacing Recommendation Model
Meta
Jul 12, 2025, 6:59 PM
hardData ScientistOnsiteMachine Learning
63
0
Evaluate Factors Before Replacing a Recommendation Model
A large ads platform has built a new recommendation or ranking model and plans to deprecate the existing production model. You need to evaluate whether the replacement is safe and valuable.
Constraints & Assumptions
The model affects users, advertisers, platform revenue, and policy or quality outcomes.
Assume offline model metrics, online experiment infrastructure, ads delivery logs, user engagement, advertiser conversion data, and revenue metrics are available.
A model replacement should not be decided from one metric alone.
Include launch, rollback, and executive communication considerations.
Clarifying Questions to Ask Guidance
What does the ads model optimize: CTR, conversion value, revenue, advertiser ROI, user utility, or a combined objective?
Is the new model a ranking model, candidate generator, calibration model, or targeting model?
What are the hard guardrails around user experience, advertiser fairness, policy, and latency?
Are there known risks from changing auction dynamics or advertiser budgets?
Part 1 - Evaluate the Replacement
What factors must be evaluated before fully replacing the existing model?
Sample-ratio mismatch checks, instrumentation validation, guardrail alerts, and rollback criteria.
Post-launch monitoring for drift and delayed advertiser outcomes.
Part 4 - Present to the CFO
How would you visualize and present the final results to the CFO?
What This Part Should Cover Guidance
Clear executive summary of incremental revenue, confidence intervals, downside risks, and guardrail status.
Charts showing revenue, advertiser ROI, user guardrails, segment impact, rollout plan, and sensitivity analysis.
Recommendation with launch conditions and monitoring commitments.
What a Strong Answer Covers Guidance
A strong answer balances user, advertiser, and platform goals, uses offline and online evidence, handles conflicting metrics through a predefined decision framework, and communicates business impact clearly to executives.
Follow-up Questions Guidance
What would make you stop a model ramp immediately?
How would you evaluate delayed advertiser conversions?
How would you detect that small advertisers are harmed while aggregate revenue improves?