Evaluate Factors Before Replacing a Recommendation Model
A large ads platform has built a new recommendation or ranking model and plans to deprecate the existing production model. You need to evaluate whether the replacement is safe and valuable.
Constraints & Assumptions
-
The model affects users, advertisers, platform revenue, and policy or quality outcomes.
-
Assume offline model metrics, online experiment infrastructure, ads delivery logs, user engagement, advertiser conversion data, and revenue metrics are available.
-
A model replacement should not be decided from one metric alone.
-
Include launch, rollback, and executive communication considerations.
Clarifying Questions to Ask
-
What does the ads model optimize: CTR, conversion value, revenue, advertiser ROI, user utility, or a combined objective?
-
Is the new model a ranking model, candidate generator, calibration model, or targeting model?
-
What are the hard guardrails around user experience, advertiser fairness, policy, and latency?
-
Are there known risks from changing auction dynamics or advertiser budgets?
Part 1 - Evaluate the Replacement
What factors must be evaluated before fully replacing the existing model?
What This Part Should Cover
-
Offline metrics, calibration, ranking quality, bias, robustness, latency, coverage, freshness, interpretability, and failure modes.
-
Online metrics for revenue, CTR, conversion rate, advertiser ROI, user engagement, complaints, hide/report rate, policy violations, and long-term value.
-
Segment-level impact across advertisers, users, placements, markets, and budget sizes.
-
Operational readiness: monitoring, rollback, logging, and model retraining.
Part 2 - Interpret Conflicting Metrics
If CTR falls but revenue or other business metrics increase, how would you decide which model to ship?
What This Part Should Cover
-
Predefined objective hierarchy and guardrail thresholds.
-
Understanding whether CTR decline reflects worse user experience or better targeting toward higher-value conversions.
-
Advertiser ROI, user satisfaction, auction health, and long-term revenue effects.
-
Segment-level and long-term analysis before launch.
Part 3 - Design the A/B Test
Outline the A/B testing procedure from canary to full rollout.
What This Part Should Cover
-
Offline validation, shadow testing, canary rollout, ramp schedule, randomization, sample size, duration, and monitoring.
-
Sample-ratio mismatch checks, instrumentation validation, guardrail alerts, and rollback criteria.
-
Post-launch monitoring for drift and delayed advertiser outcomes.
Part 4 - Present to the CFO
How would you visualize and present the final results to the CFO?
What This Part Should Cover
-
Clear executive summary of incremental revenue, confidence intervals, downside risks, and guardrail status.
-
Charts showing revenue, advertiser ROI, user guardrails, segment impact, rollout plan, and sensitivity analysis.
-
Recommendation with launch conditions and monitoring commitments.
What a Strong Answer Covers
A strong answer balances user, advertiser, and platform goals, uses offline and online evidence, handles conflicting metrics through a predefined decision framework, and communicates business impact clearly to executives.
Follow-up Questions
-
What would make you stop a model ramp immediately?
-
How would you evaluate delayed advertiser conversions?
-
How would you detect that small advertisers are harmed while aggregate revenue improves?