Evaluate New Model's Performance Against Existing System
Quick Overview
This interview question evaluates core ML concepts, assumptions, math intuition, training/evaluation trade-offs, and practical failure modes in a realistic interview setting. A strong answer for Evaluate New Model's Performance Against Existing System states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.
Evaluate New Model's Performance Against Existing System
Company: Meta
Role: Data Scientist
Category: Machine Learning
Difficulty: medium
Interview Round: Technical Screen
##### Scenario
A new machine-learning model flags harmful posts; leadership wants evidence it outperforms the old system.
##### Question
How would you evaluate the performance of the new harmful-content detection model versus the existing model or no model? Describe both offline evaluation (confusion matrix metrics) and online A/B testing approaches, addressing precision-recall trade-offs.
##### Hints
Mention metrics (precision, recall, F1, ROC), calibration, business KPIs, guardrails, and experiment design.
Quick Answer: This interview question evaluates core ML concepts, assumptions, math intuition, training/evaluation trade-offs, and practical failure modes in a realistic interview setting. A strong answer for Evaluate New Model's Performance Against Existing System states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.
Evaluate New Model's Performance Against Existing System
Scenario
You are evaluating a new machine-learning model that detects harmful content on a large consumer platform. Leadership needs evidence that the new model outperforms the existing model, and to understand trade-offs between catching more harmful posts and avoiding over-removal of benign posts.
Task
Design a comprehensive evaluation plan to compare the new model against:
the existing (production) model, and
optionally, a minimal/no-model baseline (only if safe via safeguards).
Address both:
Offline evaluation using labeled data and confusion-matrix-based metrics.
Online A/B testing and experiment design.
Make the precision–recall trade-offs explicit, and connect model metrics to business outcomes.