Evaluate Classifier with Precision, Recall, and Fairness Metrics

Quick Overview

This interview question evaluates core ML concepts, assumptions, math intuition, training/evaluation trade-offs, and practical failure modes in a realistic interview setting. A strong answer for Evaluate Classifier with Precision, Recall, and Fairness Metrics states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

Evaluate Classifier with Precision, Recall, and Fairness Metrics

Company: Meta

Role: Data Scientist

Category: Machine Learning

Difficulty: medium

Interview Round: Technical Screen

##### Scenario Data science team must offline-evaluate a classifier that labels videos as harmful. ##### Question What offline evaluation framework would you use? Detail suitable metrics (e.g., precision, recall, PR-AUC), handling class imbalance, ground-truth collection, threshold selection based on business costs, and potential fairness checks. ##### Hints Discuss label skew, cost of false positives vs negatives, and calibration.

Quick Answer: This interview question evaluates core ML concepts, assumptions, math intuition, training/evaluation trade-offs, and practical failure modes in a realistic interview setting. A strong answer for Evaluate Classifier with Precision, Recall, and Fairness Metrics states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

|Home/Machine Learning/Meta
Meta logo
Meta
Aug 4, 2025, 10:55 AM
mediumData ScientistTechnical ScreenMachine Learning
4
0

Evaluate Classifier with Precision, Recall, and Fairness Metrics

Offline Evaluation Framework for a Harmful-Content Video Classifier

Context

You are evaluating a binary classifier that assigns each video a score (interpretable as the probability it violates a harmful-content policy). Harmful videos are rare (label skew). The business incurs different costs for false positives (over-blocking/over-review) and false negatives (missed harm). Moderation capacity may also be limited.

Task

Design an offline evaluation plan that covers:

  1. Metrics
  • Which ranking and operating-point metrics to report (e.g., precision, recall, PR-AUC), including calibration metrics.
  1. Class Imbalance
  • How to evaluate meaningfully under severe label skew and when the evaluation sample is not a simple random draw (e.g., stratified by model score).
  1. Ground-Truth Collection
  • How to collect high-quality labels for harmful content, including rater setup, agreement, sampling, and quality controls.
  1. Threshold Selection and Business Costs
  • How to choose a decision threshold given asymmetric costs of false positives vs. false negatives and potential moderation capacity constraints.
  1. Fairness Checks
  • What subgroup analyses and fairness metrics to run to guard against disparate impact.

Include assumptions where necessary, and provide formulas and examples for thresholding and weighting.

Constraints & Assumptions

  • Preserve the scope, facts, inputs, and requested outputs from the prompt above.
  • If the prompt leaves a detail unspecified, state a reasonable assumption before relying on it.
  • Keep the answer interview-ready: concise enough to present, but concrete enough to implement or evaluate.

Clarifying Questions to Ask Guidance

  • Clarify the task, data shape, labels, constraints, and evaluation metric.
  • State assumptions behind the math or modeling technique you choose.
  • Connect theory to practical training, debugging, and deployment implications.

What a Strong Answer Covers Guidance

  • Correct definitions and formulas where the prompt requires them.
  • A practical explanation of how the method behaves on real data.
  • Trade-offs, failure modes, diagnostics, and mitigation strategies.
  • Evaluation choices that match the product or modeling objective.

Follow-up Questions Guidance

  • How would noisy labels, class imbalance, or distribution shift affect the answer?
  • What would you monitor after deployment?
  • Which baseline would you compare against first?
Loading comments...

Browse More Questions

More Machine Learning•More Meta•More Data Scientist•Meta Data Scientist•Meta Machine Learning•Data Scientist Machine Learning