Evaluate Classifier with Precision, Recall, and Fairness Metrics

Quick Overview

This interview question evaluates core ML concepts, assumptions, math intuition, training/evaluation trade-offs, and practical failure modes in a realistic interview setting. A strong answer for Evaluate Classifier with Precision, Recall, and Fairness Metrics states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

Evaluate Classifier with Precision, Recall, and Fairness Metrics

Company: Meta

Role: Data Scientist

Category: Machine Learning

Difficulty: medium

Interview Round: Technical Screen

##### Scenario Data science team must offline-evaluate a classifier that labels videos as harmful. ##### Question What offline evaluation framework would you use? Detail suitable metrics (e.g., precision, recall, PR-AUC), handling class imbalance, ground-truth collection, threshold selection based on business costs, and potential fairness checks. ##### Hints Discuss label skew, cost of false positives vs negatives, and calibration.

Overview: This interview question evaluates core ML concepts, assumptions, math intuition, training/evaluation trade-offs, and practical failure modes in a realistic interview setting. A strong answer for Evaluate Classifier with Precision, Recall, and Fairness Metrics states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

|Home/Machine Learning/Meta
Meta logo
Meta
Aug 4, 2025
mediumData ScientistTechnical ScreenMachine Learning
6
0

Evaluate Classifier with Precision, Recall, and Fairness Metrics

Offline Evaluation Framework for a Harmful-Content Video Classifier

Context

You are evaluating a binary classifier that assigns each video a score (interpretable as the probability it violates a harmful-content policy). Harmful videos are rare (label skew). The business incurs different costs for false positives (over-blocking/over-review) and false negatives (missed harm). Moderation capacity may also be limited.

Task

Design an offline evaluation plan that covers:

  1. Metrics
  • Which ranking and operating-point metrics to report (e.g., precision, recall, PR-AUC), including calibration metrics.
  1. Class Imbalance
  • How to evaluate meaningfully under severe label skew and when the evaluation sample is not a simple random draw (e.g., stratified by model score).
  1. Ground-Truth Collection
  • How to collect high-quality labels for harmful content, including rater setup, agreement, sampling, and quality controls.
  1. Threshold Selection and Business Costs
  • How to choose a decision threshold given asymmetric costs of false positives vs. false negatives and potential moderation capacity constraints.
  1. Fairness Checks
  • What subgroup analyses and fairness metrics to run to guard against disparate impact.

Include assumptions where necessary, and provide formulas and examples for thresholding and weighting.

Clarifying Questions to Ask Guidance

  • Clarify the task, data shape, labels, constraints, and evaluation metric.
  • State assumptions behind the math or modeling technique you choose.
  • Connect theory to practical training, debugging, and deployment implications.

What a Strong Answer Covers Guidance

  • Correct definitions and formulas where the prompt requires them.
  • A practical explanation of how the method behaves on real data.
  • Trade-offs, failure modes, diagnostics, and mitigation strategies.
  • Evaluation choices that match the product or modeling objective.

Follow-up Questions Guidance

  • How would noisy labels, class imbalance, or distribution shift affect the answer?
  • What would you monitor after deployment?
  • Which baseline would you compare against first?
Loading comments...

Browse More Questions

More Machine Learning•More Meta•More Data Scientist•Meta Data Scientist•Meta Machine Learning•Data Scientist Machine Learning