Evaluate fraud classifier with cost-sensitive metrics
Company: Meta
Role: Data Scientist
Category: Machine Learning
Difficulty: hard
Interview Round: Technical Screen
You inherit a binary fraud classifier. a) Given a holdout confusion matrix TP=200, FP=800, FN=100, TN=99,900 at a specific threshold, compute precision, recall, F1, and false-positive rate. b) If a false negative costs $20 and a false positive costs $0.20 with a 1% prevalence, describe how to choose a threshold using predicted probabilities to maximize expected utility; specify which metric (e.g., PR-AUC vs. ROC-AUC) better reflects improvements at low prevalence and why. c) Outline a calibration check and fix (e.g., reliability plot, isotonic/Platt) and how calibration interacts with threshold selection. d) Propose an online evaluation plan with guardrails to prevent over-blocking legitimate users while improving catch rate (e.g., interleaved shadow evaluation, two-stage review for low-confidence positives), and define success criteria and rollback triggers.
Overview: This question evaluates competency in binary classifier evaluation, cost-sensitive decision-making, probability calibration, and safe online deployment within the Machine Learning domain for a Data Scientist role, requiring practical application informed by conceptual understanding.
Read the full Meta Data Scientist interview experience this question came from