Evaluate fraud classifier with cost-sensitive metrics

Quick Overview

This question evaluates competency in binary classifier evaluation, cost-sensitive decision-making, probability calibration, and safe online deployment within the Machine Learning domain for a Data Scientist role, requiring practical application informed by conceptual understanding.

Evaluate fraud classifier with cost-sensitive metrics

Company: Meta

Role: Data Scientist

Category: Machine Learning

Difficulty: hard

Interview Round: Technical Screen

You inherit a binary fraud classifier. a) Given a holdout confusion matrix TP=200, FP=800, FN=100, TN=99,900 at a specific threshold, compute precision, recall, F1, and false-positive rate. b) If a false negative costs $20 and a false positive costs $0.20 with a 1% prevalence, describe how to choose a threshold using predicted probabilities to maximize expected utility; specify which metric (e.g., PR-AUC vs. ROC-AUC) better reflects improvements at low prevalence and why. c) Outline a calibration check and fix (e.g., reliability plot, isotonic/Platt) and how calibration interacts with threshold selection. d) Propose an online evaluation plan with guardrails to prevent over-blocking legitimate users while improving catch rate (e.g., interleaved shadow evaluation, two-stage review for low-confidence positives), and define success criteria and rollback triggers.

Quick Answer: This question evaluates competency in binary classifier evaluation, cost-sensitive decision-making, probability calibration, and safe online deployment within the Machine Learning domain for a Data Scientist role, requiring practical application informed by conceptual understanding.

|Home/Machine Learning/Meta
Meta logo
Meta
Oct 13, 2025, 9:49 PM
hardData ScientistTechnical ScreenMachine Learning
3
0

Binary Fraud Classifier: Metrics, Thresholding, Calibration, and Online Evaluation

You inherit a binary fraud classifier used to decide whether to block an event. At one operating threshold, the holdout confusion matrix is:

  • TP = 200, FP = 800, FN = 100, TN = 99,900

Answer the following:

(a) Compute precision, recall, F1, and false-positive rate (FPR) at this threshold.

(b) Assuming a false negative costs 20andafalsepositivecosts20 and a false positive costs 0.20 in an environment with 1% fraud prevalence, describe how to choose a threshold using predicted probabilities to maximize expected utility. State which ranking metric (PR-AUC vs. ROC-AUC) better reflects improvements at low prevalence and why.

(c) Outline how you would check and fix probability calibration (e.g., reliability plot, isotonic or Platt scaling) and explain how calibration interacts with threshold selection.

(d) Propose an online evaluation plan with guardrails that avoids over-blocking legitimate users while improving catch rate. Include ideas such as shadow evaluation and a two-stage review for low-confidence positives, and define concrete success criteria and rollback triggers.

Loading comments...