Choose evaluation metrics for imbalanced risk model
Company: OneMain Financial
Role: Data Scientist
Category: Machine Learning
Difficulty: medium
Interview Round: Technical Screen
You build a fraud detector with 1% positives. Business costs: false negative = $100, false positive = $1; true positives/true negatives have zero cost. 1) Derive the Bayes-optimal probability threshold for a calibrated classifier that minimizes expected cost and compute its numeric value. 2) Decide whether ROC-AUC, PR-AUC, F1, MCC, KS, or cost-based metrics best reflect business goals and justify. 3) Describe how you would choose a threshold on a validation set to maximize expected profit while ensuring a cap of ≤0.5% manual review rate. 4) Explain how you would verify calibration (e.g., reliability diagrams, Brier score) and recalibrate (Platt vs isotonic) without leakage.
Quick Answer: This question evaluates understanding of cost-sensitive decision making, probabilistic thresholding, evaluation metric selection under class imbalance, constrained operating points for manual review caps, and probability calibration.