Choose evaluation metrics for imbalanced risk model

Q: Choose evaluation metrics for imbalanced risk model

This question evaluates understanding of cost-sensitive decision making, probabilistic thresholding, evaluation metric selection under class imbalance, constrained operating points for manual review caps, and probability calibration.

Q: How do I approach Machine Learning interview questions?

Machine Learning questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master machine learning interviews.

Question

Loading...

Cost-Sensitive Fraud Detection: Thresholding, Metrics, and Calibration

Assume a binary fraud classifier outputs calibrated probabilities p = P(y=1|x). The base rate of fraud is 1%. Business costs:

False negative (missed fraud): $100
False positive (flagging a non-fraud): $1
True positives and true negatives: $0

Answer the following:

Derive the Bayes-optimal probability threshold that minimizes expected cost for a calibrated classifier, and compute its numeric value with the given costs.
Among ROC-AUC, PR-AUC, F1, MCC, KS, or cost-based metrics, identify which best reflect these business goals and justify your choice.
You must cap manual reviews to ≤0.5% of all cases. Describe how to choose a validation-set threshold that maximizes expected profit (minimizes expected cost) subject to that cap.
Explain how to verify calibration (e.g., reliability diagrams, Brier score) and how to recalibrate (Platt scaling vs. isotonic regression) without data leakage.

Choose evaluation metrics for imbalanced risk model

Cost-Sensitive Fraud Detection: Thresholding, Metrics, and Calibration

Solution

Comments (0)

Choose evaluation metrics for imbalanced risk model

Overview

Cost-Sensitive Fraud Detection: Thresholding, Metrics, and Calibration

Solution

Comments (0)