Tune classifier and compute key metrics
Overview
This question evaluates practical proficiency in classifier evaluation, threshold tuning, precision/recall trade-offs, cost-sensitive decision rules, and designing hybrid models to predict error amounts within the Machine Learning / Data Science domain.
Payment Error Classifier — Evaluation, Thresholding, and Cost-Sensitive Design
Context
You built a binary classifier to flag incorrect payments (positive class). On a validation set of 50,000 transactions, the true prevalence of incorrect payments is 2% (1,000). At your current threshold, the model flags 1,800 positives, of which 600 are truly incorrect.
Tasks
A) Compute precision, recall, F1, specificity, false-positive rate, and the confusion-matrix counts (TP, FP, TN, FN). Show arithmetic.
B) If the business requires precision ≥ 0.80 while maximizing recall, describe how you would choose the operating threshold using PR curves and cross-validation; specify the selection rule and how you’d report the trade-off.
C) In what product scenarios would you prioritize recall over precision, and vice versa, for payment accuracy operations? Give one concrete example each.
D) The cost to manually review a flagged transaction is 200. Propose a cost-sensitive objective or decision rule and explain how it changes thresholding.
E) Would you model incorrect-payment detection as classification or regression if you also want to predict the monetary error amount? Outline a hybrid approach (e.g., two-stage model) and justify it.
Solution
Login required