Cost-Sensitive Thresholding and Calibration
Asked of: Data Scientist
Last updated
-
What it is — Cost-sensitive thresholding sets a classifier’s decision cutoff to minimize expected business loss when false positives and false negatives have different costs. Calibration adjusts model scores so predicted probabilities match observed frequencies, making cost-based thresholding valid.
-
Why interviewers ask about it — At companies like Meta, models power safety, integrity, and ads. A poor threshold can flood human reviewers, block good users, or leak harm; miscalibrated probabilities break budget pacing, auctions, and alerting SLAs.
-
Core ideas to know
- With calibrated p = P(y=1|x) and only misclassification costs, optimal threshold t* = C_fp / (C_fp + C_fn).
- Choose label minimizing expected cost: predict positive if C_fp(1−p) < C_fn p.
- Calibration methods: Platt/sigmoid, isotonic regression, and temperature scaling; fit on held-out data.
- AUC doesn’t pick a threshold; use expected cost, or constraints on precision/recall/service load.
- Down/over-sampling skews base rates; calibrate after sampling and compute thresholds using true deployment prevalence.
- Monitor ECE/Brier and business loss post-deployment; recalibrate and re-tune thresholds as drift or segment differences emerge.
- For deep nets, temperature scaling is a strong, low-variance post-hoc calibrator for multiclass outputs.
-
A common pitfall — Candidates optimize AUC and ship a 0.5 cutoff without asking about costs or calibration. Example: in fraud, C_fn = 1 implies t* ≈ 1/(100+1) ≈ 0.0099; using 0.5 misses costly fraud. Another trap is thresholding scores from a downsampled dataset without recalibration, causing wild offline→online deltas. Always calibrate on a validation set reflecting production, then compute and validate the cost-minimizing cutoff.
-
Further reading
- The Foundations of Cost-Sensitive Learning (Elkan, IJCAI 2001) — canonical derivation of optimal decision rules and thresholds. (cseweb.ucsd.edu)
- scikit-learn: Probability calibration — practical guidance and APIs for sigmoid/isotonic/temperature scaling, with caveats and examples. (scikit-learn.org)
- On Calibration of Modern Neural Networks (Guo et al., ICML 2017) — shows modern nets are overconfident; temperature scaling as a simple, effective fix. (proceedings.mlr.press)