Choose ML metrics under asymmetric costs
Overview
This question evaluates a data scientist's competency in cost-sensitive binary classification, covering skills such as defining business cost matrices, threshold selection and probability calibration, handling extreme class imbalance, fairness assessment across segments, monitoring for drift and feedback loops, and designing experiments to measure long-term product impact. It is a machine learning domain question commonly asked to probe practical judgment about trade-offs between precision and recall, operational metrics, and product-level consequences, testing both conceptual understanding of decision theory and practical application to production ML.
Binary Classifier With Asymmetric Costs: Fraud vs. Cancer
Context: You own a production binary classifier and must make product/ML decisions under asymmetric error costs. Compare two use cases:
- (A) Credit-card fraud detection
- (B) Cancer detection (screening/triage)
Tasks:
- For each use case, define a business cost matrix (TP, FP, FN, TN) and state the metric(s) you would optimize (e.g., expected cost, recall at fixed precision, PR AUC). Explain why ROC AUC may be misleading in these settings.
- Describe how you would set thresholds using cost curves or iso-F1/iso-precision lines. Explain how probability calibration (Platt scaling or isotonic regression) changes the decision rule and threshold.
- Explain how you would handle extreme class imbalance (e.g., sampling, class weights, focal loss) and evaluate with stratified, time-aware cross-validation.
- Discuss fairness and the distribution of false-positive burden by segment; propose how to assess and mitigate disparities.
- List online metrics and logs to monitor drift and feedback loops after launch.
- Explain how short-term metric gains might reduce long-term product engagement, and propose an experiment to measure long-term impact.
Solution
Login required