Choose Metrics for Evaluating Fake-User Classifier
Overview
This question evaluates a Data Scientist's ability to choose and interpret evaluation metrics for imbalanced binary classifiers, covering concepts such as precision/recall trade-offs, ROC-AUC versus PR-AUC, capacity-aware metrics and cost-sensitive thresholding, and is categorized in the Machine Learning domain as a practical application requiring conceptual understanding. It is commonly asked because production fraud- and abuse-detection problems demand reasoning about class imbalance, asymmetric business costs of false positives versus false negatives, and limited human-review or enforcement capacity when selecting metrics and operating thresholds.
Classifier Evaluation for Detecting Fake Users
Scenario
A sudden spike in daily average comments may be driven by fake users. You are asked to build a binary classifier that flags fake accounts for enforcement.
Task
Which evaluation metrics would you choose for the fake-user classifier and why?
Assumptions (explicit for clarity)
- Binary classification with strong class imbalance (fake users are rare).
- There is a limited human-review capacity and/or automated enforcement at high-confidence scores.
- Business costs of false positives (blocking a real user) and false negatives (missing a fake user) are not equal.
Guidance
Discuss:
- Precision, recall, and F1/Fβ and how to pick β based on business cost.
- ROC-AUC vs PR-AUC and when each is appropriate.
- Capacity-aware metrics (e.g., precision@k, recall at fixed precision).
- Cost-sensitive evaluation (business cost of false positives vs false negatives) and threshold selection.
Solution
Login required