Choose Metrics for Evaluating Fake-User Classifier

Q: Choose Metrics for Evaluating Fake-User Classifier

This question evaluates a Data Scientist's ability to choose and interpret evaluation metrics for imbalanced binary classifiers, covering concepts such as precision/recall trade-offs, ROC-AUC versus PR-AUC, capacity-aware metrics and cost-sensitive thresholding, and is categorized in the Machine Learning domain as a practical application requiring conceptual understanding. It is commonly asked because production fraud- and abuse-detection problems demand reasoning about class imbalance, asymmetric business costs of false positives versus false negatives, and limited human-review or enforcement capacity when selecting metrics and operating thresholds.

Q: How do I approach Machine Learning interview questions?

Machine Learning questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master machine learning interviews.

Question

Classifier Evaluation for Detecting Fake Users

Scenario

A sudden spike in daily average comments may be driven by fake users. You are asked to build a binary classifier that flags fake accounts for enforcement.

Task

Which evaluation metrics would you choose for the fake-user classifier and why?

Assumptions (explicit for clarity)

Binary classification with strong class imbalance (fake users are rare).
There is a limited human-review capacity and/or automated enforcement at high-confidence scores.
Business costs of false positives (blocking a real user) and false negatives (missing a fake user) are not equal.

Guidance

Discuss:

Precision, recall, and F1/Fβ and how to pick β based on business cost.
ROC-AUC vs PR-AUC and when each is appropriate.
Capacity-aware metrics (e.g., precision@k, recall at fixed precision).
Cost-sensitive evaluation (business cost of false positives vs false negatives) and threshold selection.