Choose Metrics for Evaluating Fake-User Classifier
Quick Overview
Evaluates metric selection for a fake-user classifier under class imbalance and asymmetric error costs. Strong answers use precision, recall, F-beta, PR-AUC, precision at K, calibration, thresholding, review capacity, and production guardrails to protect legitimate users.
Choose Metrics for Evaluating Fake-User Classifier
Company: Meta
Role: Data Scientist
Category: Machine Learning
Difficulty: medium
Interview Round: Onsite
##### Scenario
Spike in daily average comments may be driven by fake users; you must build a classifier to detect them.
##### Question
Which evaluation metrics would you choose for the fake-user classifier and why?
##### Hints
Discuss precision, recall, F1, ROC-AUC, business cost of false positives vs. false negatives.
Quick Answer: Evaluates metric selection for a fake-user classifier under class imbalance and asymmetric error costs. Strong answers use precision, recall, F-beta, PR-AUC, precision at K, calibration, thresholding, review capacity, and production guardrails to protect legitimate users.
Choose Metrics for Evaluating Fake-User Classifier
Meta
Jul 12, 2025, 6:59 PM
mediumData ScientistOnsiteMachine Learning
19
0
Choose Metrics for Evaluating a Fake-User Classifier
A sudden spike in daily average comments may be driven by fake users. You are asked to build a binary classifier that flags fake accounts for review or enforcement.
Constraints & Assumptions
Fake users are rare, so the classification problem is imbalanced.
False positives can harm legitimate users, while false negatives allow platform abuse to continue.
Human review or enforcement capacity may be limited.
Metrics should support thresholding and operational decisions.
Clarifying Questions to Ask Guidance
What action follows a positive prediction: review, rate limit, challenge, or account removal?
What is the estimated prevalence of fake users?
Which error is more costly, and does that differ by enforcement action?
Are labels reliable, delayed, or biased by prior enforcement?
Part 1 - Choose Core Metrics
Which evaluation metrics would you choose for the fake-user classifier and why?
What This Part Should Cover Guidance
Precision, recall, F1 or F-beta, and how beta reflects business cost.
False positive rate, false negative rate, and confusion matrix interpretation.
Why accuracy can be misleading under class imbalance.
Part 2 - Use Ranking and Capacity Metrics
Which ranking or capacity-aware metrics would you use?
What This Part Should Cover Guidance
PR-AUC versus ROC-AUC and why PR-AUC is useful for rare positives.
Precision at K, recall at fixed review capacity, lift, and score calibration.
Threshold selection for different enforcement tiers.
Part 3 - Monitor Production Guardrails
What production guardrails and segment checks would you monitor?
What This Part Should Cover Guidance
Appeal rate, user complaints, legitimate-user harm, reviewer load, enforcement latency, drift, and adversarial adaptation.
Segment fairness and false positive rates across geography, language, user age, tenure, and activity types.
Feedback loops from model-driven enforcement.
What a Strong Answer Covers Guidance
A strong answer chooses metrics that match the operational action and error costs, handles rare positives with PR and capacity-aware metrics, and protects legitimate users through calibration, thresholds, and segment guardrails.
Follow-up Questions Guidance
What precision would you require for automatic account removal?
How would you evaluate labels if fake-account ground truth is noisy?
What if the model catches obvious fake users but misses new attack patterns?