Metrics for Extremely Imbalanced Classification: ROC-AUC vs PR-AUC, Precision vs Recall
Company: ByteDance
Role: Software Engineer
Category: Machine Learning
Difficulty: medium
Interview Round: Technical Screen
In the machine learning fundamentals segment of a software engineering screen (no coding), the interviewer asks how you would evaluate a binary classifier when the data is extremely imbalanced, with positives a tiny fraction of all examples. The discussion moves from the area under the ROC curve to the area under the precision-recall curve, then to precision and recall read from the confusion matrix, and finally to why the emphasis differs between medical, financial and fake-account risk applications.
### Clarifying Questions
- Is the positive class the rare one: the disease, the fraudulent transaction, the fake account?
- Does the model output a score that will be thresholded, and who chooses the threshold?
- What happens to a flagged case: an automatic action, or human review with limited capacity?
- Is the test set drawn from the real class distribution, or was it resampled?
### Part 1 — Pick a metric for extreme imbalance
Explain why accuracy is a poor choice here, what ROC-AUC measures, and when PR-AUC is the better summary.
```hint Count the false alarms
Picture a test set with a million negatives and a thousand positives. Ask how much the false positive rate moves when a model wrongly flags ten thousand negatives, and how much precision moves.
```
#### What This Part Should Cover
- Why accuracy rewards predicting the majority class
- What ROC-AUC measures and why it does not change with the class ratio
- What PR-AUC measures, what a random model scores on it, and when to prefer it
### Part 2 — Precision and recall from the confusion matrix
Define the confusion matrix, precision and recall, and explain how precision and recall move as the decision threshold changes.
```hint Denominators
Lay out the four cells first, then ask which cells each metric uses as its denominator.
```
#### What This Part Should Cover
- Correct definitions in terms of the four cells
- The threshold trade-off and single-number summaries such as F1 or F-beta
- A worked example with imbalanced counts
### Part 3 — Different priorities in different domains
Medical, financial and fake-account risk applications weigh the two kinds of error differently. Explain how, and how you would choose the operating threshold in each.
```hint Who pays for each mistake
For each setting, ask who pays for a missed positive, who pays for a false alarm, and whether a person reviews a flagged case before anything happens.
```
#### What This Part Should Cover
- The relative cost of false negatives and false positives in each setting
- Turning those costs, or a review capacity, into a threshold
- How the action taken on a flag (block, review, extra verification) changes which metric matters
### What a Strong Answer Covers
- Recognizes that the majority class dominates accuracy and hides the volume of false alarms behind a small false positive rate
- Defines precision, recall and the axes of both curves correctly
- Connects the choice of metric and threshold to the cost of each error and to review capacity
- Evaluates on the natural class distribution with enough positives, and reports uncertainty
### Follow-up Questions
- If you train with class weights or by downsampling negatives, how do the predicted probabilities and the evaluation change?
- PR-AUC depends on prevalence. How would you compare one model across two markets with different fake-account rates?
- A review team can examine only a fixed number of accounts a day. Which metric do you report?
- Labels arrive only for accounts that someone reviewed. How does that bias your evaluation?
Overview: A machine learning fundamentals question on evaluating a binary classifier when positives are extremely rare. It asks why accuracy misleads, how ROC-AUC and PR-AUC differ, how precision and recall come from the confusion matrix, and how the preferred trade-off changes between medical, financial and fake-account risk settings.
Read the full ByteDance Software Engineer interview experience this question came from