Metrics for Extremely Imbalanced Classification: ROC-AUC vs PR-AUC, Precision vs Recall

Quick Overview

A machine learning fundamentals question on evaluating a binary classifier when positives are extremely rare. It asks why accuracy misleads, how ROC-AUC and PR-AUC differ, how precision and recall come from the confusion matrix, and how the preferred trade-off changes between medical, financial and fake-account risk settings.

Metrics for Extremely Imbalanced Classification: ROC-AUC vs PR-AUC, Precision vs Recall

Company: ByteDance

Role: Software Engineer

Category: Machine Learning

Difficulty: medium

Interview Round: Technical Screen

In the machine learning fundamentals segment of a software engineering screen (no coding), the interviewer asks how you would evaluate a binary classifier when the data is extremely imbalanced, with positives a tiny fraction of all examples. The discussion moves from the area under the ROC curve to the area under the precision-recall curve, then to precision and recall read from the confusion matrix, and finally to why the emphasis differs between medical, financial and fake-account risk applications. ### Clarifying Questions - Is the positive class the rare one: the disease, the fraudulent transaction, the fake account? - Does the model output a score that will be thresholded, and who chooses the threshold? - What happens to a flagged case: an automatic action, or human review with limited capacity? - Is the test set drawn from the real class distribution, or was it resampled? ### Part 1 — Pick a metric for extreme imbalance Explain why accuracy is a poor choice here, what ROC-AUC measures, and when PR-AUC is the better summary. ```hint Count the false alarms Picture a test set with a million negatives and a thousand positives. Ask how much the false positive rate moves when a model wrongly flags ten thousand negatives, and how much precision moves. ``` #### What This Part Should Cover - Why accuracy rewards predicting the majority class - What ROC-AUC measures and why it does not change with the class ratio - What PR-AUC measures, what a random model scores on it, and when to prefer it ### Part 2 — Precision and recall from the confusion matrix Define the confusion matrix, precision and recall, and explain how precision and recall move as the decision threshold changes. ```hint Denominators Lay out the four cells first, then ask which cells each metric uses as its denominator. ``` #### What This Part Should Cover - Correct definitions in terms of the four cells - The threshold trade-off and single-number summaries such as F1 or F-beta - A worked example with imbalanced counts ### Part 3 — Different priorities in different domains Medical, financial and fake-account risk applications weigh the two kinds of error differently. Explain how, and how you would choose the operating threshold in each. ```hint Who pays for each mistake For each setting, ask who pays for a missed positive, who pays for a false alarm, and whether a person reviews a flagged case before anything happens. ``` #### What This Part Should Cover - The relative cost of false negatives and false positives in each setting - Turning those costs, or a review capacity, into a threshold - How the action taken on a flag (block, review, extra verification) changes which metric matters ### What a Strong Answer Covers - Recognizes that the majority class dominates accuracy and hides the volume of false alarms behind a small false positive rate - Defines precision, recall and the axes of both curves correctly - Connects the choice of metric and threshold to the cost of each error and to review capacity - Evaluates on the natural class distribution with enough positives, and reports uncertainty ### Follow-up Questions - If you train with class weights or by downsampling negatives, how do the predicted probabilities and the evaluation change? - PR-AUC depends on prevalence. How would you compare one model across two markets with different fake-account rates? - A review team can examine only a fixed number of accounts a day. Which metric do you report? - Labels arrive only for accounts that someone reviewed. How does that bias your evaluation?

Overview: A machine learning fundamentals question on evaluating a binary classifier when positives are extremely rare. It asks why accuracy misleads, how ROC-AUC and PR-AUC differ, how precision and recall come from the confusion matrix, and how the preferred trade-off changes between medical, financial and fake-account risk settings.

Read the full ByteDance Software Engineer interview experience this question came from

ByteDance logo
ByteDance
Nov 10, 2026
mediumSoftware EngineerTechnical ScreenMachine Learning
1
0

In the machine learning fundamentals segment of a software engineering screen (no coding), the interviewer asks how you would evaluate a binary classifier when the data is extremely imbalanced, with positives a tiny fraction of all examples. The discussion moves from the area under the ROC curve to the area under the precision-recall curve, then to precision and recall read from the confusion matrix, and finally to why the emphasis differs between medical, financial and fake-account risk applications.

Clarifying Questions Guidance

  • Is the positive class the rare one: the disease, the fraudulent transaction, the fake account?
  • Does the model output a score that will be thresholded, and who chooses the threshold?
  • What happens to a flagged case: an automatic action, or human review with limited capacity?
  • Is the test set drawn from the real class distribution, or was it resampled?

Part 1 — Pick a metric for extreme imbalance

Explain why accuracy is a poor choice here, what ROC-AUC measures, and when PR-AUC is the better summary.

What This Part Should Cover Guidance

  • Why accuracy rewards predicting the majority class
  • What ROC-AUC measures and why it does not change with the class ratio
  • What PR-AUC measures, what a random model scores on it, and when to prefer it

Part 2 — Precision and recall from the confusion matrix

Define the confusion matrix, precision and recall, and explain how precision and recall move as the decision threshold changes.

What This Part Should Cover Guidance

  • Correct definitions in terms of the four cells
  • The threshold trade-off and single-number summaries such as F1 or F-beta
  • A worked example with imbalanced counts

Part 3 — Different priorities in different domains

Medical, financial and fake-account risk applications weigh the two kinds of error differently. Explain how, and how you would choose the operating threshold in each.

What This Part Should Cover Guidance

  • The relative cost of false negatives and false positives in each setting
  • Turning those costs, or a review capacity, into a threshold
  • How the action taken on a flag (block, review, extra verification) changes which metric matters

What a Strong Answer Covers Guidance

  • Recognizes that the majority class dominates accuracy and hides the volume of false alarms behind a small false positive rate
  • Defines precision, recall and the axes of both curves correctly
  • Connects the choice of metric and threshold to the cost of each error and to review capacity
  • Evaluates on the natural class distribution with enough positives, and reports uncertainty

Follow-up Questions Guidance

  • If you train with class weights or by downsampling negatives, how do the predicted probabilities and the evaluation change?
  • PR-AUC depends on prevalence. How would you compare one model across two markets with different fake-account rates?
  • A review team can examine only a fixed number of accounts a day. Which metric do you report?
  • Labels arrive only for accounts that someone reviewed. How does that bias your evaluation?