Explain AUC, imbalance, losses, and networks
Company: Boston Consulting Group
Role: Data Scientist
Category: Machine Learning
Difficulty: medium
Interview Round: Take-home Project
Answer all parts. A) Compute ROC points and ROC-AUC for the following toy scores by sweeping thresholds from +∞ down to −∞ (break ties by ranking positives above negatives at equal score): Positives: P1=0.99, P2=0.80, P3=0.60, P4=0.40, P5=0.10. Negatives: N1=0.95, N2=0.70, N3=0.55, N4=0.30, N5=0.05. Show the trapezoidal calculation and interpret the AUC as the probability a random positive ranks above a random negative. B) With prevalence = 1%, explain why accuracy can be misleading; propose two better metrics and when to prefer ROC-AUC vs PR-AUC. C) Contrast MSE vs MAE as training losses for a regression head: derive their gradients, explain robustness to outliers, and give one scenario where each is preferable. D) For a neural network binary classifier on the same imbalanced task, propose two architecture/training changes (e.g., focal loss with typical γ,α; class weighting; positive down/up-sampling; thresholding strategy) and discuss likely effects on calibration and recall.
Quick Answer: This question evaluates a candidate's understanding of imbalanced classification and regression concepts, including ROC/PR curves and AUC, prevalence effects on metrics, loss functions (MSE vs MAE) and their gradients, and neural network training strategies for calibration and recall.