Explain AUC, activations, ensembles, and imbalance
Overview
This question evaluates competency in model evaluation metrics (ROC AUC and Average Precision), handling class imbalance, choice of output activations and loss functions, robustness to outliers (MSE vs MAE), ensemble methods, and overfitting diagnostics within the Machine Learning domain.
Machine Learning Metrics and Modeling Choices — Multi-part
You are given model scores and binary labels for a small dataset and asked to compute ROC AUC manually, then answer modeling and evaluation questions.
Given:
- Scores s = [0.10, 0.40, 0.35, 0.80, 0.60]
- Labels y = [0, 1, 0, 1, 0]
Answer all sub-questions precisely:
1) AUC / Ranking
- Compute ROC AUC exactly via pairwise positive–negative comparisons (no libraries). Treat ties as 0.5 if any.
- List the ROC points (FPR, TPR) by thresholding from highest score to lowest and compute the AUC by trapezoids; both methods should match.
- With extreme class imbalance (1% positives), explain how you would interpret AUC vs Average Precision (AP) and which you would favor.
2) Output Activations and Losses
For each scenario, choose an output-layer activation and loss, and justify:
- (a) Single-label multi-class (K = 7)
- (b) Multi-label (K = 7)
- (c) Bounded regression in [0, 1]
- (d) Unbounded regression with outliers
Also discuss vanishing gradients for sigmoid/tanh and why leaky-ReLU or GELU might help in hidden layers.
3) MSE vs MAE
Explain optimization and robustness differences: gradients, influence of outliers, and mean vs median optimality.
4) Ensembles
Contrast bagging vs boosting in terms of bias/variance and when you’d choose each for noisy data.
5) Overfitting
Name two concrete, testable diagnostics (with plots/metrics) and two mitigation tactics that won’t leak validation information.
Solution
Login required