Imbalanced Binary Classification: Learning, Evaluation, and Model Selection

Context

You are training a binary classifier where the positive class is rare (for example, 0.1–5% prevalence). You need to choose training strategies, evaluation metrics, cross-validation structure, and tuning methods that remain reliable under severe class imbalance and potential dataset shift.

Tasks

1. Explain the impact of class imbalance on both learning and evaluation.
2. Compare strategies to handle imbalance:
   • Random over-sampling and under-sampling
   • Synthetic methods (e.g., SMOTE, ADASYN)
   • Class weighting / cost-sensitive learning
   • Focal loss
   • Threshold moving (post-hoc decision thresholding)
3. Describe how to structure cross-validation to avoid leakage:
   • Perform any resampling within each training fold only
   • Use stratified folds; consider grouped or time-based splits when relevant
4. Recommend appropriate metrics (e.g., PR AUC, recall at fixed precision, balanced accuracy) and how to choose among them.
5. Outline how to tune hyperparameters under imbalance, including threshold selection.
6. Discuss trade-offs across variance, bias, runtime, and calibration for the above strategies.