Handling Class Imbalance: Metrics, Resampling, Class Weights and Decision Thresholds

How do you handle imbalanced data, where one class in a classification problem is far rarer than the other? ```hint Start from the mistakes Before choosing a technique, ask what score a model that always predicts the common class would get, and which mistake costs more here: missing a rare case or raising a false alarm. ``` ### Clarifying Questions - What is the prediction task, and roughly how rare is the minority class: a few percent of examples, or one in many thousands? - Which error is more expensive, a missed rare case or a false alarm, and is there a cap on how many flagged cases people can review? - Will the model see the same class mix in production as in the training data, and does it need calibrated probabilities or only a good ranking? - How many minority examples exist in absolute terms, and can more be labeled? ### What a Strong Answer Covers - Why accuracy misleads, and which metrics and validation splits to use instead - Data-level fixes such as collecting more minority examples, undersampling, oversampling and synthetic examples, with their pitfalls - Algorithm-level fixes such as class weights and cost-sensitive losses - Choosing the decision threshold from error costs or review capacity, and correcting probabilities after resampling - When the imbalance is extreme enough to reframe the problem, for example as anomaly detection ### Follow-up Questions - After undersampling the majority class, the model's predicted probabilities look far too high. Why, and how do you fix them? - How would you explain to a business stakeholder that a model with lower accuracy is the better model? - What changes if you have only a few hundred labeled examples of the rare class?