Fixing a Churn Classifier: Encoding, Imbalance, Evaluation, and Fairness

Context

You inherit a binary classifier that predicts churn=1. The current implementation has the following issues: (a) the feature "month" is encoded as an integer 1–12 and used linearly; (b) continuous features of very different scales are not standardized; (c) positives are 3% of the data; (d) model performance is reported by overall accuracy. The business requirement is Recall ≥ 80% at Precision ≥ 30%.

Tasks

1. Identify and explain the modeling mistakes. For the "month" feature, propose and justify two encodings (e.g., one-hot vs. cyclical sin/cos) and state when you’d prefer each.
2. Describe a training procedure to address scale and class imbalance, including:
   • Feature standardization choices.
   • Class-weighted loss vs. resampling (SMOTE vs. undersampling).
   • How to select the weight ratio or sampling rate without biasing validation.
3. Specify an evaluation protocol covering:
   • Stratified cross-validation.
   • Primary metrics (PR AUC, recall at precision ≥ 0.30).
   • Calibration assessment and how to select a threshold that satisfies the operating point.
   • How to compute and report confidence intervals for these metrics.
4. If the dataset is imbalanced across subgroups, define a cost-sensitive or group-weighted objective and the fairness/coverage checks you’d add before deployment. Provide a concrete example of a failure mode you would catch and how you’d mitigate it.