Take‑home: Imbalanced Binary Classification with Temporal Split, Calibration, and Operating Point Selection

Context

You are given an event‑level dataset for a binary classification problem with severe class imbalance (positive rate ≈ 1%). The goal is to build a reproducible modeling pipeline, evaluate it with appropriate metrics, and propose a principled operating point for production.

Data

• Columns per row:
  • id
  • event_date (YYYY-MM-DD)
  • region ∈ {NA, EU, APAC, LATAM, MEA}
  • Numerical features f1, f2, …, f50
  • Label y ∈ {0, 1}

Temporal Splits (no leakage)

• Train: event_date ≤ 2025-06-01
• Validation: 2025-06-02 – 2025-08-01
• Test: 2025-08-02 – 2025-09-01

Tasks

(a) Build a reproducible training pipeline that:

1. Performs the temporal split as specified.
2. Applies StandardScaler to numeric features and OneHotEncoder to region.
3. Handles class imbalance inside CV folds (e.g., class_weight='balanced' or SMOTE/SMOTENC within each fold without leaking validation data).
4. Trains a strong baseline (e.g., calibrated logistic regression or gradient boosting) and outputs well‑calibrated probabilities (Platt/sigmoid or isotonic, calibrated on the validation split).

(b) Report on the test split:

• ROC-AUC and PR-AUC
• Recall at 5% FPR
• The decision threshold that maximizes F1 subject to recall ≥ 0.90 (select the threshold using validation, then report the chosen value and resulting test performance)

(c) Describe how to choose the operating point for a production system with a hard requirement of at most 2 false positives per 1,000 predictions.

(d) Discuss how calibration might drift over time and one technique to monitor and re‑calibrate without label leakage.