Debug a failing ML classifier

Q: Debug a failing ML classifier

This question evaluates a candidate's ability to diagnose a poorly generalizing ML classifier, testing practical knowledge of data leakage, feature drift, and model calibration. It assesses applied machine learning debugging skills — including cross-validation design and error analysis — commonly tested in senior engineering and ML roles.

Q: How do I approach Machine Learning interview questions?

Machine Learning questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master machine learning interviews.

Q: What difficulty level is this interview question?

This is a hard difficulty Machine Learning question, commonly asked during Technical Screen rounds at OpenAI.

Q: What role is this question designed for?

This question is commonly asked for Software Engineer candidates at OpenAI during technical interviews.

Question

Debugging a Churn Prediction Pipeline With Poor Generalization

Context

You have inherited a binary churn prediction system. The goal is to predict whether a customer will churn in the next period, using only information available up to an "as-of" cutoff time. The current numbers are:

Training ROC AUC: 0.95
Validation ROC AUC (random split): 0.62
Time-based holdout ROC AUC (most recent month): 0.55
Predicted probabilities are overconfident (scores cluster near 0 and 1, but observed outcomes do not match).
Positive-class prevalence ≈ 1:10 (about 9–10% positive).

Task

Describe, step by step, how you would debug this system. Your answer should cover:

Data validation and leakage checks (including temporal leakage).
Label and feature drift analysis.
Cross-validation scheme selection.
Error analysis — by slices, calibration, and threshold-dependent confusion matrices.
Ablations and feature audits.
Training issues — regularization, class weighting, resampling.

Propose concrete experiments to isolate the root causes, name the metrics you would inspect, and recommend fixes plus a plan to verify improvements and prevent regressions (tests, data versioning, monitoring).

Constraints & Assumptions

Features must use only events strictly before as_of_date ; the label is derived strictly after it, over a fixed horizon $[t, t+H)$ .
The model is trained offline and scored in batch; a deployed prior (base rate) may differ from the training prior, especially after any resampling/weighting.
Customers can have multiple rows over time (one per (customer_id, as_of_date) ), so naive row-level random splits can place the same customer on both sides.
A retention action is taken at a chosen operating point (e.g. "contact the top-k highest-risk customers"), so decision quality at a threshold matters, not just global ranking.
Assume you can re-run the training pipeline, re-cut splits, and add tests, but you cannot collect new ground-truth faster than labels mature.

Clarifying Questions to Ask Guidance

What is the exact churn label definition and horizon $H$ — voluntary vs involuntary, paid-activity-based vs login-based, and how are late-within-horizon churners and right-censored (not-yet-matured) recent customers handled?
How are features built relative to the cutoff — fixed lookback windows? Any absolute dates or fields (contract-end, cancellation flags) that could be set by or after the churn event?
How were the current splits constructed — random by row or grouped by customer? Were preprocessing transforms and any target encoding fit before or after the split?
What is the business operating point and cost structure — what fraction of customers can be contacted, intervention cost $c$ , saved-customer value $v$ , and intervention success rate $s$ ?
Has the base rate or population shifted recently (pricing change, new acquisition channel, seasonality) that would explain the recent-month drop?
What are the acceptance bar and constraints — latency, interpretability, a baseline to beat, and how much label-maturation delay is tolerable before promoting a model?

What a Strong Answer Covers Guidance

Reads the three numbers correctly : identifies two distinct gaps (overfit/leakage in training; entity-leakage and/or drift between random and time splits) and treats 0.55 as near-chance under deployment.
Separates ranking from calibration : recognizes overconfidence as a calibration failure invisible to AUC, and adds PR AUC, log loss, Brier, ECE, and top-k capture given 1:10 imbalance.
Enumerates leakage types (temporal, entity, preprocessing/target-encoding) and proposes concrete checks/assertions plus cheap probes (shuffle-label, single-feature scan).
Switches to the right evaluator : rolling-origin, group-aware (by customer), time-ordered CV with an untouched recent holdout and a regularized baseline.
Quantifies drift (base-rate/label, covariate via PSI/KS, concept drift, prediction drift) and distinguishes drift from leakage as root cause.
Does decision-relevant error analysis : per-slice metrics, reliability curves + post-hoc calibration (Platt/isotonic), threshold-dependent confusion matrices, and expected-utility thresholding.
Addresses training and imbalance : regularization/early stopping to close the train–val gap; class weights/focal loss over aggressive oversampling; prior-odds correction when deploy ≠ train base rate.
Closes the loop : prioritized experiments with exit criteria, a verification/backtest plan with acceptance criteria, and regression guards (CI leakage tests, data/feature/label/model versioning, production drift+calibration monitoring with a fallback).

Follow-up Questions Guidance

After you fix the splits and leakage, what should happen to the train AUC , and why is a train AUC that stays at 0.95 itself a red flag?
Calibration improves log loss/Brier/ECE but does not change AUC — explain why, and when calibration alone is the right fix vs. when it masks a ranking problem.
If covariate distributions are stable but the feature→churn relationship flips month over month (concept drift), which of your fixes still help and which are useless?
Suppose retraining is expensive and you can ship only one change this week — given these symptoms, which single intervention has the best expected payoff, and what evidence would justify it?

Debug a failing ML classifier

Quick Overview

Debug a failing ML classifier

Debugging a Churn Prediction Pipeline With Poor Generalization

Context

Task

Constraints & Assumptions

Clarifying Questions to Ask Guidance

What a Strong Answer Covers Guidance

Follow-up Questions Guidance

Write your answer