This question evaluates proficiency in end-to-end machine learning engineering, including exploratory data analysis, time-aware train/validation/test splitting, model baseline and improvement, class imbalance strategies, leakage-safe hyperparameter tuning, metric computation and calibration, error analysis, reproducible training pipelines with CLI/config/seed control, explainability (feature importance/SHAP and ablations), and documentation of risks, fairness checks and monitoring hooks. It is commonly asked to assess practical ability to manage the full ML lifecycle and data hygiene in production-like scenarios, testing applied Data Manipulation (SQL/Python) and Machine Learning competencies with an emphasis on practical application while also requiring conceptual understanding of evaluation, calibration and fairness.
Given a labeled dataset for binary classification, implement an end-to-end Python solution to train and analyze a classifier. Tasks:
(
1) perform EDA (missingness, outliers, leakage checks, target/feature drift over time),
(
2) create time-aware, stratified train/validation/test splits with proper cross-validation,
(
3) build a strong baseline and at least one improved model,
(
4) handle class imbalance (cost-sensitive loss, resampling, thresholds),
(
5) tune hyperparameters without leakage,
(
6) compute and compare metrics (ROC-AUC, PR-AUC, F1, calibration/Brier, confusion matrix at chosen threshold),
(
7) conduct error analysis by slice and feature,
(
8) produce a reproducible training script with CLI, config, and seed control,
(
9) explain feature importance/SHAP and validate with ablations, and
(
10) document risks, fairness checks, and monitoring hooks for production. Provide code snippets and explain your design choices.
Overview: This question evaluates proficiency in end-to-end machine learning engineering, including exploratory data analysis, time-aware train/validation/test splitting, model baseline and improvement, class imbalance strategies, leakage-safe hyperparameter tuning, metric computation and calibration, error analysis, reproducible training pipelines with CLI/config/seed control, explainability (feature importance/SHAP and ablations), and documentation of risks, fairness checks and monitoring hooks. It is commonly asked to assess practical ability to manage the full ML lifecycle and data hygiene in production-like scenarios, testing applied Data Manipulation (SQL/Python) and Machine Learning competencies with an emphasis on practical application while also requiring conceptual understanding of evaluation, calibration and fairness.
You are given a labeled dataset for binary classification. Implement an end-to-end Python solution that trains a classifier and analyzes it to a production standard. This is an open-ended ML coding exercise: the interviewer cares less about the final ROC-AUC number than about whether you build a leakage-free, reproducible, production-aware pipeline and can justify every design choice. Provide working code snippets and explain your decisions as you go.
The work is organized into the parts below. Treat them as a single coherent pipeline (later parts depend on the splits and models from earlier parts), and call out any cross-cutting concerns — especially data leakage and reproducibility — as they arise.
Constraints & Assumptions
The target is binary; assume it may be
imbalanced
(rare positive class is common in fraud/churn/medical-style data).
The dataset is
tabular
with mixed numeric and categorical features, and may contain missing values.
Assume a
timestamp column
is available (or ask whether one exists) so time-aware handling can be considered.
Standard Python ML stack is available:
pandas
,
numpy
,
scikit-learn
, and optionally
imbalanced-learn
,
xgboost
/
lightgbm
, and
shap
.
The end goal is a model intended for
deployment
, so train/serve consistency, monitoring, and risk handling are in scope.
Clarifying Questions to Ask Guidance
A candidate should scope the whole problem before coding:
Is the deployment setting
temporal
(predictions made on future data) or is the dataset a static i.i.d. snapshot? This determines whether splits must be time-aware.
What is the
base rate
of the positive class, and what are the
relative costs
of a false positive vs. a false negative? This drives the metric and the operating threshold.
Is there an
entity/group key
(e.g. user or account) that recurs across rows, requiring group-aware splitting to prevent contamination?
Are predicted
probabilities
consumed downstream (ranking, expected-value decisions), or only hard labels? This determines whether calibration matters.
Are there
sensitive attributes
or fairness/regulatory constraints to honor?
What are the dataset size and any
latency/throughput
requirements at serving time?
Part 1 — Exploratory Data Analysis
Perform EDA covering: missingness patterns, outliers, leakage checks (target leakage, time leakage, train/test contamination, ID-as-feature), and drift over time (covariate drift and label/prior drift).
What This Part Should Cover Guidance
Distinguishing informative missingness from MCAR (and adding a missingness indicator rather than silently imputing signal away).
Treating outliers (report vs. remove) appropriately for the model family chosen later.
A concrete leakage-detection procedure, not just naming the categories.
Both covariate drift and label/prior drift, with a test (not eyeballing alone).
Part 2 — Splits and Cross-Validation
Create time-aware, stratified train/validation/test splits with a proper cross-validation scheme. Explain when each splitting strategy is correct.
Clarifying Questions for this Part Guidance
Does the same entity appear in multiple rows (e.g. repeated users), and must groups be kept intact across splits?
What This Part Should Cover Guidance
Why a random split overstates performance when predictions are made on future data.
The interaction (and tension) between stratification, time ordering, and grouping.
Why the
test set is touched exactly once
, with validation/CV driving all decisions.
Part 3 — Baseline and Improved Model
Build a strong baseline and at least one improved model, both inside leakage-safe pipelines so that all learned preprocessing refits per CV fold.
What This Part Should Cover Guidance
A trivial baseline that exposes why accuracy is the wrong metric under imbalance.
A leakage-safe
Pipeline
/
ColumnTransformer
so preprocessing is refit per fold.
A justified improved model with the right preprocessing for its family (e.g. categorical handling).
Part 4 — Class Imbalance
Handle class imbalance using the available levers: cost-sensitive loss, resampling, and threshold selection. Explain which you prefer and why.
What This Part Should Cover Guidance
The most common resampling leakage trap (resample before split) and how to avoid it.
Choosing the operating threshold from
validation
data against business cost, separate from the model.
Stating the FP/FN cost asymmetry explicitly for the use case.
Part 5 — Hyperparameter Tuning Without Leakage
Tune hyperparameters in a way that does not leak — the search itself can overfit the validation data if done carelessly.
What This Part Should Cover Guidance
Why preprocessing must live inside the searched pipeline.
Tuning on a ranking metric appropriate to imbalance, not accuracy.
Keeping the threshold out of the threshold-free AUC search (set it afterward).
Part 6 — Metrics
Compute and compare a metric suite: ROC-AUC, PR-AUC, F1 (at the chosen threshold), calibration / Brier score, and the confusion matrix at the chosen threshold. Explain what each tells you and when it matters.
What This Part Should Cover Guidance
Why PR-AUC is preferred over ROC-AUC under heavy imbalance.
How calibration ties to earlier choices (
class_weight
/resampling distort probabilities; recalibrate on a held-out slice, never on test).
Reporting all test-set numbers from a
single
final evaluation.
Part 7 — Error Analysis
Conduct error analysis by slice and by feature to find where the model systematically fails, and tie findings to action.
What This Part Should Cover Guidance
Per-slice metrics (recall / FN-rate / PR-AUC), including a time/cohort slice that can reveal drift.
Inspecting high-confidence errors individually.
Translating a weak slice into a concrete action (missing feature, sub-model, slice-specific threshold, more data).
Part 8 — Reproducible Training Script
Produce a reproducible training script with a CLI, external config, and seed control — byte-identical results given the same config and seed.
What This Part Should Cover Guidance
A CLI + config-as-code design with no magic numbers in the body.
The full reproducibility surface (seeds everywhere, pinned deps, data hash).
Versioning the model
together with its threshold and feature list
— a model without its threshold is unusable.
Part 9 — Explainability and Ablation Validation
Explain feature importance / SHAP and validate the explanations with ablations — importance claims are credible only if removing the "important" features actually hurts the metric.
What This Part Should Cover Guidance
Using a generalization-based importance method, not biased train-set tree importances alone.
The ablation step that makes importance claims credible.
Circling a "hugely important + near-perfect predictor" feature back to the leakage check from Part 1.
Part 10 — Production Risks, Fairness, and Monitoring
Document the production risks, fairness checks, and monitoring hooks needed to operate the model safely.
What This Part Should Cover Guidance
Concrete monitoring hooks: input distribution/missingness, prediction distribution, post-label performance by slice, and a retraining trigger.
Fairness criteria (e.g. equalized odds, demographic parity, calibration-within-group), the trade-offs between them, and proxy-feature auditing.
Shipping the
whole pipeline
(not just the estimator) to prevent train/serve skew, with versioned rollback.
What a Strong Answer Covers Guidance
Across all parts, the interviewer is looking for the discipline of what you fit on which data, and what you measure — the code is secondary. Strong answers thread two invariants through every part:
Leakage discipline:
the test set is touched exactly once; every transform that learns from data (imputation, scaling, encoding, resampling, calibration, threshold selection) is fit on training folds only and applied to validation/test.
Production awareness:
the deliverable is a shippable pipeline plus its threshold behind drift and fairness monitoring, not just a high offline metric.
The interviewer says the same user appears in many rows and you must predict for
new users
. How do your splits, CV, and leakage checks change?
A "top" feature by SHAP turns out to be available only
after
the outcome is known. Walk through how you detect this and what you do.
After deployment, offline PR-AUC was 0.82 but live recall is collapsing. Which monitors fire first, and how do you distinguish covariate drift from label drift from train/serve skew?
You used
class_weight="balanced"
and the product team now needs well-calibrated risk scores. What changes, and what do you re-measure?