Design and validate a cost-sensitive classifier
Company: Snowflake
Role: Data Scientist
Category: Machine Learning
Difficulty: hard
Interview Round: Technical Screen
You must ship a binary classifier that predicts whether a user will purchase within the next 7 days. Positives are rare (~3%). There is a label delay of up to 10 days; features include recent session stats, counts, and recency by category. Business values: TP yields +$2 expected margin, FP costs -$0.10 (annoyance/discount), FN costs -$0.50 (missed margin), TN yields $0. Requirements:
A) Propose the end-to-end training/evaluation design that avoids leakage under delayed labels. Specify the exact time-based cross-validation scheme (fold boundaries, feature/label windows) and why it’s unbiased.
B) Choose offline metrics and show how you would calibrate the model (e.g., Platt/Isotonic). Provide the formula to select the decision threshold that maximizes expected profit under the given costs; explain how you’d check threshold stability across cohorts.
C) Handle distribution shift: outline drift detection on covariates and on calibration (e.g., PSI, ECE). Propose an online monitoring dashboard with guardrails.
D) Latency and interpretability: with a 50 ms p95 budget and 64 MB RAM per request, describe a deployable modeling choice and featurization plan (including any precomputed features) that meets constraints, plus a fallback rule when the model is unavailable.
E) Explain the model and decisions to a non-technical stakeholder and reconcile if they insist on a different threshold. What evidence would you present to align on the target operating point?
Quick Answer: This question evaluates competence in cost-sensitive binary classification, handling delayed and imbalanced labels, calibration and decision-threshold selection, distributional-drift detection, monitoring, and low-latency deployment within the Machine Learning / Data Science domain.