Build and evaluate a conversion prediction model
Company: Coinbase
Role: Data Scientist
Category: Machine Learning
Difficulty: hard
Interview Round: Onsite
You receive a CSV of email exposures with one row per user-email send: cols = user_id, send_ts, treatment_flag, opened, clicked, purchased_within_7d (label), user_region, device_type, tenure_days, prior_sessions_28d, prior_purchases_180d, avg_cart_value_180d, categories_viewed_28d, email_personalization_score, deliverability_score, and other anonymized features. Task:
1) EDA: Identify leakage risks and fix them by ensuring features are computable at send_ts (e.g., avoid using post-send behaviors like opened/clicked unless modeling an uplift chain). Explore class imbalance, missingness, outliers, and high-cardinality categoricals. Propose data quality checks.
2) Modeling: Build a baseline logistic regression and a gradient-boosted tree to predict purchased_within_7d. Use time-based splits: train=2025-06, valid=2025-07, test=2025-08. Use cross-validation on train, tune hyperparameters, and implement monotonicity or regularization as appropriate.
3) Evaluation: Report ROC AUC, PR AUC, calibration (reliability curve, Brier score), and incremental lift at the top 10% scored users versus control. Provide confidence intervals via bootstrap and assess stability by region/device.
4) Deployment: Choose a decision threshold to maximize expected incremental revenue given an email cost of $0.003 and an estimated treatment effect from a calibration experiment. Describe monitoring (data drift, performance drift, alerting), retraining cadence, and next steps to improve (feature engineering, causal uplift modeling, de-biasing via IPS or DR estimators).
Quick Answer: This question evaluates proficiency in predictive modeling, feature engineering with target-leakage control, time-aware validation, uplift and treatment-effect estimation, calibration and uncertainty quantification, and end-to-end deployment considerations within the Machine Learning domain.