Design and sample for credit default prediction
Company: Boston Consulting Group
Role: Data Scientist
Category: Machine Learning
Difficulty: medium
Interview Round: Technical Screen
A bank wants a model to predict 90-day credit card default at account-month level for proactive outreach. Class prevalence in production is about 2% defaults. Design the end-to-end approach and address sampling in depth.
a) Problem framing: Define the label precisely (observation window, prediction date, horizon), features available at scoring time, and a temporal data split that avoids leakage (e.g., train on data up to a cutoff date, validate on future months). List three concrete leakage risks unique to credit cards and how you would detect them.
b) Metrics: Choose evaluation metrics and operating points for imbalanced data (e.g., PR-AUC, recall at 5% FPR, expected utility). Justify why they match business goals under 2% prevalence.
c) Sampling strategy: You downsample negatives to speed training and target a 1:3 positive:negative ratio in the training set. Let π be true prevalence (0.02 in production) and π_s be the training prevalence (0.25 after downsampling). If the model outputs p_s = P(default | x, sampled) = 0.60 for an account, derive and compute the calibrated population probability p_pop = P(default | x) using prior-probability correction under prior shift:
- odds_s = p_s / (1 - p_s)
- odds_pop = odds_s × [(π / (1 - π)) / (π_s / (1 - π_s))]
- p_pop = odds_pop / (1 + odds_pop)
Provide the numeric p_pop and explain when class weights vs. probability recalibration are sufficient or when you need both.
d) Threshold via cost-benefit: Outreach costs $2 per account. If an account would default and you contact them, there is a 30% chance the intervention averts an average $150 loss. Choose the action rule and compute the breakeven probability threshold p* for contacting, then decide whether to contact the account from part (c) using p_pop.
e) Validation: Propose a time-based cross-validation scheme and a backtest showing stability under covariate and prior shift across regions and macro regimes. Include how you would monitor calibration and drift post-deployment and re-tune sampling if the true default rate changes from 2% to 1%.
f) Fairness and operations: Name two fairness checks (e.g., equal opportunity across age groups where legally permitted) and one operational guardrail to prevent over-throttling credit limits due to model uncertainty. Explain how sampling choices can bias these checks if not corrected.
Overview: This question evaluates a data scientist's competencies in imbalanced binary classification, temporal label definition and leakage detection, sampling and calibration under prior-probability shift, metric and operating-point selection, cost-sensitive decision rules, time-based validation and monitoring, and fairness and operational guardrails.
Community answers
Answer by sushantswarup00
import pandas as pd
import numpy as np
HORIZON_DAYS = 90
EMBARGO_DAYS = 90
def month_end(ts: pd.Timestamp) -> pd.Timestamp:
ts = pd.Timestamp(ts)
return (ts + pd.offsets.MonthEnd(0)).normalize()
def build_account_month_index(accounts: pd.Series, start_month: str, end_month: str) -> pd.DataFrame:
months = pd.date_range(month_end(start_month), month_end(end_month), freq="M")
idx = pd.MultiIndex.from_product([accounts.unique(), months], names=["account_id", "asof_month"])
return idx.to_frame(index=False)
def asof_join_snapshots(account_month: pd.DataFrame, snapshots: pd.DataFrame) -> pd.DataFrame:
"""
snapshots: account_id, effective_time, features...
"""
s = snapshots.sort_values(["account_id", "effective_time"]).copy()
am = account_month.sort_values(["account_id", "asof_month"]).copy()
out = pd.merge_asof(
am,
s,
left_on="asof_month",
right_on="effective_time",
by="account_id",
direction="backward",
allow_exact_matches=True,
)
out["max_source_time_snapshots"] = out["effective_time"]
return out
def rolling_event_features(account_month: pd.DataFrame, events: pd.DataFrame) -> pd.DataFrame:
"""
events: account_id, event_time, amount, type, dpd, ...
Create features using ONLY event_time <= asof_month.
"""
e = events.sort_values(["account_id", "event_time"]).copy()
am = account_month.sort_values(["account_id", "asof_month"]).copy()
# As-of merge to get all events up to cutoff, then aggregate in windows
# Efficient approach in production: pre-aggregate by day/month in SQL; here is a