Evaluate and monitor a credit risk model
Overview
This question evaluates competency in designing, evaluating, and monitoring cost-sensitive consumer credit probability-of-default (PD) models, emphasizing calibration, rank ordering, threshold selection, backtesting, champion–challenger experimentation, and production monitoring under regulatory stability and explainability constraints.
Credit-Risk PD Model: Evaluation Priorities and End-to-End Plan
Context: You are deploying a consumer credit probability-of-default (PD) model for 12-month default with a 1% base default rate. Misclassification costs are asymmetric: approving a future defaulter (false negative) costs 60. The regulator emphasizes stability and explainability over pure accuracy. You must choose the three most important evaluation priorities and design a complete evaluation and deployment plan.
Assumptions (minimal, explicit):
- The model outputs well-ordered PD scores p ∈ [0, 1]. Higher p indicates higher risk.
- Decision policy: decline if p ≥ threshold t; approve otherwise.
- Decline rate is the share of total applicants declined at decision time.
Tasks
- Pick the three most important evaluation priorities for this context and justify them.
-
Design an end-to-end plan covering:
- Offline evaluation: temporal cross-validation design; class imbalance handling; choice of metrics (e.g., AUC-PR, expected cost, calibration error, KS) and justification for each.
- Threshold selection to minimize expected cost subject to a maximum 4% decline rate.
- Backtesting on out-of-time cohorts and stress periods.
- A champion–challenger live test with guardrails.
- Production monitoring: data/label drift, calibration monitoring, PSI thresholds, stability by segment, and an alert/runbook.
Be specific about calculations, acceptance criteria, and what you would do if calibration drifts while rank ordering remains stable.
Solution
Login required