Evaluate and monitor a credit risk model

Q: Evaluate and monitor a credit risk model

This question evaluates competency in designing, evaluating, and monitoring cost-sensitive consumer credit probability-of-default (PD) models, emphasizing calibration, rank ordering, threshold selection, backtesting, champion–challenger experimentation, and production monitoring under regulatory stability and explainability constraints.

Q: How do I approach Machine Learning interview questions?

Machine Learning questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master machine learning interviews.

Q: What difficulty level is this interview question?

This is a hard difficulty Machine Learning question, commonly asked during Technical Screen rounds at Capital One.

Q: What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Capital One during technical interviews.

Question

Credit-Risk PD Model: Evaluation Priorities and End-to-End Plan

Context: You are deploying a consumer credit probability-of-default (PD) model for 12-month default with a 1% base default rate. Misclassification costs are asymmetric: approving a future defaulter (false negative) costs $1,200; declining a future non-defaulter (false positive) costs$ 60. The regulator emphasizes stability and explainability over pure accuracy. You must choose the three most important evaluation priorities and design a complete evaluation and deployment plan.

Assumptions (minimal, explicit):

The model outputs well-ordered PD scores p ∈ [0, 1]. Higher p indicates higher risk.
Decision policy: decline if p ≥ threshold t; approve otherwise.
Decline rate is the share of total applicants declined at decision time.

Tasks

Pick the three most important evaluation priorities for this context and justify them.
Design an end-to-end plan covering:
1. Offline evaluation: temporal cross-validation design; class imbalance handling; choice of metrics (e.g., AUC-PR, expected cost, calibration error, KS) and justification for each.
2. Threshold selection to minimize expected cost subject to a maximum 4% decline rate.
3. Backtesting on out-of-time cohorts and stress periods.
4. A champion–challenger live test with guardrails.
5. Production monitoring: data/label drift, calibration monitoring, PSI thresholds, stability by segment, and an alert/runbook.

Be specific about calculations, acceptance criteria, and what you would do if calibration drifts while rank ordering remains stable.

Evaluate and monitor a credit risk model

Quick Overview

Evaluate and monitor a credit risk model

Credit-Risk PD Model: Evaluation Priorities and End-to-End Plan

Tasks

Write your answer