Evaluate and monitor a credit risk model

Read the full interview experience this question came from →

Quick Overview

This question evaluates competency in designing, evaluating, and monitoring cost-sensitive consumer credit probability-of-default (PD) models, emphasizing calibration, rank ordering, threshold selection, backtesting, champion–challenger experimentation, and production monitoring under regulatory stability and explainability constraints.

Evaluate and monitor a credit risk model

Company: Capital One

Role: Data Scientist

Category: Machine Learning

Difficulty: hard

Interview Round: Technical Screen

You are shipping a credit-risk model (probability of default within 12 months) for a consumer lender with a 1% default rate. Misclassification costs: a false negative (approve a defaulter) costs $1,200; a false positive (decline a good applicant) costs $60. The regulator expects stability and explainability over pure accuracy. Pick the three most important evaluation priorities for this context and justify them. Then design an end-to-end plan covering: (1) offline evaluation (temporal cross-validation, class imbalance handling, choice of metrics such as AUC-PR, expected cost, calibration error, KS, and reason for each); (2) threshold selection to minimize expected cost subject to max 4% decline rate; (3) backtesting on out-of-time cohorts and stress periods; (4) a champion–challenger live test with guardrails; (5) production monitoring (data/label drift, calibration monitoring, PSI thresholds, stability by segment, and an alert/runbook). Be specific about calculations, acceptance criteria, and what you would do if calibration drifts while rank ordering remains stable.

Overview: This question evaluates competency in designing, evaluating, and monitoring cost-sensitive consumer credit probability-of-default (PD) models, emphasizing calibration, rank ordering, threshold selection, backtesting, champion–challenger experimentation, and production monitoring under regulatory stability and explainability constraints.

Read the full Capital One Data Scientist interview experience this question came from

|Home/Machine Learning/Capital One
Capital One logo
Capital One
Oct 13, 2025
hardData ScientistTechnical ScreenMachine Learning
17
0

Credit-Risk PD Model: Evaluation Priorities and End-to-End Plan

Context: You are deploying a consumer credit probability-of-default (PD) model for 12-month default with a 1% base default rate. Misclassification costs are asymmetric: approving a future defaulter (false negative) costs 1,200;decliningafuturenon−defaulter(falsepositive)costs1,200; declining a future non-defaulter (false positive) costs 60. The regulator emphasizes stability and explainability over pure accuracy. You must choose the three most important evaluation priorities and design a complete evaluation and deployment plan.

Assumptions (minimal, explicit):

  • The model outputs well-ordered PD scores p ∈ [0, 1]. Higher p indicates higher risk.
  • Decision policy: decline if p ≥ threshold t; approve otherwise.
  • Decline rate is the share of total applicants declined at decision time.

Tasks

  1. Pick the three most important evaluation priorities for this context and justify them.
  2. Design an end-to-end plan covering:
    1. Offline evaluation: temporal cross-validation design; class imbalance handling; choice of metrics (e.g., AUC-PR, expected cost, calibration error, KS) and justification for each.
    2. Threshold selection to minimize expected cost subject to a maximum 4% decline rate.
    3. Backtesting on out-of-time cohorts and stress periods.
    4. A champion–challenger live test with guardrails.
    5. Production monitoring: data/label drift, calibration monitoring, PSI thresholds, stability by segment, and an alert/runbook.

Be specific about calculations, acceptance criteria, and what you would do if calibration drifts while rank ordering remains stable.

Loading comments...

Browse More Questions

More Machine Learning•More Capital One•More Data Scientist•Capital One Data Scientist•Capital One Machine Learning•Data Scientist Machine Learning