Capital One Model Risk Governance And SR 11-7

What's being tested

Interviewers are probing your practical grasp of model risk management under regulatory expectations (SR 11-7) as a working Data Scientist: how you design, evaluate, document, and monitor models so they are auditable, reproducible, and materially safe for business use. They want concrete, actionable answers about validation tests you would run, what operational metrics you would monitor, how you would communicate limitations, and how you would prepare artifacts for an independent validator — not how to implement infra or run Kafka pipelines. Capital One cares because model failures produce financial, compliance, and reputational loss; they expect Data Scientists to limit model risk through defensible analytics, proper evaluation, and clear documentation.

Core knowledge

• SR 11-7: Federal Reserve guidance requiring institutions to maintain robust model risk management: model inventory, materiality assessment, independent validation, and lifecycle controls; expect reproducible artifacts and documentation.

• Materiality: classify models as high/medium/low impact based on financial exposure, regulatory sensitivity, or consumer impact; material models get more frequent validation and stricter change controls.

• Model documentation: include model purpose, conceptual design, data sources and definitions, feature engineering, training/validation/test splits, hyperparameters, performance metrics, limitations, and deployment checklist.

• Independent validation: should be independent functionally; as a Data Scientist you must provide reproducible code/notebooks, fixed seeds, data snapshots, and validation test plans so validators can replicate results.

• Performance metrics: choose metrics aligned to business use — classification: AUC, KS, precision@k, lift, calibration (Brier score, calibration plot); regression: RMSE, MAE, and economic loss functions if available.

• Calibration testing: for probability models use reliability diagrams, Brier score ($\frac{1}{N}\sum (p_i - y_i)^2$), and Hosmer–Lemeshow for grouped expected vs observed; miscalibration is a model-risk flag.

• Stability & drift: monitor Population Stability Index (PSI) where PSI = $\sum (actual\% - expected\%) \ln(\frac{actual\%}{expected\%})$; interpret: PSI <0.1 stable, 0.1–0.25 moderate, >0.25 significant.

• Backtesting & lift: perform backtests comparing predicted vs realized outcomes over time (e.g., PD models: expected default rate vs observed defaults) and compute cumulative gains or lift curves to detect degradation.

• Statistical significance & sample sizes: use confidence intervals and hypothesis tests; expect validation samples with sufficient events — rough rule: >200–500 events for preliminary tests, >1000 for robust PD calibration.

• Overfitting checks: compare training/validation/test performance, use cross-validation, and report optimism via bootstrap or nested CV; document hyperparameter search space and selection criterion.

• Explainability & reason codes: provide local/global explainability artifacts (SHAP, permutation importance) and list key feature assumptions so validators can assess conceptual soundness and fairness.

• Change control & versioning: maintain model version, data snapshot references, and a clear change-log describing retrains, feature changes, and threshold adjustments; note rollback criteria and post-deployment tests.

• Monitoring thresholds & alerts: define concrete breach triggers (e.g., AUC drop >0.02, PSI >0.25, calibration shift p<0.05) and planned remediation (retrain, feature investigation, freeze decisions).

Worked example — "How would you prepare a model for SR 11-7 validation?"

Frame quickly: clarify the model's business use, materiality, input data sources, and expected decision cadence (online vs batch). Outline deliverables: (1) reproducible training artifacts (code, fixed seeds, environment spec), (2) model documentation (purpose, assumptions, variables), (3) performance and robustness evidence (metrics, cross-val, calibration), and (4) monitoring plan and stress tests. Skeleton of answer: (A) Describe dataset snapshots and how splits mimic production timelines; (B) show core evaluation — AUC, calibration plots, Brier score, plus backtest comparing predicted vs actual outcomes; (C) robustness tests — adversarial cases, PSI, feature ablation, and sensitivity to labeling delay; (D) governance — model ID, materiality justification, and rollback plan. Flag a tradeoff: investing in more complex robustness tests vs validator timelines — propose a prioritized test plan (must-haves first: reproducibility, calibration, stability) and deferred checks. Close with next steps: "If I had more time, I'd run nested cross-validation for hyperparameter uncertainty, produce SHAP explanation suites for top segments, and simulate scenario stress tests for extreme macro conditions."

A second angle — "Design monitoring and alerting for a deployed model under SR 11-7"

Same core concept (manage model risk) but framed toward ongoing surveillance. Start by defining key observability signals: predictive performance (AUC, precision@k), calibration drift, input feature distributions (PSI), and business KPIs (conversion, loss rates). Propose concrete thresholds and alert channels, and an escalation playbook mapping breach severity to actions (retrain, investigate data pipeline, pause automated decisions). Emphasize quick diagnostics: an initial triage dashboard showing top feature distribution changes, cohort performance, and recent data-label delays. Highlight a communication plan for validators and business stakeholders: weekly health summaries for low-severity, immediate reports for material breaches. This shows transfer of the validation mindset into operational monitoring and governance.

Common pitfalls

Pitfall: Overfocusing on a single metric.
Many candidates present only AUC; auditors expect a suite: discrimination, calibration, stability, and business-aligned metrics (e.g., monetary loss).

Pitfall: Treating validators as auditors to "pass".
Deliver transparent failures and assumptions; hiding post-hoc fixes or ad-hoc thresholds undermines trust and violates SR 11-7 intent.

Pitfall: Ignoring label lag and selection bias.
Reporting apparent model degradation without checking label availability, censoring, or policy changes will misattribute data problems to the model.

Connections

This topic naturally pivots to fairness & bias testing, explainability (SHAP, LIME), and model deployment/monitoring practices; interviewers may also ask about linking monitoring alerts to business operations or to MLE/DevOps for remediation.

Further reading

• SR 11-7: Guidance on Model Risk Management (Federal Reserve) — primary regulatory text and expectations.

• [Elements of Statistical Learning — Hastie, Tibshirani, Friedman] — deep reference on validation, regularization, and model assessment.