Present a project to non-technical leaders

# Presentation: Increasing Profitable Originations with a Next-Gen Credit Risk & Pricing Model Note: The following is a concise, end-to-end narrative you can adapt to your own project. It targets a mixed audience (PM, engineering, finance) and fits in ~10–15 minutes plus 5 minutes Q&A. ## 1) Business Problem Framing - Context: Our personal loan business was constrained by a traditional scorecard and flat pricing. We had high decline rates and left profitable customers on the table. Finance wanted more bookings without increasing charge-offs; PM wanted a better customer experience and faster decisions; Engineering required low-latency inference. - Goal: Increase profitable loan originations while holding risk (charge-off rate) flat and maintaining compliance. - Decision levers: - Approvals/declines based on predicted risk. - Risk-based pricing to align APR with expected loss. ## 2) Baseline and Success Metrics - Baseline performance (last 12 months, illustrative numbers): - Model: Legacy logistic scorecard - AUC: 0.72; KS: 32; Brier score: 0.188 - Approval rate: 41% - Bad/charge-off rate (12-month): 7.8% - Avg APR: 23% - Profit per booked loan: $185 (after CAC and servicing) - Success metrics (set pre-project): - Primary: Incremental risk-adjusted profit ≥ $5M/year at steady state - Secondary: - Approval rate +3–5% at equal or lower expected loss per dollar booked - AUC ≥ 0.78 and well-calibrated PDs (calibration slope 0.9–1.1) - p95 latency < 150 ms; uptime ≥ 99.9% - Fairness guardrail: disparate impact ratio (DIR) ≥ 0.80 with no significant widening of TPR/FPR gaps - Stability: monthly PSI < 0.25; auto-retrain ≤ quarterly ## 3) Experiment/Design Choices - Modeling approach: - Two-stage framework: 1) PD (probability of default/charge-off) via gradient-boosted trees (XGBoost) with monotonic constraints on key risk features 2) LGD (loss given default) via regularized regression with coarse bins for interpretability - Expected Loss (EL): EL = PD × LGD × EAD - Profit model per application: Profit = Interest Income − Funding Cost − EL − Servicing Cost − CAC - Feature engineering: - Bureau and internal features: utilization, delinquencies, inquiries, debt-to-income, payment histories, income verification signals - Leakage controls: exclude post-application signals; align to application timestamp - Fairness hygiene: exclude protected-class proxies (e.g., ZIP granularities), cap influence of unstable features; use monotonicity for consistent risk direction - Validation strategy: - Time-based splits (train: older vintages, valid: more recent) to mimic deployment - K-fold with grouped time blocks; backtest on 24 months - Calibration via isotonic regression - Deployment architecture: - Shadow mode for 4 weeks → Champion/Challenger traffic at 10% → staged ramps to 50% → 100% - Online scoring service: feature store + model server; p95 latency target < 150 ms ## 4) Key Trade-offs - Interpretability vs performance: XGBoost (+SHAP explanations, monotone constraints) instead of a pure scorecard; balanced to meet model risk governance - Latency vs complexity: pruned tree depth, 60 key features; bureau features cached to hit latency SLOs - Exploration vs risk: Staged exposure with guardrails (see below) rather than full A/B on approvals to respect risk/compliance - Pricing precision vs customer experience: Smoothed price curves and caps to avoid customer confusion and APR oscillation ## 5) Risks and Mitigations - Bias/fairness: - Proxy detection and removal; monitored DIR and equalized-odds gaps using accepted proxy methodologies - Adverse action reason codes mapped to stable, human-understandable features - Data leakage and drift: - Event-time feature construction; holdout by vintage; monthly PSI, calibration drift, KS change monitoring - Auto-retrain policy with human review if PSI > 0.25 or calibration slope deviates > 0.1 - Regulatory/compliance: - Model documentation, challenger validation, and governance approvals - Pricing floors/ceilings; reason codes; reject-infer analysis documented - Operational risk: - Blue/green deploy, kill switch to champion - Backfill logging, idempotent decisioning, deterministic versioned models ## 6) Offline Performance (Backtest) - Discrimination: - AUC: 0.80 (↑ from 0.72); KS: 45 (↑ from 32) - Calibration: - Brier score: 0.160 (↓ 15%); calibration slope ~0.98 - Policy simulation (12 months of historical apps): - At equal expected loss per dollar booked, approval rate +5.2% - Risk-based pricing optimization increases average unit profit by $38 per booked loan - Expected annual uplift: $8.7M (assumes 230k apps/year, 41%→43.1% bookings, +$38 unit profit, steady mix) Formula example: - EL_i = PD_i × LGD_i × EAD_i - Profit_i = APR_i × EAD_i − COF_i − EL_i − Servicing_i − CAC_i - Maximize sum_i Profit_i subject to guardrails (fairness DIR ≥ 0.8, APR caps, latency SLO) ## 7) Online Experiment (Champion/Challenger) - Pre-test: 4-week shadow run to validate latency, stability, and reason-code coverage - Test design: 10% traffic to challenger for 8 weeks; APR within ±200 bps of champion; no expansion into high-risk bands beyond pre-approved limits - Guardrails: - Stop-loss if EL per dollar booked worsens by >15 bps - Approval increase cap in top-risk deciles - Weekly fairness checks (DIR and TPR gaps) with auto-pause triggers - Results (stat-sig at 95%): - Bookings: +3.9% uplift on treated traffic - EL per dollar booked: no significant change (+1 bp, within guardrail) - Avg unit profit: +$29/loan - Latency p95: 95 ms; uptime: 99.98% - Fairness: DIR improved from 0.84 → 0.86; TPR/FPR gaps unchanged within ±0.5 pp - Annualized uplift (after full rollout, steady state): ~$7.1M with observed online unit economics and volume ## 8) Concrete Business Impact - Financial: - Incremental annual profit realized YTD: $5.3M (9 months post-rollout) - CAC per booked loan: −$12 via better targeting - Charge-off rate: flat at 7.8% (within ±0.2 pp) - Customer and ops: - Instant decisions share: 76% → 88% - Manual reviews: −18% - Fewer adverse action disputes due to clearer reason codes - ROI: - Project cost (data infra, licenses, headcount): ~$1.7M - First-year profit uplift (run-rate): ~$7.1M - ROI ≈ 4.2×; payback < 4 months ## 9) What I Owned - Led problem framing with PM/finance; defined success metrics and guardrails - Designed PD/LGD models, feature pipeline, calibration, and pricing optimizer - Co-led experiment design; built fairness and drift monitors; wrote model documentation for governance ## 10) Lessons and Next Steps - Calibrated PDs and clear reason codes smoothed governance and customer support - Next: add income verification signals, improve thin-file treatment, and explore uplift modeling for retention --- ## 5-Minute Q&A Plan: Tough Questions and Responses Q1 (Finance): Your ROI assumes volume scales linearly and macro stays stable. What if unemployment rises 150 bps and charge-offs spike? Is the $7.1M uplift still credible? - Concise, data-backed response: - We ran stress tests using macro-linked PD shifts (ΔPD ≈ +12% for +150 bps unemployment, based on our vintage analysis). Under stress, EL per dollar booked rises by ~10–12 bps. - Sensitivity table: - Base: +$7.1M uplift - Moderate stress: +$4.6M - Severe stress: +$2.9M - Why still positive: price optimizer raises APR within caps for higher PD cohorts and trims approvals at the margin to hold EL in check. Guardrail triggers auto-tighten cutoffs if EL drifts >15 bps. - Governance: monthly macro overlay review with finance; if stress persists, we switch to conservative policy preset (pre-approved by risk). Q2 (PM/Risk): How do you ensure the model is fair and doesn’t worsen outcomes for protected groups, especially given we don’t directly observe protected class? - Concise, data-backed response: - We use accepted proxy methods (e.g., BISG) to estimate group membership for monitoring only, not for decisioning. - Metrics tracked: disparate impact ratio (DIR), TPR/FPR gaps, and approval/profit parity by decile. Pre-launch baseline DIR = 0.84; challenger improved to 0.86 with no significant change in TPR/FPR gaps. - Design choices that help fairness: removed high-risk proxies, monotonic constraints, calibrated PDs, and smooth price curves; fairness guardrails in optimizer; weekly fairness monitors with auto-pause. - Compliance: generated specific, stable adverse action reasons; conducted reject-infer documentation and model risk review before rollout. --- ## Pitfalls, Edge Cases, and Guardrails - Pitfalls: data leakage (post-application verifications), volatile derived ratios, selection bias from prior declines, overfitting recent vintages - Guardrails: - Time-based validation; feature freeze window (e.g., exclude signals within 7 days post-app) - Reject inference sensitivity checks; regularization and monotonicity - Online stop-loss, APR caps, fairness DI ≥ 0.80; kill switch to champion - Drift monitors (PSI, calibration, AUC), auto-retrain with human-in-the-loop ## Mini-Example (for Clarity in the Room) - Applicant A: PD=6%, LGD=60%, EAD=$5,000 → EL=$180 - Pricing: APR yields $650 interest over loan life; COF=$250; Servicing=$60; CAC=$100 - Profit ≈ 650 − 250 − 180 − 60 − 100 = $60 - Optimizer accepts A with APR near cap if fairness and guardrails satisfied; legacy policy would have declined A, missing $60 profit. ## Close - We achieved higher profitable growth at constant risk through better discrimination, calibration, and controlled experimentation with strong governance. The rollout plan, monitoring, and guardrails ensure resilience across market regimes.