Explain tackling ambiguity and defending a decision

# Example, structured answer (STAR+R: Situation, Task, Action, Result, Reflection) Situation and Task - Date/context: On March 6, 2023, Marketing planned a paid acquisition push starting March 20. My team had 5 business days to recommend whether to relax a near-cutoff approval policy to hit a quarterly booking target without breaching risk guardrails. - Ambiguity/incomplete data: A new bank-transaction aggregator covered only ~60% of applicants; income labels were missing for the remaining 40%. Performance data for these new features had just 3 months of outcome maturity (right-censored defaults), and historical backtests risked sample bias. - Decision to make: Should we expand approvals for applicants within a tight score band near the current cutoff, and if so, under what pricing and guardrails? - Constraints: 5-day deadline; engineering could only support one policy change; regulatory/compliance required documented rationale; portfolio loss-rate guardrail ≤ 7.0% 12-month 90+ DPD; compute budget limited to existing infra. - Success metrics (defined up front): - Primary: Incremental expected NPV per booked loan ≥ +$5 vs. status quo, while meeting portfolio guardrails. - Secondary: Approval rate lift ≥ +5% relative (same traffic mix), with no more than +0.3 pp absolute 3-month 60+ DPD early delinquency. Action: Options Considered and Analyses - Options considered: 1) Status quo (no change). 2) Global APR +20–30 bps to absorb potential risk; keep cutoff unchanged. 3) Expand approvals by 30 bps of score around the cutoff using new bank features; keep pricing. 4) Hybrid: Expand approvals by 20 bps around the cutoff plus +30 bps APR only for the expanded band; stage-gated rollout with guardrails. - Key analyses I ran (lead analyst; I wrote the analysis doc, SQL, and Python notebooks): - Decision definition/formula: Expected NPV per loan NPV = Expected Revenue − Expected Loss − CAC − Funding Cost where Expected Loss = EAD × PD × LGD. - Missingness: Multiple imputation for missing income features (m=5) and a conservative worst-case bound (treat missing as 10th percentile quality) to produce a range of outcomes. I reported both midpoint and worst-case NPV. - Off-policy backtest: Used out-of-time windows (Oct–Dec 2022) to simulate expanding the cutoff; guarded against leakage by excluding any post-application outcomes. Calibrated PD using Platt scaling on a holdout. - Sensitivity: Tornado chart across PD shift (+0.5 to +1.5 pp), LGD ±5 pp, APR elasticity (−0.1 to −0.3 rel. demand per +100 bps APR). Goal: ensure decision robust within realistic ranges. - Experiment plan: 10% traffic A/B to the expanded policy, pre-registered guardrails and stop-loss: halt if early 60+ DPD exceeds control by ≥0.5 pp or if observed gross margin/loan falls below −$2 vs control at 95% sequential confidence. - Findings (point estimates; 95% ranges with worst-case in parentheses): - Option 1: 0% approval lift; NPV/loan baseline $0; Loss unchanged. - Option 2: +1% approvals; +$2 NPV/loan; risk neutral; elasticity cost offset benefit. - Option 3: +7% approvals; +$8 ($+2 worst-case) NPV/loan; +0.2 pp early delinquency. - Option 4: +6% approvals; +$9 ($+4 worst-case) NPV/loan; +0.1 pp early delinquency; operationally feasible. - Decision under uncertainty: Chose Option 4 (hybrid, stage-gated). Rationale: best risk-adjusted NPV with conservative protection from pricing, robust to missingness stress tests, and experiment guardrails to cap downside. Stakeholder Alignment and Trade-offs - Stakeholders: Growth wanted ≥+10% approvals; Risk preferred status quo; Product needed one policy change only; Compliance required clear documentation. - Alignment tactics I led: - 2-page pre-read framing the decision, constraints, metrics, and ranges (point/worst-case). I separated facts from assumptions and tagged each assumption with an owner. - RACI clarified: I was DRI; Risk had veto on guardrails; Product owned rollout plan; Compliance reviewed the doc. - Made the stop-loss explicit and automated (dashboard + pager): if guardrail breached, rollback in <24 hours. - Explicit trade-offs rejected: - Rejected a broader 50 bps expansion that modeled +12% approvals but pushed worst-case early delinquency +0.6 pp (beyond guardrail). - Rejected a global +50 bps APR that harmed funnel conversion (−3%) with marginal NPV benefit and potential adverse selection. Result (initial) - Launch: March 20, 2023 to 10% traffic. - First 14 days: - Approval lift: +6.3% (CI: +4.7 to +7.9%). - NPV/loan: +$8.60 vs control (bootstrap 95% CI: +$3.10 to +$13.70). - Early 60+ DPD: +0.12 pp vs control (below 0.5 pp guardrail). - We stage-gated to 25% on April 5, 2023, with the same guardrails. Six Weeks Later: Suboptimal Outcome and Response - On May 1, 2023 (6 weeks post-launch), cohort performance indicated the expanded band had a higher-than-expected default hazard as income verification coverage dropped (aggregator outage in late March). Observed NPV/loan for the expanded band trended to −$4 vs control; early delinquency difference widened to +0.48 pp and projected to breach the 0.5 pp guardrail within a week. - Immediate actions (within 48 hours; I led): 1) Triggered rollback to status quo for the expanded band per pre-registered stop-loss. 2) Published a red/amber/green incident update: cause hypothesis (coverage shock changed the missingness mechanism from MAR to NMAR), impact (−$4 NPV/loan on 22% of booked loans in cohort), and remediation plan. 3) Ran a root cause analysis with Risk and Data Eng: found drift in income coverage from 60% to 42% for certain geos; our imputation prior was mis-specified under NMAR. - Communication: - Same-day Slack summary to exec channel with the decision, data, and rollback; 1-pager emailed to stakeholders with charts; took responsibility for the assumption miss and highlighted what worked (guardrails contained downside). - Set expectations: pause on expansion; target new recommendation by May 19 after fixes. - Prevention and improved plan (implemented by May 17, 2023): - Model/process changes: - Added a missingness-aware feature: explicit binary indicators plus a separate scorecard calibrated on the “missing income” segment. - Introduced a conservative prior for PD when coverage <50% (Bayesian shrinkage toward higher PD) and a dynamic pricing add-on for missing segments (+15–25 bps). - Added synthetic missingness stress tests in pre-launch: randomly mask 30–60% of income features and require NPV ≥ $0 under NMAR assumptions before shipping. - Monitoring/guardrails: - Live coverage monitor with alerting if coverage drops >5 pp week-over-week. - Experiment design updated to sequential gating by segment (present vs missing income) with independent guardrails. - Governance: - Updated the decision template to require an “assumptions registry” with a tripwire metric for each assumption and an owner; added a mandatory pre-mortem section. - Outcome after relaunch (June 2023): - Segment-aware policy delivered +4.1% approvals; +$6.10 NPV/loan overall; early delinquency +0.08 pp vs control; no guardrail breaches. Why this demonstrates judgment under ambiguity - Clear decision framing with explicit constraints and success metrics. - Multiple options evaluated with sensitivity ranges and worst-case bounds, not just point estimates. - Pre-defined guardrails and stop-loss limited downside when reality diverged from assumptions. - Transparent communication, rapid rollback, root-cause fixes, and process changes to prevent recurrence. Reusable template you can adapt - Define the decision, constraints, and one primary success metric with guardrails. - Generate 2–4 materially different options; quantify point and worst-case outcomes. - Plan an experiment with automatic stop-loss guardrails. - Align stakeholders with a concise pre-read and a RACI; explicitly list rejected trade-offs. - If suboptimal: roll back fast, own the miss, publish a root cause, and institutionalize fixes (stress tests, monitoring, assumptions registry).