Diagnose Sudden Drop in Credit-Card Approval Rate
Overview
This question evaluates a data scientist's competency in root-cause analysis for production metric anomalies, testing diagnostic reasoning across decision logs, model and rule changes, feature pipelines, traffic-mix shifts, experiment assignments, and external dependencies.
Fraud Risk: Sudden Drop in Credit Approval Rate
Context
You are the on-call Data Scientist supporting the risk/underwriting system. Historically, the daily approval rate has been stable. Yesterday, it dropped sharply in a single day.
Assume you have access to decision logs, rule engine configs, model registry, feature store, experiment platform, observability/monitoring dashboards, and vendor status pages.
Approval rate = approvals / submitted applications, measured on decision-event timestamps. (If your org uses a different definition, state and use it.)
Task
How would you diagnose the root cause? Provide:
- A step-by-step investigative plan to localize and explain the drop.
- The specific data you would pull.
- A set of concrete hypotheses to test (and how to test them).
Hints
Consider:
- Traffic mix shifts (channel, merchant/partner, geo, device, new vs. returning).
- Rule or model changes (new version, thresholds, reason-code distribution).
- Data/feature pipeline issues (null spikes, late features, schema changes).
- Seasonality/calendar anomalies and promotions.
- Experiment/AB-test effects (assignments, guardrails, SRM).
- External dependencies (KYC/identity, bureau pulls, device fingerprinting, payment networks).
- Infrastructure/latency/timeouts and fail-closed policies.
Solution
Login required