Lead structured response to accuracy incident
Company: CVS Health
Role: Data Scientist
Category: Behavioral & Leadership
Difficulty: hard
Interview Round: Technical Screen
You are the on-call lead for a payment-accuracy team. At 09:00 you learn that, since 00:00 today, the share of incorrect payments for users aged 18–24 in CA rose from a historical 2.0% to 3.1% across ~120k transactions; finance is escalating and wants a mitigation within 24 hours. You have access to feature logs, model scores, recent code/deployment diffs, and a sampling tool to pull raw cases for audit.
Describe, in a structured, time-boxed plan, exactly how you would: 1) Triage and quantify impact (owners, metrics, and acceptance criteria for 'all clear'), 2) Form testable hypotheses and design the minimum experiments/queries to isolate root cause (data slices, holdouts, backfills), 3) Decide on a safe mitigation (e.g., threshold change, feature hotfix, rule fallback), including rollback triggers and predicted side effects (precision/recall trade-offs), 4) Communicate status to execs and partner teams at specific times with concrete artifacts (dashboards/runbooks), and 5) Prevent recurrence (observability, guardrails, pre-deploy checks). Be explicit about risks you will accept vs. avoid and how you’d measure success by end of day.
Quick Answer: This question evaluates incident leadership, operational decision-making, and data-driven troubleshooting skills for a data scientist, focusing on triage, hypothesis formation, mitigation choice, cross-functional communication, and recurrence prevention.
Solution
# Assumptions and Notation
- CA = California (confirm immediately; if Canada, repeat all slices using country_region).
- Score s ∈ [0,1] is the model-estimated probability a payment is incorrect.
- Action: if s ≥ t (threshold), route to review/decline; if s < t, auto-approve. Lowering t means catching more potential incorrect cases (lower incorrect rate, higher friction).
- Observed cohort volume since 00:00: n ≈ 120,000 for CA age 18–24 (validate; if 120k is total site volume, re-compute for the cohort).
- Incorrect rate (IR) = incorrect_count / total_count.
# 0. Timeline Overview (Day-of Plan)
- 09:00 Detect + page. Create an incident channel and ticket. Assign roles.
- 09:05–09:30 Rapid triage: confirm anomaly, scope who/what/when. Freeze risky deploys. Build live dashboard by cohort.
- 09:30–10:30 Root-cause probes: config/threshold, model version, feature missingness/drift, label pipeline, traffic mix. Pull 100–200 raw cases for audit.
- 10:30–11:00 Decision checkpoint 1: if clear cause found (config/model), roll back. If unknown, ship gated mitigation (tighten threshold for CA 18–24 on 25% canary) with guardrails.
- 11:00–14:00 Continue experiments (shadow scoring, backfills, merchant/device slices). Adjust mitigation as evidence arrives.
- 14:00 Decision checkpoint 2: expand/roll back mitigation based on metrics.
- 16:00 Draft postmortem outline; finalize prevention actions and owners.
- 17:30 End-of-day (EOD) report: results, metrics, next steps.
# 1) Triage and Quantify Impact
## Owners (assign in incident ticket)
- Incident commander (IC): You (on-call DS). Decision maker/timekeeper.
- ML engineer: model configs, thresholds, rollbacks.
- Data engineer: feature pipeline and data quality.
- Analyst/BI: dashboards, SQL queries, CIs.
- Risk Ops: manual audit sampling, case categorization.
- Finance liaison: impact translation ($, volume), stakeholder updates.
## Metrics to compute now
- IR_18_24_CA(t): 15-min rolling incorrect rate for CA age 18–24.
- Baselines: last 4 weeks, same day-of-week/time-of-day; mean and bands.
- Volume: transactions per 15 min.
- Confusion components (proxy in near-real time):
- Block/review rate (BR): share routed to review/decline.
- Auto-approve rate (AR): share approved immediately.
- If labels are fast: False approvals (incorrect among auto-approvals), False reviews (correct among reviewed).
- Feature health: top-20 features’ null rate, staleness, distribution shift.
- Model health: score distribution per cohort; KS statistic vs baseline; PSI for features.
- Label pipeline health: label latency distribution and completeness.
## Quick significance check (example)
- Observed p̂ = 3.1% with n = 120,000. 95% CI ≈ p̂ ± 1.96 * sqrt[p̂(1−p̂)/n] ≈ 0.031 ± 0.001 → (3.0%, 3.2%). Historical baseline = 2.0% → highly significant.
- Counts: expected incorrect at 2.0% = 2,400; observed = 3,720; delta ≈ +1,320.
## Acceptance criteria for "all clear"
Any one of:
- IR_18_24_CA returns to ≤ 2.3% (≤ +15% rel vs 2.0% baseline) for 3 consecutive 30-min windows with stable volume; OR
- IR_18_24_CA within baseline 95% control limits and no active anomaly flags on score/feature drift; AND
- No new anomalies introduced in adjacent cohorts (other ages in CA, 18–24 in other states); AND
- Manual review queue remains < 80% capacity for 95th percentile SLA.
# 2) Hypotheses and Minimum Experiments/Queries
Prioritize checks that are fast, discriminative, and reversible.
## H1. Threshold/config drift or bad rollout
- Check: Compare current threshold t, rule weights, and model_version in prod vs last stable. Diff config files, env vars, feature flags, and change logs since 00:00.
- Query: SELECT distinct model_version, threshold, count(*) FROM events WHERE ts >= today AND cohort = 'CA_18_24' GROUP BY 1,2;
- Action if true: Immediate rollback to last known-good config/model.
## H2. Feature pipeline issue (missingness/staleness/time-zone)
- Check: For top features by importance, compare null rate, timestamp skew, PSI.
- Query (sketch):
- Feature nulls: SELECT feature_name, avg(is_null(value)) null_rate FROM feature_logs WHERE ts >= today AND cohort='CA_18_24' GROUP BY 1 ORDER BY null_rate DESC LIMIT 20;
- Staleness: SELECT avg(ts_event - ts_feature) FROM feature_logs WHERE cohort='CA_18_24';
- Experiment: Re-score a 1k-sample from today with yesterday’s pipeline artifact (backfill) and compare scores; large divergence indicates pipeline bug.
## H3. Labeling pipeline change (definition/latency)
- Check: Label arrival latency and label completeness today vs prior.
- Query: SELECT percentile(latency, [50,90,99]), completeness FROM labels WHERE ts >= today AND cohort='CA_18_24';
- If labels slowed, observed IR may be inflated/biased. Validate with manual audit sample.
## H4. Traffic mix shift (merchant, device, payment method, promo)
- Check: Contribution analysis by merchant_id, device_os, payment_method, BIN, referrer.
- Query: SELECT merchant_id, count(*) n, avg(is_incorrect) ir FROM tx WHERE ts>=today AND cohort='CA_18_24' GROUP BY 1 ORDER BY ir DESC LIMIT 20;
- Experiment: If 1–2 merchants drive delta, isolate mitigation to those.
## H5. Geography/age parsing bug (CA vs CA-Province; age boundary at midnight)
- Check: Validate geo parser inputs; compare distribution of state/province codes; examine ages 17–18 and 24–25 edges around midnight.
- Query: SELECT state, country, count(*) FROM tx WHERE ts>=today AND age BETWEEN 18 AND 24 GROUP BY 1,2;
## H6. Score distribution drift/miscalibration
- Check: Compare score histogram to baseline; KS test; calibration curves if labels available.
- Experiment: Shadow scoring with previous model on today’s features; if shadow aligns with baseline but prod drifts, model or feature change likely.
## H7. Third-party dependency degradation
- Check: Timeouts/error rates for external services (device fingerprint, address verification). If fallbacks applied, feature quality degrades.
- Query: SELECT service, error_rate FROM svc_metrics WHERE ts>=today AND cohort='CA_18_24';
## Minimum manual audit
- Pull 100–200 incorrect cases (and 100 near-threshold approvals) for CA 18–24.
- Tag root-cause categories: feature missing, merchant anomaly, time-zone, data mapping, truly new behavior. This directs mitigation choice.
# 3) Safe Mitigation Decision, Rollback Triggers, Side Effects
Decide by 10:30–11:00, even if root cause is not fully confirmed. Use canary and guardrails.
## Candidate mitigations (ordered by safety)
1) Roll back config/model to last known-good
- Use if any diff is detected in threshold/model/rules since 00:00.
- Predicted effect: restores prior precision/recall. Validate on 30–60 min window.
- Rollback trigger: N/A; this is the rollback.
2) Targeted threshold tightening for CA 18–24 only (lower t)
- Goal: reduce incorrect approvals (improve recall) at cost of higher review/false positive rate.
- Deployment: 25% canary of CA 18–24 for 60 minutes; then ramp to 100% if improving and within capacity.
- Guardrails:
- Manual review queue length < 80% capacity; 95th percentile review SLA < target.
- IR improvement ≥ 20% relative within 60–90 minutes vs control.
- Side effects (quantify): Use yesterday’s ROC/PR to simulate. Example:
- At t0: TPR=0.72, FPR=0.04.
- At t1=t0−0.05: TPR=0.80 (+8 pp), FPR=0.055 (+1.5 pp).
- With today’s cohort volume ~120k and incorrect prevalence ~3.1%:
- Added reviews ≈ ΔFPR * (1−prevalence) * n ≈ 0.015 * 0.969 * 120k ≈ 1,747 extra reviews.
- Incorrect approvals reduced ≈ ΔTPR * prevalence * n ≈ 0.08 * 0.031 * 120k ≈ 298 fewer incorrect approvals.
- Ensure Ops can absorb +1.7k reviews over the hour(s).
- Rollback triggers:
- Review SLA breach > 2x baseline for 2 consecutive 15-min windows.
- No IR improvement ≥ 10% rel after 90 minutes.
3) Feature hotfix: drop degraded features and re-weight via rules fallback
- Use if one/few features show >30% missingness or high drift PSI>0.25.
- Implementation: Temporarily exclude affected features; enable rule-based overrides (e.g., merchant-block list, AVS mismatch strictness) for CA 18–24.
- Side effects: Loss of model discrimination; expect higher false positives; monitor BR and SLA.
4) Merchant-targeted policy
- If 1–2 merchants drive majority of delta, add temporary stricter rules for those merchants (e.g., lower t only for those IDs or route to review).
## Measuring mitigation effect quickly
- Use interleaved A/B (canary vs control within CA 18–24) with 15-min metrics and CIs.
- Primary: IR among auto-approveds; Secondary: total IR, review queue metrics, customer impact (approval rate).
- If labels are delayed, use proxy: reduction in high-risk approvals near threshold and audit sampling outcomes.
# 4) Communication Plan and Artifacts
## Cadence
- 09:15 Initial incident notification (execs + finance + partner teams)
- What: anomaly detected; scope (cohort, magnitude), immediate actions (deploy freeze, triage), next update at 09:45.
- 09:45 Triage readout
- Share dashboard link, IR time series with CIs, top slices, preliminary hypotheses, and owner assignments.
- 11:00 Decision checkpoint 1
- Announce chosen mitigation (rollback/threshold canary), guardrails, expected side effects, and when to expect impact (next 60–90 min).
- 13:00 Midday update
- Results of canary/experiments, whether to ramp/adjust; any merchant-specific actions.
- 15:00 Decision checkpoint 2
- Finalize mitigation state (ramp to 100% or revert), outline prevention work.
- 17:30 EOD report
- Metrics vs baseline, incident timeline, root cause (or leading hypothesis), actions taken, next-day items.
## Artifacts to produce/share
- Live dashboard (Looker/Superset):
- IR by 15-min for CA 18–24 vs baseline; adjacent cohorts for regression-to-mean check.
- Score distribution (KS), feature nulls/staleness, label latency.
- Review queue and SLA utilization.
- Runbook page: mitigation levers, thresholds, rollback commands, owners and escalation paths.
- JIRA/Incident ticket: timeline, decisions, diffs, SQL links, shadow/backfill results.
- One-pager for execs: problem, impact ($/volume), mitigation, risk trade-offs, ETA for steady state.
# 5) Prevent Recurrence
## Observability and Guardrails
- Segment monitors: real-time IR for key cohorts (age, geo, merchant), with control charts and anomaly alerts.
- Model monitors: score drift (KS), calibration drift (Brier/NLL), feature PSI and missingness alerts with auto-fallback to rules when thresholds exceeded.
- Labeling monitors: label latency/completeness SLOs; alert on shifts.
- Capacity monitors: review queue SLA and auto-throttle on mitigation to avoid overload.
## Pre-deploy checks
- Canary by cohort: 10–20% rollout with guardrail checks (IR, approval rate, SLA) and auto-rollback on breach.
- Config integrity: checksums/signature and diff approvals; disallow same-day threshold changes without canary.
- Data contracts and schema tests for upstream features; time-zone and geo parsing unit tests.
- Backtest gate: require day-of-week and cohort-sliced validation on last 2–4 weeks before promoting a model/config.
## Process
- Blameless postmortem within 48 hours with concrete action items and due dates.
- Expand coverage of runbooks; add decision trees for common root causes (feature missingness, merchant surge, label delays).
# Risks: Accepted vs Avoided
- Accepted: temporary higher friction (more reviews/false positives) within Ops capacity; localized mitigations (CA 18–24 and/or specific merchants); short-term revenue impact to protect accuracy.
- Avoided: broad global threshold changes without canary; untested feature code changes; changes that exceed Ops capacity or breach SLAs system-wide; permanent policy shifts without analysis.
# Success by End of Day
- Quantitative:
- IR_18_24_CA reduced ≥ 20% relative from peak and within baseline tolerance bands for 90 minutes; OR explained by a verified label/measurement artifact and mitigated.
- No new anomalies in adjacent cohorts; review SLA within target; approval rate impact ≤ agreed guardrail (e.g., ≤ 2 pp drop).
- Qualitative:
- Root cause identified or top hypothesis validated with evidence; mitigation in place; rollback plan documented.
- Dashboards, runbook updates, and postmortem draft complete; owners assigned for prevention items.
# Example SQL/Analysis Snippets (Sketches)
- IR by cohort/time:
- SELECT time_bucket('15 min', ts) t, count(*) n, avg(is_incorrect::int) ir FROM tx WHERE ts>=today AND geo='CA' AND age BETWEEN 18 AND 24 GROUP BY 1 ORDER BY 1;
- Score drift:
- SELECT width_bucket(score,0,1,20) b, count(*) FROM scores WHERE ts>=today AND cohort='CA_18_24' GROUP BY 1 ORDER BY 1;
- Feature nulls:
- SELECT feature, avg((value IS NULL)::int) null_rate FROM feature_logs WHERE ts>=today AND cohort='CA_18_24' GROUP BY 1 ORDER BY null_rate DESC LIMIT 20;
- Merchant contribution:
- SELECT merchant_id, count(*) n, avg(is_incorrect::int) ir FROM tx WHERE ts>=today AND cohort='CA_18_24' GROUP BY 1 HAVING count(*)>200 ORDER BY ir DESC LIMIT 20;
# Final Notes
- Confirm whether "CA" means California or Canada immediately; misclassification alone can cause the apparent spike.
- If the 120k volume refers to all users, recompute n for CA 18–24 before declaring statistical significance and sizing mitigation.
- Keep all changes small, targeted, and reversible; prefer canaries and automatic rollbacks with clear guardrails.