A) 30/60/90-day plan and concrete outputs Assumptions: ATO labels have 14-day median lag; feature flags exist for rules/step-ups; BI = Looker; modeling stack = SQL + Python; device/IP novelty = first-seen within 30 days. Days 1–30 (Discover, instrument, baseline) - Week 1: Align on goals and definitions - Output: 1-pager with canonical metric definitions (ATO, legit, blocked, step-up, novelty), time-of-event conventions (attempt vs authorization), lookback windows, label-lag handling. - Stakeholders: Risk Ops, PM, CS/Disputes. - Week 2: Foundational dashboard MVP - Output: Looker dashboard v1 with daily ATO loss, ATO rate, legit block rate; sliced by device/IP novelty and geography. Includes data freshness and label-lag banners. - Week 3: Data QA and lineage - Output: dbt or SQL tests (freshness, volume, nulls, referential integrity), lineage doc, and data contract draft for key tables/fields (schemas, SLAs, PII handling). - Week 4: Hypothesis and rule ideation - Output: Shortlist of rule candidates (e.g., risk-based step-up on novel device+velocity+amount), with backtestable SQL specs and risk score thresholds. Days 31–60 (Backtest, CBA, experiment design) - Week 5: Offline backtests - Output: Rule backtest report with TPR/FPR by segment, expected coverage, and collision with existing controls. - Week 6: Cost–benefit analysis (CBA) and design review - Output: CBA deck and memo; propose experiment design (A/B or sequential test), sample size, guardrails, and rollback conditions. Present to PM + Risk Ops. - Week 7: Experiment readiness - Output: Feature flag config, monitoring tiles (near-real-time), on-call/rollback runbook. - Week 8: Launch limited ramp - Output: 5–10% traffic ramp; daily experiment digest to stakeholders. Days 61–90 (Scale, codify, handoff) - Week 9: Analyze interim results - Output: Mid-experiment readout with power check and preliminary net impact; adjust thresholds if needed. - Week 10: Ramp to 50% if guardrails pass - Output: Updated CBA; PRD addendum for policy change; finalize operational playbooks (appeals, overrides). - Week 11: Decision and rollout - Output: Go/No-Go doc; change log; staged rollout plan (by region/user risk tiers). - Week 12: Codify and handoff - Output: Final dashboard v2, metric contracts in BI semantic layer, runbook, and postmortem/retrospective. B) Five core dashboard tiles with precise formulas Notation: T = all transaction attempts in window; L = legit transactions (not later confirmed ATO/fraud within 30 days); A = confirmed ATO transactions; $loss_i = reimbursed USD loss for ATO txn i. Novel_N = transactions where device_id_first_seen_days ≤ N OR ip_first_seen_days ≤ N (use N=30 unless stated). Use cohorting to handle label lag (report ATO on a 30-day completed cohort) and a provisional view for near-term monitoring. 1) ATO Loss per 1K Transactions - Formula: 1000 × (Σ_{i∈A} $loss_i) / |T| - Denominator: all transaction attempts in window. - Example: 1,000,000 tx; 500 ATO; avg loss $200 → Loss/1K = 1000 × (500×200)/1,000,000 = 100. 2) Confirmed ATO Rate per 1K Transactions - Formula: 1000 × |A| / |T| - Denominator: all transaction attempts. - Example: 500/1,000,000 × 1000 = 0.5 per 1K. 3) Legitimate Block Rate per 1K Legit Transactions - "Blocked" = hard decline or abandoned due to added friction; exclude system/timeouts not tied to risk. - Formula: 1000 × |Blocked ∩ L| / |L| - Denominator: legit transactions (post-30d label). Provide a provisional view using L_provisional (no known risk flags within 7d) with a banner. - Example: 2,000 blocked legit out of 995,000 legit → 1000 × 2000/995000 ≈ 2.01 per 1K. 4) Step-up Success Rate - "Challenged" = users who received MFA/step-up; "Completed" = passed step-up and continued. - Formula (overall): |Completed| / |Challenged| - Breakouts recommended: by device novelty, OS, and legit-only completion rate = |Completed ∩ L| / |Challenged ∩ L|. - Example: 10,000 challenged, 8,500 completed → 85%. 5) Novelty ATO Rate per 1K Novel Transactions - Novelty definition (N=30 days): device_first_seen_days ≤ 30 OR ip_first_seen_days ≤ 30. - Formula: 1000 × |A ∩ Novel_30| / |T ∩ Novel_30| - Example: 100,000 novel tx; 300 ATO → 1000 × 300/100000 = 3 per 1K. Implementation notes - Time alignment: For tiles 1–3 and 5, prefer transaction-date cohorts with a 30-day labeling window; show both cohort-complete and near-real-time provisional panels. - Currency: Convert losses to a single currency daily (FX at txn date). - Dedup: Count at attempt level; exclude retries after hard declines to avoid double counting. C) ~150-word executive update (proposed rule change) Proposal: Introduce risk-based step-up for P2P sends when a novel device/IP (first seen ≤30 days) is combined with high velocity (≥3 sends in 1 hour) or amount ≥$200. Offline backtests show 42% coverage of historical ATOs with a 6.5% false-positive rate on legit traffic in this segment. Expected impact: reduce ATO loss by ~25% (≈$30k/week from a $120k/week baseline) while challenging ~1.2% of legit sends; with 85% step-up completion, projected incremental drop-off is ~0.18%, costing ≈$8k/week in foregone throughput; added manual review cost <$2k/week. Net benefit ≈+$20k/week. Guardrails: holdout 10%; ramp 10%→50%; auto-rollback if (a) legit block rate +0.5 per 1K, (b) step-up completion <80%, or (c) CS ATO complaints +20% w/w. Assumptions: 14-day label lag; loss recovery rate unchanged; novelty defined as first-seen ≤30 days. We will monitor by geography/device and adjust thresholds to minimize friction on established users. D) SQL vs. Python and governance for trusted BI When to use SQL - Canonical metrics and dashboards: aggregations, filters, window functions, joins on modeled tables. - Reproducible transformations: implemented in dbt/ETL with tests and the BI semantic layer. - Backtests expressible as set logic (rule coverage, TPR/FPR) and daily monitoring queries. When to use Python - Statistical inference and experimentation: power/sequential tests, CUPED, bootstrap CIs, uplift modeling. - Modeling/simulation: risk score calibration, threshold optimization, Monte Carlo for loss distributions. - Feature engineering beyond SQL ergonomics (NLP on device strings, graph features), API integrations, and notebook-driven EDA that graduates to packaged scripts. Governance to ensure trust - Version control: all SQL/Python in Git; PR reviews with code owners; linting; CI runs query tests. - Semantic layer: single source of truth for metric definitions (Looker Explores/dbt metrics). No ad-hoc redefinitions. - Data contracts: schemas, SLAs, PII policies, and deprecation rules for core tables/fields; breaking changes require approvals. - Testing/observability: dbt tests (unique/not null/referential), freshness/volume monitors, anomaly alerts on key metrics. - Reproducibility: analysis templates capture dataset snapshot, commit hash, parameters, and seeds; notebooks are parameterized and promoted to scripts/jobs. - Experiment governance: pre-registered analysis plans, guardrail metrics, holdouts, and change logs; rollback runbooks and audit trails for rule changes. CBA blueprint (use in Week 6 and decisioning) - Net impact per 1K tx = Avoided ATO loss − Friction cost (legit drop-off × margin) − Ops cost (manual reviews) − Vendor/auth cost. - Validate with sensitivity bands (±10–20%) on prevalence, completion, and loss severity; decide using worst-case acceptable net.