Design fraud detection from raw transactions
Overview
This question evaluates a candidate's ability to design an end-to-end transaction fraud detection system, testing competencies in feature engineering, handling delayed and partial labels and class imbalance, real-time feature retrieval and model serving, decision thresholding under manual-review constraints, feedback-loop integration, monitoring, and backtesting. It is commonly asked in ML system design interviews to assess both conceptual understanding and practical application of production machine learning and data engineering trade-offs in the ML System Design domain for data scientist roles.
System Design: End-to-End Transaction Fraud Detection
Context
You are given a large, multi-table dataset of transactions and customer/merchant metadata. Fraud labels arrive with delays (e.g., chargebacks weeks later) and may be partial (e.g., only for reviewed or disputed transactions). Design an end-to-end system to decide, in real time, whether to approve, decline, or send each transaction to manual review.
Requirements
Cover the following aspects with clear assumptions and rationale:
-
Data and Feature Engineering
- Velocity features (multi-horizon counts/sums/uniques).
- Graph/link features across entities (user/card/email/device/IP/merchant).
- Device and IP signals (fingerprinting, geolocation, proxy/TOR, ASN risk).
-
Labels, Class Imbalance, and Latency
- Handling severe class imbalance.
- Handling delayed/partial labels and selective-label bias.
-
Training and Validation Splits
- Splitting to avoid leakage in time and across entities.
- Ensuring offline/online feature parity.
-
Decision Thresholding and Review Capacity
- Approve/Decline/Review policy with cost-sensitive thresholds.
- Meeting a fixed manual review capacity.
-
Real-Time Scoring and Latency Budgets
- Online feature retrieval and model serving under strict latency.
- Fallbacks and degradation strategies.
-
Feedback Loops
- Incorporating manual review outcomes and chargebacks.
- Exploration/holdout strategies to mitigate bias.
-
Monitoring and Alerting
- Drift (input/output), TPR/FPR with delayed labels, calibration, approval/decline rates, review queue health.
-
Backtesting Plan
- Time-ordered replay, off-policy evaluation, simulation of review capacity, metrics and confidence intervals.
State reasonable assumptions if needed and justify key design choices.
Solution
Login required