Design a traditional fraud detection system
Overview
This question evaluates a Machine Learning Engineer's competency in end-to-end ML system design for real-time payments fraud detection, including labeling under delayed confirmations, handling extreme class imbalance and sampling, feature engineering across behavioral, graph, device and merchant signals, model selection for latency and scale, and production scoring and monitoring architecture. It is commonly asked in the ML System Design category to assess how an engineer balances low-latency decision-making with delayed sparse labels, calibration and threshold trade-offs, operational scalability and resiliency, and drift/adversarial detection, testing both conceptual understanding and practical application.
Design an End-to-End Real-Time Payments Fraud Detection System
Context: You are designing a fraud detection system for a large-scale online payments platform. Decisions must be made synchronously at authorization time with tight latency budgets, while confirmed fraud labels (e.g., chargebacks) arrive late and are scarce.
Specify and justify the following:
- Labeling strategy under delayed, scarce confirmations
- How to define positive/negative labels when chargebacks arrive weeks later.
- Aging/observation windows, handling disputed outcomes, and avoiding target leakage.
- Sampling to handle extreme class imbalance
- Offline training strategies (downsampling, weighting) and how to keep calibration.
- Online serving considerations.
- Feature sets
- Behavioral/velocity features.
- Graph/link features across users, devices, payment instruments.
- Device/network features.
- Merchant/context features.
- Model choices and justification
- Baseline and advanced models suitable for latency and scale.
- Handling graphs, sequences, and semi-/weak supervision.
- Real-time scoring architecture and latency constraints
- Event ingestion, online/offline feature store, streaming aggregations, model serving, and fallbacks.
- Expected P99 latency budget and resiliency.
- Thresholding and precision/recall trade-offs
- Decision policies (approve/review/decline) using cost-aware thresholds and calibration.
- Evaluation metrics
- PR-AUC, precision@k, expected-cost/profit metrics, and how to evaluate with delayed labels and policy bias.
- Monitoring for drift and adversarial adaptation
- Detecting data/model drift, label delay proxies, and adversarial pattern monitoring.
- Retraining cadence, rollout, and guardrails.
Solution
Login required