Design real-time payments fraud model under constraints
Quick Overview
This question evaluates a data scientist's competency in real-time fraud detection and policy design, including cost-sensitive modeling, handling delayed/positive–unlabeled labels, severe class imbalance, low-latency feature engineering and online feature stores, drift and adversarial monitoring, offline policy evaluation, and fairness and UX constraints; it belongs to the Machine Learning domain and tests both conceptual understanding and practical system-level application. It is commonly asked to assess an interviewee's ability to balance latency and business costs, reason about delayed and noisy labels, design deployable low-latency architectures, and define evaluation metrics and safety guardrails for production fraud policies.
Real-Time ML Policy Design: Prevent Unauthorized Purchases by Minors
Context: You need to reduce unauthorized purchases by minors using their parents' credit cards on a large gaming platform. Decisions must be made at checkout in real time from actions {allow, step-up auth (e.g., CVV/SCA), hold-for-review, block} under a 30 ms p99 latency budget.
Answer precisely:
-
Problem framing and labels
- Chargebacks/disputes arrive 2–8 weeks later and some cases are never disputed. Define what constitutes positive and negative outcomes. Would you treat this as positive–unlabeled (PU) learning, cost-sensitive classification, or uplift modeling for action choice? Justify your choice.
-
Class imbalance
- Positives are ~0.2%. Specify the loss and sampling/weighting strategy (e.g., focal loss vs class weights) and how you will calibrate scores. Show the decision threshold formula that minimizes expected cost: argmin_t [FP(t)*C_fp + FN(t)*C_fn + ActionCosts].
-
Features
- Propose high-signal, low-latency features (e.g., payment velocity, device consistency, age-on-payment, billing-IP mismatch, historical dispute rates, network/household signals). Explain leakage risks and how you will implement out-of-fold target encoding safely.
-
Real-time architecture
- Sketch the online feature store, TTLs, and fallbacks for cold-start or feature timeouts. What is cached at the edge versus computed on demand? How do you enforce p99 < 30 ms?
-
Drift and adversaries
- Describe backtesting with strictly forward time splits, population stability (PSI) monitors, and online shadow evaluation. How do you update without amplifying feedback loops?
-
Evaluation
- Choose metrics beyond PR-AUC (e.g., cost curves, expected profit, constrained ROC for max FP rate). Describe offline policy evaluation (IPS/DR) to estimate the impact of step-up auth vs block before running a risky full A/B test.
-
Safety and UX
- Propose a tiered action policy (risk score → action), human review routing, and appeals. What fairness/age-related checks do you implement, and what business guardrails (e.g., max block rate for verified adults) do you enforce?