Design fraud detection across channels with unknowns
Company: Amazon
Role: Data Scientist
Category: Machine Learning
Difficulty: hard
Interview Round: Onsite
A marketplace sees fraud across multiple channels (web, app, in-store) with evolving attacker behavior and sparse labels.
1) Problem framing: Define precise objectives for real-time screening vs. offline investigation. Translate business costs (false positive customer friction, false negative loss) into a cost-sensitive objective or decision rule; justify AUCPR vs. ROC, expected cost, or custom utility.
2) Data strategy: Propose segmentation (by channel, geography, new/returning users, device fingerprint), feature families (behavioral sequences, velocity features, device/network, payment, graph), and how you would augment with third-party or consortium data. Address cold-start and label scarcity.
3) Modeling approach: Compare baseline rules + gradient-boosted trees vs. deep models. When would you train a global model with channel features vs. per-channel models? How would you incorporate graph features or embeddings? Specify regularization and class-imbalance handling.
4) Unknown bad actors: Detail a pipeline that discovers emerging fraud patterns: unsupervised/anomaly detection or contrastive self-supervision to surface clusters/signals, human labeling to curate exemplars, then supervised fine-tuning. How do you prevent feedback loops and label bias?
5) Evaluation: Define offline metrics (AUCPR, cost curves, calibration) and online guardrails (customer friction rate, review queue load). Design a holdout/temporal split to avoid leakage; quantify expected dollar impact under several thresholds.
6) Robustness & drift: Describe drift detection, shadow deployment, threshold adaptation, and rollback. What leading indicators would you monitor daily?
7) Second-chance improvements: If you were to redo a past model, what concrete changes (features, objective, sampling, thresholding, data contracts) would you make and why?
8) LLM leverage: Propose a safe way to use LLMs for analyst triage or rule suggestion (prompt templates, retrieval grounding, safety filters), and how you would A/B test the workflow impact without exposing PII.
Quick Answer: This question evaluates a data scientist's competence in designing and operationalizing multi-channel fraud detection systems, covering cost-sensitive objective formulation, segmentation and feature engineering, model selection (including sequence and graph approaches), anomaly detection for unknown actors, drift monitoring, and safe LLM-assisted workflows. Commonly asked in the Machine Learning domain to assess end-to-end systems thinking and trade-off reasoning between customer experience and fraud loss, it tests both conceptual understanding and practical application-level skills such as evaluation, deployment guardrails, and handling label sparsity and delay.
