Fraud And Anomaly Detection In Retail Marketplace

What's being tested

Interviewers are probing your ability to detect, quantify, and mitigate fraud/anomalies in a retail marketplace using statistical reasoning, metric design, and model evaluation. Expect to justify metric choices under heavy class imbalance, reason about label delay/noise, design experiments that measure interventions’ causal impact, and trade off business cost versus customer friction. At Walmart scale they care about robust diagnostics, explainability, and tight decision thresholds that optimize expected loss, not just raw ML metrics.

Core knowledge

Fraud types: Understand account takeovers, fake listings, payment fraud, return/refund abuse, and review manipulation; each implies different signals, latency, and remediation cost.
Signal sources: Instrumentation includes order events, payment events, account creation/changes, device/IP telemetry, review activity, and manual review labels — treat these as upstream signals, not infra design.
Labels & delay: Chargebacks and manual reviews are reliable but delayed; use positive-unlabeled (PU) learning, survival analysis, or delayed-label evaluation windows to avoid optimistic bias.
Imbalanced metrics: Prefer precision@k, precision-recall AUC, and business-oriented expected-utility over ROC AUC; precision = TP/(TP+FP), recall = TP/(TP+FN), expected value = TPB - FPC.
Supervised vs unsupervised: Use supervised (XGBoost, LightGBM) when labels exist; use Isolation Forest, LOF, or reconstruction autoencoders for unlabeled anomaly scoring; hybridize with rule-based heuristics.
Feature engineering: Temporal aggregates, session statistics, velocity features, graph features (buyer-seller degree, edge churn), embeddings for sequences, and cross-feature interactions capture behavior patterns.
Model validation: Use time-based cross-validation to prevent leakage; backtest with rolling windows and simulate interventions; calibrate scores with Platt or isotonic methods for threshold decisions.
Cost-sensitive thresholds: Optimize threshold using business cost matrix (Cost_FP, Cost_FN) via expected-utility maximization or constrained optimization (maximize TP subject to FP ≤ budget).
Multiple testing & alerting: For many concurrent anomalies, control false discoveries with Benjamini–Hochberg or use hierarchical alerting to prioritize impactful signals.
Drift & monitoring: Monitor feature distributions, precision@k over time, and post-decision outcomes (reversal rates) to detect concept drift; set SLA for investigation latency.
Explainability & compliance: Use SHAP for feature attribution, produce human-readable rules for blocking actions, and track appeal/override rates to refine models.
Experimental design: For interventions (blocks, holds), run randomized rollouts or difference-in-differences when randomization isn't possible; monitor retention, conversion, and false-positive harm.

Worked example — “Detect anomalous spike in seller chargebacks”

Frame: ask what defines a chargeback spike (absolute vs relative), acceptable detection latency, business impact per chargeback, and whether rollout is global or cohorted. Skeleton: (1) define metric chargebacks_per_1000_orders and rolling baseline (7- and 28-day), (2) run statistical anomaly detectors (EWMA/CUSUM) and segment by seller cohorts, (3) drill down with cohort and graph features to distinguish correlated spikes (e.g., bot campaign) from isolated seller fraud, (4) if labels exist, train a classifier to predict future chargebacks. Tradeoff to flag: sensitivity vs operational load — set thresholds by expected cost (TPBenefit - FPCost). Close: if more time, build seller-graph propagation features, run an A/B test of a soft-hold intervention, and instrument downstream merchant appeals to measure false-positive harm.

A second angle — “Design a model to score new accounts for onboarding risk”

Same core ideas apply but constraints shift: new accounts lack history, so rely on device/IP signals, behavioral heuristics, and graph-based transductive features (shared payment instruments). Use XGBoost with strong regularization and calibrated outputs. Because false rejections hurt acquisition, optimize for high recall at low FP cost using cost-sensitive learning or low-friction mitigation (phone verification) instead of outright block. Validate with synthetic labels (simulated fraud) and monitor cohort conversion and lifetime-value to ensure onboarding friction doesn't reduce long-run revenue.

Common pitfalls

Pitfall: Over-relying on ROC AUC for imbalanced fraud problems. ROC can be misleading when positive prevalence is tiny; prefer precision-recall curves, precision@k, and business-utility metrics.

Pitfall: Ignoring label delay and leakage. Training on chargeback-confirmed events without accounting for delay leads to optimistic models; always define an evaluation window and consider PU or survival approaches.

Pitfall: Presenting only model metrics without business impact. Saying “AUC=0.92” is weaker than “At threshold X, we prevent $Y of expected loss while causing Z customer holds per day.”

Connections

Interviewers may pivot to graph analytics (fraud rings), causal inference/experimentation (measuring interventions), or model-monitoring and MLOps questions (drift detection, rollback policies). Familiarity with cross-functional tradeoffs (legal/compliance, operations) strengthens answers.