Integrity, Harm, And Fraud Measurement — Tech Interview Concept

What's being tested
Judgment about designing robust measurement for integrity/fraud interventions: choosing business-relevant metrics, experimental design (randomization, unit, power), and handling noisy/biased labels and adversarial drift. Also tested is tradeoff reasoning between false positives (user harm) and false negatives (undetected abuse).
Core knowledge

Precision, recall, FDR, FPR, and F-beta: when to emphasize each for user-harm vs fraud-loss tradeoffs.
Precision@k, lift, and calibrated risk scores for operational triage and human-review prioritization.
A/B test design: unit of randomization, SUTVA violations, cluster randomization, and power/sample-size basics.
Causal methods: randomized trials, difference-in-differences, and instrumental variables for nonrandom rollout.
Labeling pitfalls: sampling bias, reviewer error, delayed labels, and techniques like active sampling.
Model/tooling: XGBoost/LightGBM for tabular, Isolation Forest/LOF for anomaly detection, GNN/label-propagation for graph fraud.
Adversarial dynamics: concept drift, attacker adaptation, feedback loops, and need for continuous monitoring and decay windows.

Worked example — "Design metrics to evaluate a fraud-detection model balancing false positives and false negatives"
Frame the problem by defining concrete objectives (reduce fraud loss $L_f$ ) and guardrails (minimize user-disruption rate UDR). Pick primary and secondary metrics: expected fraud dollars prevented per 1k impressions (lift) and precision at operational threshold, plus daily UDR and appeals rate as guardrails. Choose unit of randomization (account or session), run power calc on expected fraud rate and cost asymmetry, and plan pre-specified stopping rules, monitoring windows (incubation for chargebacks), and fallback procedures for catastrophic FP spikes.
A common pitfall
Focusing solely on AUC or aggregate accuracy is tempting but misleading: AUC ignores calibration and class-imbalance costs, and accuracy obscures asymmetric business impact. Another common error is treating offline labeled data as ground truth without accounting for reviewer bias, delayed detection, or attacker adaptation — leading to optimistic offline performance that fails in production.
Further reading

Kohavi et al., "Trustworthy Online Controlled Experiments" (principles for large-scale A/B testing and guardrails).
Davis & Goadrich, "The relationship between Precision-Recall and ROC curves" (why PR is better for imbalanced fraud problems).

Related concepts