Optimize thresholds under fraud costs

Quick Overview

This English summary evaluates a candidate's competency in cost-sensitive binary classification, threshold selection, and ROC-based operating-point analysis, including calculating expected costs and predictive values under varying prevalence and asymmetric error costs.

Optimize thresholds under fraud costs

Company: PayPal

Role: Data Scientist

Category: Statistics & Math

Difficulty: medium

Interview Round: Technical Screen

Cost-sensitive thresholding: You have a binary ATO classifier scoring transfers. Prevalence (fraud rate) is 0.2% initially. Costs: false positive (blocking a legit transfer) costs $2; false negative (letting a fraud through) costs $120. Consider the following operating points measured on a large validation set: Threshold | TPR | FPR 0.90 | 0.50| 0.0010 0.80 | 0.65| 0.0030 0.70 | 0.75| 0.0060 0.60 | 0.82| 0.0100 0.50 | 0.88| 0.0180 Assume 1,000,000 evaluated transfers. Tasks: A) For each threshold, compute expected total cost = (FP * $2) + (FN * $120) given prevalence 0.2%. Choose the cost-minimizing threshold and report PPV and NPV at that point. B) If prevalence drops to 0.1% due to seasonality, recompute and discuss whether the optimal threshold changes. C) Using ROC theory, derive the cost-optimal decision rule slope λ = (C_fp/C_fn) * ((1−π)/π) and explain how it maps to choosing a point on the ROC; interpret how prevalence shifts move the optimal operating point without retraining.

Quick Answer: This English summary evaluates a candidate's competency in cost-sensitive binary classification, threshold selection, and ROC-based operating-point analysis, including calculating expected costs and predictive values under varying prevalence and asymmetric error costs.

|Home/Statistics & Math/PayPal
PayPal logo
PayPal
Oct 13, 2025, 9:49 PM
mediumData ScientistTechnical ScreenStatistics & Math
9
0

Cost-sensitive Thresholding for Fraud (ATO) Classifier

Context

You are evaluating a binary classifier for account takeover (ATO) fraud on a large validation set. The model outputs a score; you can choose a decision threshold. Fraud prevalence initially is 0.2%. Costs are asymmetric: a false positive (blocking a legitimate transfer) costs 2,whileafalsenegative(lettingafraudthrough)costs2, while a false negative (letting a fraud through) costs 120. Assume 1,000,000 evaluated transfers.

The validation ROC operating points are:

ThresholdTPRFPR
0.900.500.0010
0.800.650.0030
0.700.750.0060
0.600.820.0100
0.500.880.0180

Assume these TPR/FPR values are stable when prevalence shifts (ROC is prevalence-invariant) and that correct classifications have zero cost.

Tasks

A) At prevalence π = 0.2% (0.002), compute for each threshold the expected total cost over 1,000,000 transfers:

  • Total cost = (FP × 2)+(FN×2) + (FN × 120) Then choose the cost-minimizing threshold and report PPV and NPV at that point.

B) If prevalence drops to π = 0.1% (0.001) due to seasonality, recompute the expected costs and discuss whether the optimal threshold changes.

C) Using ROC theory, derive the cost-optimal slope

  • λ = (C_fp / C_fn) × ((1 − π) / π) Explain how this slope maps to choosing a point on the ROC curve (i.e., the tangent condition on the ROC/ROC convex hull), and interpret how prevalence shifts move the optimal operating point without retraining.
Loading comments...