Incentive Targeting: Threshold Selection, Uncertainty, Calibration, and Drift

Context: You deploy a model that sends an incentive to predicted positives. Purchases can occur without incentives; the incentive creates incremental profit only when sent to a true positive. You have validation operating points and want to choose a threshold, quantify uncertainty, check calibration, and monitor drift.

Given:

• Base rate (no-incentive purchase probability): π = 4% = 0.04
• Incremental benefit per true positive: B = $50
• Cost per false positive (incentive + email): C = $1
• Cohort size for deployment: N = 100,000
• Candidate thresholds from validation:
  • A: TPR = 0.70, FPR = 0.12
  • B: TPR = 0.55, FPR = 0.05
  • C: TPR = 0.80, FPR = 0.20

Use the expected profit formula: E[profit] = N × (π × TPR × B − (1 − π) × FPR × C)

Tasks:

1. Compute expected incremental profit for A, B, and C using the formula above. Which threshold is best?
2. Provide a 95% confidence interval for the chosen threshold’s expected profit using either a delta method or a nonparametric bootstrap. State what you resample and why.
3. Your model outputs probabilities. Describe and compute two calibration diagnostics you would include (e.g., Brier score and a reliability curve with ECE). Provide small numeric examples.
4. Outline a monthly population drift test to ensure the chosen threshold remains optimal under shifting π (base rate).