Choose threshold under costs and uncertainty

Q: Choose threshold under costs and uncertainty

This question evaluates cost-sensitive threshold selection, uncertainty quantification for expected profit, calibration diagnostics, and population drift detection in a data science setting.

Q: How do I approach Statistics & Math interview questions?

Statistics & Math questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master statistics & math interviews.

Q: What difficulty level is this interview question?

This is a medium difficulty Statistics & Math question, commonly asked during Technical Screen rounds at Stripe.

Q: What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Stripe during technical interviews.

Question

Incentive Targeting: Threshold Selection, Uncertainty, Calibration, and Drift

Context: You deploy a model that sends an incentive to predicted positives. Purchases can occur without incentives; the incentive creates incremental profit only when sent to a true positive. You have validation operating points and want to choose a threshold, quantify uncertainty, check calibration, and monitor drift.

Given:

Base rate (no-incentive purchase probability): π = 4% = 0.04
Incremental benefit per true positive: B = $50
Cost per false positive (incentive + email): C = $1
Cohort size for deployment: N = 100,000
Candidate thresholds from validation:
- A: TPR = 0.70, FPR = 0.12
- B: TPR = 0.55, FPR = 0.05
- C: TPR = 0.80, FPR = 0.20

Use the expected profit formula: E[profit] = N × (π × TPR × B − (1 − π) × FPR × C)

Tasks:

Compute expected incremental profit for A, B, and C using the formula above. Which threshold is best?
Provide a 95% confidence interval for the chosen threshold’s expected profit using either a delta method or a nonparametric bootstrap. State what you resample and why.
Your model outputs probabilities. Describe and compute two calibration diagnostics you would include (e.g., Brier score and a reliability curve with ECE). Provide small numeric examples.
Outline a monthly population drift test to ensure the chosen threshold remains optimal under shifting π (base rate).

Choose threshold under costs and uncertainty

Quick Overview

Choose threshold under costs and uncertainty

Incentive Targeting: Threshold Selection, Uncertainty, Calibration, and Drift

Write your answer