Assume a deployment sends an incentive to predicted positives. Base rate π = 4% purchase without incentive. Benefit per true positive (incremental profit) = $50. Cost per false positive (incentive + email) = $1.00. Consider three candidate operating points from validation: A: TPR=0.70, FPR=0.12; B: TPR=0.55, FPR=0.05; C: TPR=0.80, FPR=0.20. (1) For a cohort of 100,000 users, compute expected incremental profit for A, B, C using E[profit] = 100000*(π*TPR*50 − (1−π)*FPR*1). Which threshold is best? (2) Provide a 95% CI for the chosen threshold’s profit using a delta method or a nonparametric bootstrap; specify what distributional pieces you resample and why. (3) Your model outputs probabilities; describe and compute two calibration diagnostics you would include (e.g., Brier score and reliability curve with ECE). (4) Outline a population drift test you would run monthly to ensure the chosen threshold remains optimal under shifting π.

This question evaluates cost-sensitive threshold selection, uncertainty quantification for expected profit, calibration diagnostics, and population drift detection in a data science setting.

How do I approach Statistics & Math interview questions?

Statistics & Math questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master statistics & math interviews.

What difficulty level is this interview question?

This is a medium difficulty Statistics & Math question, commonly asked during Technical Screen rounds at Stripe.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Stripe during technical interviews.

Choose threshold under costs and uncertainty

Incentive Targeting: Threshold Selection, Uncertainty, Calibration, and Drift

Context: You deploy a model that sends an incentive to predicted positives. Purchases can occur without incentives; the incentive creates incremental profit only when sent to a true positive. You have validation operating points and want to choose a threshold, quantify uncertainty, check calibration, and monitor drift.

Given:

Base rate (no-incentive purchase probability): π = 4% = 0.04
Incremental benefit per true positive: B = $50
Cost per false positive (incentive + email): C = $1
Cohort size for deployment: N = 100,000
Candidate thresholds from validation:
- A: TPR = 0.70, FPR = 0.12
- B: TPR = 0.55, FPR = 0.05
- C: TPR = 0.80, FPR = 0.20

Use the expected profit formula: E[profit] = N × (π × TPR × B − (1 − π) × FPR × C)

Tasks:

Compute expected incremental profit for A, B, and C using the formula above. Which threshold is best?
Provide a 95% confidence interval for the chosen threshold’s expected profit using either a delta method or a nonparametric bootstrap. State what you resample and why.
Your model outputs probabilities. Describe and compute two calibration diagnostics you would include (e.g., Brier score and a reliability curve with ECE). Provide small numeric examples.
Outline a monthly population drift test to ensure the chosen threshold remains optimal under shifting π (base rate).

Incentive Targeting: Threshold Selection, Uncertainty, Calibration, and Drift

Given:

Base rate (no-incentive purchase probability): π = 4% = 0.04

Incremental benefit per true positive: B = $50

Cost per false positive (incentive + email): C = $1

Cohort size for deployment: N = 100,000

Candidate thresholds from validation:

A: TPR = 0.70, FPR = 0.12
B: TPR = 0.55, FPR = 0.05
C: TPR = 0.80, FPR = 0.20

Use the expected profit formula: E[profit] = N × (π × TPR × B − (1 − π) × FPR × C)

Tasks:

Compute expected incremental profit for A, B, and C using the formula above. Which threshold is best?

Provide a 95% confidence interval for the chosen threshold’s expected profit using either a delta method or a nonparametric bootstrap. State what you resample and why.

Your model outputs probabilities. Describe and compute two calibration diagnostics you would include (e.g., Brier score and a reliability curve with ECE). Provide small numeric examples.

Outline a monthly population drift test to ensure the chosen threshold remains optimal under shifting π (base rate).

Choose threshold under costs and uncertainty

Quick Overview

Choose threshold under costs and uncertainty

Incentive Targeting: Threshold Selection, Uncertainty, Calibration, and Drift

Write your answer

Choose threshold under costs and uncertainty

Quick Overview

Choose threshold under costs and uncertainty

Incentive Targeting: Threshold Selection, Uncertainty, Calibration, and Drift

Write your answer