Select interest thresholds under skewness and cost
Overview
This question evaluates a data scientist's competency in profit-oriented threshold selection using calibrated interest scores, uplift estimation, and decision-theoretic optimization under cost and weekly capacity constraints.
Profit-Optimal Threshold Selection from an Interest Score
You have a per-user interest_score s ∈ [0, 1] for a new feature. The score distribution appears right-skewed. You can contact at most K users per week. Each contact has a known cost c. Let b(s) denote the expected incremental benefit (e.g., monetized uplift) if a user with score s is contacted.
Propose a principled method to choose a score threshold (e.g., 75th or 90th percentile) that maximizes incremental value under the cost constraint. Specifically:
- Define notation and the profit objective.
- Explain how to estimate b(s) from historical data, e.g., via isotonic calibration or splines.
- Show how to compute the profit-optimal cutoff using expected lift × margin − c, under a capacity constraint K.
- Explain how the approach adapts if the score distribution is left-skewed.
- Describe how to prevent instability due to sampling noise (e.g., Bayesian shrinkage or percentile CIs).
- Describe how to set up an ongoing backtest to validate the threshold against alternatives.
Provide formulas and a step-by-step selection algorithm.
Solution
Login required