Build an uplift model for targeting
Overview
This question evaluates a data scientist's ability to perform treatment-effect and uplift modeling for multi-armed marketing interventions, encompassing causal inference with randomized holdouts and propensity scoring, cost-sensitive channel selection, calibration and class-imbalance handling, policy evaluation, and diagnostic/operational safeguards within the Machine Learning domain. It is commonly asked because it probes practical application of causal ML and decision-policy design—requiring model selection, offline and online evaluation under budget and fairness constraints—and is primarily a practical application task with important conceptual causal-inference elements.
Flu-shot Campaign: Treatment-Effect Modeling and Targeting Policy
You have historical campaign logs from last season that include randomized holdouts. You must design a treatment-effect modeling and targeting approach to decide whether to contact a customer by SMS or Email for the upcoming flu-shot campaign.
Data Available
- Features (pre-treatment only for modeling): demographics, past visits, prior vaccinations, engagement history (prior opens/clicks), distance to store, appointment history.
- Labels: y = 1 if vaccinated within 30 days; 0 otherwise.
- Treatments: T ∈ {control, SMS, Email}, assigned at random with known propensities p_t.
- Exposure indicators (post-assignment): delivery status, opened. Use for diagnostics/mediation only (avoid leakage in ITT models).
- Costs: c_SMS = 0.001.
- Operational constraint: may contact at most 40% of eligibles.
Tasks
-
Modeling
- Choose and justify an approach among: separate response models + two-model uplift, direct uplift/meta-learners (T-/S-/DR-learner), or multiclass treatment modeling.
- Address leakage (post-treatment features), class imbalance, and probability calibration.
-
Evaluation
- Define offline evaluation: uplift/Qini curves and AUUC; compute incremental ROI including channel costs.
- Use policy evaluation with inverse propensity weighting (IPW) or doubly-robust (DR) estimators.
-
Policy
- With the 40% contact budget, describe how to rank customers by predicted incremental effect and choose the channel per customer (e.g., argmax of channel-specific uplift minus cost scaled by value).
- Explain guardrails (do-not-contact lists, fairness across age/state, frequency caps).
-
Online Validation
- Propose a gated rollout comparing model-based targeting vs uniform random targeting (both constrained to 40% contact rate).
- Define success metrics and stopping rules.
-
Diagnostics
- Describe how to detect and mitigate harmful persuasion (negative uplift) segments, and how you would handle them in targeting.
Solution
Login required