Build an uplift model for targeting
Company: CVS Health
Role: Data Scientist
Category: Machine Learning
Difficulty: hard
Interview Round: Technical Screen
You have historical campaign logs with randomized holdouts from last season. Design a treatment effect modeling approach to decide whom to contact by SMS or Email for the upcoming flu-shot campaign.
Data available
- Features: demographics, past visits, prior vaccinations, engagement (opens/clicks), distance to store, appointment history.
- Labels: y = vaccinated within 30 days; treatments T ∈ {control, SMS, Email}; randomized assignment with known probabilities; exposure indicators (delivered/opened).
- Costs: c_SMS=$0.02, c_Email=$0.001; budget allows contacting at most 40% of eligibles.
Tasks
1) Modeling
- Choose and justify an approach: separate response models + two-model uplift, direct uplift (e.g., meta-learners: T-learner/S-learner/DR-learner), or multiclass treatment modeling. Address leakage (post-treatment features), class imbalance, and calibration.
2) Evaluation
- Define offline evaluation: uplift/Qini curves, AUUC; compute incremental ROI considering channel costs; use policy evaluation with inverse propensity weighting (IPW) or doubly robust estimators.
3) Policy
- Given a budget contacting up to 40% of eligibles, describe how to rank customers by predicted incremental effect and choose the channel per customer (e.g., argmax over channel-specific uplift minus cost). Explain guardrails (do-not-contact, fairness across age/state).
4) Online validation
- Propose a gated rollout test comparing model-based targeting vs uniform random targeting. Define success metrics and stopping rules.
5) Diagnostics
- Show how you would detect harmful persuasion (negative uplift) segments and handle them in targeting.
Quick Answer: This question evaluates a data scientist's ability to perform treatment-effect and uplift modeling for multi-armed marketing interventions, encompassing causal inference with randomized holdouts and propensity scoring, cost-sensitive channel selection, calibration and class-imbalance handling, policy evaluation, and diagnostic/operational safeguards within the Machine Learning domain. It is commonly asked because it probes practical application of causal ML and decision-policy design—requiring model selection, offline and online evaluation under budget and fairness constraints—and is primarily a practical application task with important conceptual causal-inference elements.